1c85b60 · urbit.org · git.iko.soy

git.iko.soy/urbit.org

on master help sign in

mark suggestions

1c85b60JONATHAN PAPROCKI6 years ago

tutorials/arvo/parsing.md+100−94

compare with master

unifiedsplit

@@ -24,37 +24,38 @@ Parsing a string is rarely done all at once. Instead, it is usually done

24character-by-character, and the return contains the data structure representing

25what has been parsed thus far as well as the remainder of the string to be

26parsed. They also need to be able to fail in case the input is improperly

27−formed. We will see each of these standard practices implemented in Hoon below.

27+formed. We will see each of these standard practices implemented in [Hoon below](#parsing-in-hoon).

29## Functional parsers

31How parsers are built varies substantially depending on what sort of programming

32language it is written in. As Hoon is a functional programming language, we will

33be focused on understanding _functional parsers_, also known as _combinator

34−parsers_.

34+parsers_. In this section we will make light use of pseudocode, as introducing

35+the Hoon to describe functional parsers here creates a chicken-and-egg problem.

36Complex functional parsers are built piece by piece from simpler parsers

37Complex functional parsers are built piece by piece from simpler parsers

37that are plugged into one another in various ways to perform the desired task.

38that are plugged into one another in various ways to perform the desired task.

39The basic building blocks, or primitives, are parsers that read only a

40The basic building blocks, or primitives, are parsers that read only a

40single character. There are frequently a few types of possible input characters,

41single character. There are frequently a few types of possible input characters,

41−such as letters, numbers, and symbols. For example, `parse(integer, "1")` calls

42+such as letters, numbers, and symbols. For example, `parse("1", integer)` calls

42the parsing routine on the string `"1"` and looks for an integer, and so it

43the parsing routine on the string `"1"` and looks for an integer, and so it

43returns the integer `1`. However, taking into account what was said above about

44returns the integer `1`. However, taking into account what was said above about

44parsers returning the unparsed portion of the string as well, we should

45parsers returning the unparsed portion of the string as well, we should

45represent this return as a tuple. So we should expect something like this:

46represent this return as a tuple. So we should expect something like this:

46```

47```

47−> parse(integer, "1")

48+> parse("1", integer)

48(1, "")

49(1, "")

49−> parse(integer, "123")

50+> parse("123", integer)

50(1, "23")

51(1, "23")

51```

52```

52What if we wish to parse the rest of the string? We would need to apply the

53What if we wish to parse the rest of the string? We would need to apply the

53−`parse(integer, -)` function again:

54+`parse` function again:

54```

55```

55−> parse(integer, parse(integer, "123"))

56+> parse(parse("123", integer), integer)

56(12, "3")

57(12, "3")

57−> parse(integer, parse(integer, parse(integer, "123")))

58+> parse(parse(parse("123", integer), integer), integer)

58(123, "")

59(123, "")

59```

60```

60So we see that we can parse strings larger than one character by stringing

61So we see that we can parse strings larger than one character by stringing

@@ -76,8 +77,6 @@ are quite a few), but ought to be

76sufficient to get started with parsing in Hoon and be equipped to discover the

77sufficient to get started with parsing in Hoon and be equipped to discover the

77remainder yourself.

78remainder yourself.

79−We encourage you to follow along with this section by inputting the relevant

80−commands in dojo as they are presented.

82## Basic types

81## Basic types

@@ -94,7 +93,10 @@ In this section we discuss the types most commonly used for Hoon parsers. In sho

94++ hair [p=@ud q=@ud]

93++ hair [p=@ud q=@ud]

95```

94```

97−A `hair` is a pair of `@ud` used to keep track of what has already been parsed.

96+A `hair` is a pair of `@ud` used to keep track of what has already been parsed

97+for stack tracing purposes. This allows the parser to reveal where the problem

98+is in case it hits something unexpected during parsing.

99+

98`p` represents the column and `q` represents the line.

100`p` represents the column and `q` represents the line.

101

100### `nail`

102### `nail`

@@ -111,20 +113,23 @@ the original input string (i.e. everything after the location indicated by the

111

113

112For example, if you wish to feed the entire `tape` `"abc"` into a parser, you

114For example, if you wish to feed the entire `tape` `"abc"` into a parser, you

113would pass it as the `nail` `[[1 1] "abc"]`. If the parser successfully parses the first

115would pass it as the `nail` `[[1 1] "abc"]`. If the parser successfully parses the first

114−character, the `nail` it returns will be `[[2 1] "bc"]` (though we note that

116+character, the `nail` it returns will be `[[1 2] "bc"]` (though we note that

115parser outputs are actually `edge`s which contain a `nail`, see the following).

117parser outputs are actually `edge`s which contain a `nail`, see the following).

118+The `nail` only matters for book-keeping reasons - it could be any value here

119+since it doesn't refer to a specific portion of the string being input, but only

120+what has theoretically already been parsed up to that point.

116

121

117### `edge`

122### `edge`

118

123

119```hoon

124```hoon

120++ edge [p=hair q=(unit [p=* q=nail])]

125++ edge [p=hair q=(unit [p=* q=nail])]

121```

126```

122−An `edge` is the output of a parser. If the parse succeeded, `p` is the location

127+An `edge` is the output of a parser. If parsing succeeded, `p` is the location

123−of the original input `tape `up to which the text has been parsed. If the parse

128+of the original input `tape `up to which the text has been parsed. If parsing

124−failed, `p` will be the first `hair` at which the parse failed.

129+failed, `p` will be the first `hair` at which parsing failed.

125

130

126−`q` may be `~`, indicating that the parse has failed .

131+`q` may be `~`, indicating that parsing has failed .

127−If the parse did not fail, `p.q` is the data structure that is the result of the

132+If parsing did not fail, `p.q` is the data structure that is the result of the

128parse up to this point, while `q.q` is the `nail` which contains the remainder

133parse up to this point, while `q.q` is the `nail` which contains the remainder

129of what is to be parsed. If `q` is not null, `p` and `p.q.q` are identical.

134of what is to be parsed. If `q` is not null, `p` and `p.q.q` are identical.

130

135

@@ -134,13 +139,7 @@ of what is to be parsed. If `q` is not null, `p` and `p.q.q` are identical.

134++ rule _|:($:nail $:edge)

139++ rule _|:($:nail $:edge)

135```

140```

136A `rule` is a gate which takes in a `nail` and returns an `edge` - in other

141A `rule` is a gate which takes in a `nail` and returns an `edge` - in other

137−words, a parser. Not totally sure why its written like this, though.

142+words, a parser.

138−

139−Maybe mention this from `/app/shoe.hoon` as well?

140−```hoon

141−|~(nail *(like [? command]))

142−```

143−

144

143

145## Parser builders

144## Parser builders

146

145

@@ -164,29 +163,31 @@ of the input `nail`.

164[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]

163[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]

165```

164```

166We note that `p.edg` is `[p=1 q=2]`, indicating that the next character to be

165We note that `p.edg` is `[p=1 q=2]`, indicating that the next character to be

167−parsed is in line 1, column 2. `q.edg` is not null, indicating that the parse

166+parsed is in line 1, column 2. `q.edg` is not null, indicating that parsing

168succeeded. `p.q.edg` is `'a'`, which is the result of the parse. `p.q.q.edg` is the same as `p.edg`, which is always the case for

167succeeded. `p.q.edg` is `'a'`, which is the result of the parse. `p.q.q.edg` is the same as `p.edg`, which is always the case for

169−`rule`s built using standard library functions when the parse succeeds. Lastly,

168+`rule`s built using standard library functions when parsing succeeds. Lastly,

170`q.q.edg` is `"bc"`, which is the part of the input `tape` that has yet to be parsed.

169`q.q.edg` is `"bc"`, which is the part of the input `tape` that has yet to be parsed.

171

170

172−Now let's see what happens when the parse fails.

171+Now let's see what happens when parsing fails.

173```

172```

174> =edg ((just 'b') [[1 1] "abc"])

173> =edg ((just 'b') [[1 1] "abc"])

175> edg

174> edg

176[p=[p=1 q=1] q=~]

175[p=[p=1 q=1] q=~]

177```

176```

178Now we have that `p.edg` is the same as the input `hair`, `[1 1]`, meaning the

177Now we have that `p.edg` is the same as the input `hair`, `[1 1]`, meaning the

179−parser has not advanced since the parse failed. `q.edg` is null, indicating that

178+parser has not advanced since parsing failed. `q.edg` is null, indicating that

180−the parse has failed.

179+parsing has failed.

181

180

182Later we will use [+star](#star) to string together a sequence of `+just`s in

181Later we will use [+star](#star) to string together a sequence of `+just`s in

183−order to parse multiple characters at once.

182+order to parse multiple characters at once. However, it should be noted that

183+`+jest`, which is covered next, ought to be used in favor of `+just`, which is

184+mostly included here for pedagogical reasons.

184

185

185### `+jest`

186### `+jest`

186

187

187`+jest` is a `rule` builder used to match a `cord`. It takes an input `cord` and

188`+jest` is a `rule` builder used to match a `cord`. It takes an input `cord` and

188produces a `rule` that attempts to match that `cord` against the beginning of

189produces a `rule` that attempts to match that `cord` against the beginning of

189−the `tape` in an input `nail`.

190+the input.

190

191

191Let's see what happens when we successfully parse the entire input `tape`.

192Let's see what happens when we successfully parse the entire input `tape`.

192```

193```

@@ -236,7 +237,7 @@ and returns a `rule`.

236

237

237`+cold` is a `rule` builder that takes in a constant noun we'll call `cus` and a

238`+cold` is a `rule` builder that takes in a constant noun we'll call `cus` and a

238`rule` we'll call `sef` and returns a `rule` identical to the `sef` except it

239`rule` we'll call `sef` and returns a `rule` identical to the `sef` except it

239−returns `cus` when it parses successfully.

240+replaces the parsing result with `cus`.

240

241

241Here we see that `p.q` of the `edge` returned by the `rule` created with `+cold`

242Here we see that `p.q` of the `edge` returned by the `rule` created with `+cold`

242is `%foo`.

243is `%foo`.

@@ -268,13 +269,15 @@ themselves. Thus we have several functions for exactly this purpose.

268

269

269These functions take in either a `tape` or an `@` (typically representing a

270These functions take in either a `tape` or an `@` (typically representing a

270`cord`) with a `rule` and attempts to parse the input with the `rule`. If the

271`cord`) with a `rule` and attempts to parse the input with the `rule`. If the

271−parse succeeds, it returns the result, otherwise it crashes with a stack trace or returns null.

272+parse succeeds, it returns the result. There are crashing and unitized versions

273+of each caller, corresponding to what happens when a parse fails.

272

274

273−Here we cover the basics, for complete information including additional examples see [4g: Parsing (Outside Caller)](@/docs/reference/library/4g.md).

275+For additional information including additional examples see [4g: Parsing (Outside Caller)](@/docs/reference/library/4g.md).

274

276

275### Parsing `tape`s

277### Parsing `tape`s

276

278

277−`+scan` takes in a `tape` and a `rule` and attempts to parse the `tape` with the `rule`.

279+`+scan` takes in a `tape` and a `rule` and attempts to parse the `tape` with the

280+`rule`.

278

281

279```

282```

280> (scan "hello" (jest 'hello'))

283> (scan "hello" (jest 'hello'))

@@ -284,8 +287,7 @@ Here we cover the basics, for complete information including additional examples

284'syntax-error'

287'syntax-error'

285```

288```

286

289

287−`+rust` works the same way as `+scan`, except it returns a `unit` of the result,

290+`+rust` is the unitized version of `+scan`.

288−which is null if the parse failed.

289

291

290```

292```

291> (rust "a" (just 'a'))

293> (rust "a" (just 'a'))

@@ -297,12 +299,38 @@ which is null if the parse failed.

297For the remainder of this tutorial we will make use of `+scan` so that we do not

299For the remainder of this tutorial we will make use of `+scan` so that we do not

298need to deal directly with `nail`s except where it is ilustrative to do so.

300need to deal directly with `nail`s except where it is ilustrative to do so.

299

301

302+

303+### Parsing atoms

304+

305+[Recall](@/docs/tutorials/hoon/lists.md) that `cord`s are atoms with the aura

306+`@t` and are typically used to represent strings internally as data, as atoms

307+are faster for the computer to work with than `tape`s, which are `list`s of

308+`@tD` atoms. `+rash` and `+rush` are for parsing atoms, with `+rash` being

309+analogous to `+scan` and `+rush` being analogous to `rust`. Under the hood, `+rash`

310+calls `+scan` after converting the input atom to a `tape`, and `+rush` does

311+similary for `+rust`.

312+

300## Parser modifiers

313## Parser modifiers

301

314

302The standard library provides a number of gates that take a `rule` and

315The standard library provides a number of gates that take a `rule` and

303produce a new modified `rule` according to some process. We call these _parser modifiers_. These are

316produce a new modified `rule` according to some process. We call these _parser modifiers_. These are

304documented among the [parser builders](@/docs/reference/library/4f.md).

317documented among the [parser builders](@/docs/reference/library/4f.md).

305

318

319+

320+### `+ifix`

321+

322+`+ifix` modifies a `rule` so that it matches that `rule` only when it is

323+surrounded on both sides by text that matches a pair of `rule`s, which is discarded.

324+

325+```

326+> (scan "(42)" (ifix [lit rit] (jest '42')))

327+'42'

328+```

329+`+lit` and `+rit` are shorthand for `(just '(')` and `(just ')')`, respectively. All

330+ASCII glyphs have counterparts of this sort, documented

331+[here](@/docs/reference/library/4h.md).

332+

333+

306### `+star` {#star}

334### `+star` {#star}

307

335

308`+star` is used to apply a `rule` repeatedly. Recall that `+just` only parses

336`+star` is used to apply a `rule` repeatedly. Recall that `+just` only parses

@@ -325,20 +353,6 @@ and we note that the parsing ceases when it fails.

325[p=[p=1 q=4] q=[~ [p=[i='a' t=<|a a|>] q=[p=[p=1 q=4] q="b"]]]]

353[p=[p=1 q=4] q=[~ [p=[i='a' t=<|a a|>] q=[p=[p=1 q=4] q="b"]]]]

326```

354```

327

355

328−### `+ifix`

329−

330−`+ifix` modifies a `rule` so that it matches that `rule` only when it is

331−surrounded on both sides by text that matches a pair of `rule`s, which is discarded.

332−

333−```

334−> (scan "(42)" (ifix [lit rit] (jest '42')))

335−'42'

336−```

337−`+lit` and `+rit` are shorthand for `(just '(')` and `(just ')')`, respectively. All

338−ASCII glyphs have counterparts of this sort, documented

339−[here](@/docs/reference/library/4h.md).

340−

341−

342## Parser combinators

356## Parser combinators

343

357

344Building complex parsers from simpler parsers is accomplished in Hoon with the

358Building complex parsers from simpler parsers is accomplished in Hoon with the

@@ -349,7 +363,7 @@ combinators, then we examine more closely how `;~` is used to chain them togethe

349

363

350The syntax to combine `rule`s is

364The syntax to combine `rule`s is

351```hoon

365```hoon

352−;~(combinator [list rule])

366+;~(combinator rule1 rule2 ... ruleN)

353```

367```

354The `rule`s are composed together using the combinator as an

368The `rule`s are composed together using the combinator as an

355intermediate function, which takes product of a `rule` (an `edge`) and a `rule` and turns

369intermediate function, which takes product of a `rule` (an `edge`) and a `rule` and turns

@@ -433,15 +447,7 @@ is more than one `rule`.

433syntax error

447syntax error

434```

448```

435

449

436−## Parsing atoms

437

450

438−[Recall](@/docs/tutorials/hoon/lists.md) that `cord`s are atoms with the aura

439−`@t` and are typically used to represent strings internally as data, as atoms

440−are faster for the computer to work with than `tape`s, which are `list`s of

441−`@tD` atoms. `+rash` and `+rush` are for parsing atoms, with `+rash` being

442−analogous to `+scan` and `+rush` being analogous to `rust`. Under the hood, `+rash`

443−calls `+scan` after converting the input atom to a `tape`, and `+rush` does

444−similary for `+rust`.

445

451

446## Parsing numbers

452## Parsing numbers

447

453

@@ -462,7 +468,7 @@ Naively attempting to write a recursive `rule`, i.e. like

462```

468```

463> |-(;~(pose den ;~(pose $ (easy ~))))

469> |-(;~(pose den ;~(pose $ (easy ~))))

464```

470```

465−results in an error, as the compiler will attempt to build an infinite function.

471+results in an error.

466```

472```

467-find.,.+6

473-find.,.+6

468-find.,.+6

474-find.,.+6

@@ -475,12 +481,17 @@ as the bunt value of the type that the `rule` produces, as well as a gate that

475accepts a `rule`. `+knee` produces a `rule` that implements any recursive calls

481accepts a `rule`. `+knee` produces a `rule` that implements any recursive calls

476in the `rule` in a manner acceptable to the compiler. Thus the preferred manner

482in the `rule` in a manner acceptable to the compiler. Thus the preferred manner

477to write the above `rule` is as follows:

483to write the above `rule` is as follows:

484+

478```hoon

485```hoon

486+++ pars

487+ |-

488+ ;~ plug prn

479−|-(;~(plug prn ;~(pose (knee *tape |.(^$)) (easy ~))))

489+ ;~ pose knee *tape

490+ |. ^$ (easy ~)

480```

491```

481

492

482You may want to utilize the `~+` rune when writing recursive parsers to cache

493You may want to utilize the `~+` rune when writing recursive parsers to cache

483−the results as you go to improve performance. In [parsing arithmetic

494+the parser to improve performance. In [parsing arithmetic

484expressions](#parsing-arithmetic-expressions) we will be writing a recursive

495expressions](#parsing-arithmetic-expressions) we will be writing a recursive

485parser making use of `+knee` and `~+` throughout.

496parser making use of `+knee` and `~+` throughout.

486

497

@@ -492,7 +503,7 @@ arithmetic expressions in Hoon. That is, we will make a `rule` that takes in

492`tape`s of the form `"(2+3)*4"` and returns `20` as a `@ud`.

503`tape`s of the form `"(2+3)*4"` and returns `20` as a `@ud`.

493

504

494We call a `tape` consisting of some consistent arithmetic string of numbers,

505We call a `tape` consisting of some consistent arithmetic string of numbers,

495−`+,*`, `(`, and `)` an _expression_. We wish to build a `rule` that takes in an

506+`+`, `*`, `(`, and `)` an _expression_. We wish to build a `rule` that takes in an

496expression and returns the result of the arithmetic computation described by the

507expression and returns the result of the arithmetic computation described by the

497expression as a `@ud`.

508expression as a `@ud`.

498

509

@@ -505,31 +516,28 @@ First lets look at the code we're going to use, and then dive into explaining

505it. If you'd like to follow along, save the following as `expr-parse.hoon` in

516it. If you'd like to follow along, save the following as `expr-parse.hoon` in

506your `gen/` folder.

517your `gen/` folder.

507```hoon

518```hoon

508−:: This generator parses arithmetic expressions

519+:: expr-parse: parse arithmetic expressions

509::

520::

510−|= [xprs=tape]

521+|= xprs=tape

511−=<

522+|^ (scan xprs tape)

512−(print xprs)

513−|%

514−++ int @ud

515−++ print

516− |= [message=tape]

517− (scan message expr)

518++ factor

523++ factor

519− %+ knee *int

524+ %+ knee @ud

525+ |. ~+

520− |. ~+ ;~ pose

526+ ;~ pose

521 dem

527 dem

522 (ifix [lit rit] expr)

528 (ifix [lit rit] expr)

523 ==

529 ==

524++ turm

530++ turm

525− %+ knee *int

531+ %+ knee @ud

532+ |. ~+

526− |. ~+ ;~ pose

533+ ;~ pose

527 ((slug mul) tar ;~(pose factor turm))

534 ((slug mul) tar ;~(pose factor turm))

528 factor

535 factor

529 ==

536 ==

530++ expr

537++ expr

531− %+ knee *int

538+ %+ knee @ud

539+ |. ~+

532− |. ~+ ;~ pose

540+ ;~ pose

533 ((slug add) lus ;~(pose turm expr))

541 ((slug add) lus ;~(pose turm expr))

534 turm

542 turm

535 ==

543 ==

@@ -545,7 +553,7 @@ for expressions.

545

553

546```hoon

554```hoon

547++ factor

555++ factor

548− %+ knee *int

556+ %+ knee @ud

549 |. ~+ ;~ pose

557 |. ~+ ;~ pose

550 dem

558 dem

551 (ifix [lit rit] expr)

559 (ifix [lit rit] expr)

@@ -562,26 +570,24 @@ with `+pose`, which says to try each rule in succession until one of them works.

562

570

563Since expressions ultimately reduce to factors, we are actually building a

571Since expressions ultimately reduce to factors, we are actually building a

564recursive rule. Thus we need to make use of `+knee`. The first argument for

572recursive rule. Thus we need to make use of `+knee`. The first argument for

565−`+knee` is `*int`, since our final answer should be a `@ud`, to which we have

573+`+knee` is `@ud`, since our final answer should be a `@ud`.

566−assigned the macro `int`.

567

574

568Then follows the definition of the gate utilized by `+knee`:

575Then follows the definition of the gate utilized by `+knee`:

569```hoon

576```hoon

577+ |. ~+

570− |. ~+ ;~ pose

578+ ;~ pose

571 dem

579 dem

572 (ifix [lit rit] expr)

580 (ifix [lit rit] expr)

573 ==

581 ==

574```

582```

575−`|.` is used instead of `|-` since... `~+` is used to cache results as we go -

583+`~+` is used to cache the parser, so that it does not need to be computed over

576−there is no point in computing `2+2` more than once. Then follows the `rule` we

577−described above.

584+and over again. Then it follows the `rule` we described above.

578

585

579#### Parsing expressions

586#### Parsing expressions

580

587

581An _expression_ is either a term plus an expression or a term.

588An _expression_ is either a term plus an expression or a term.

582

589

583−In the case of a

584−term plus an expression, we actually must compute what that equals. Thus we will

590+In the case of a term plus an expression, we actually must compute what that equals. Thus we will

585make use of [`+slug`](@/docs/reference/library/4f.md#slug), which parses a

591make use of [`+slug`](@/docs/reference/library/4f.md#slug), which parses a

586delimited list into `tape`s separated by a given delimiter and then composes

592delimited list into `tape`s separated by a given delimiter and then composes

587them by folding with a binary gate. In this case, our delimiter is `+` and our

593them by folding with a binary gate. In this case, our delimiter is `+` and our

@@ -590,15 +596,15 @@ and expressions separated by luses, parse each term and expression until they

590reduce to a `@ud`, and then add them together. This is accomplished with the

596reduce to a `@ud`, and then add them together. This is accomplished with the

591rule `((slug add) lus ;~(pose turm expr))`.

597rule `((slug add) lus ;~(pose turm expr))`.

592

598

593−If the above `rule` does not parse the expression, it must be a `turm`, so we

599+If the above `rule` does not parse the expression, it must be a `turm`, so the

594−pass the `tape` to `+turm` to be evaluated. Again we use `;~` and `pose` to

600+`tape` is automatically passed to `+turm` to be evaluated. Again we use `;~` and `pose` to

595accomplish this:

601accomplish this:

596

602

597```hoon

603```hoon

598;~ pose

604;~ pose

599− ((slug add) lus ;~(pose turm expr))

605+ ((slug add) lus ;~(pose turm expr))

600− turm

606+ turm

601− ==

607+==

602```

608```

603

609

604The rest of the `+expr` arm is structured just like how `+factor` is, and for

610The rest of the `+expr` arm is structured just like how `+factor` is, and for

@@ -609,7 +615,7 @@ the same reasons.

609

615

610A _term_ is either a factor times a term or a factor. This is handled similarly

616A _term_ is either a factor times a term or a factor. This is handled similarly

611for expressions, we just need to swap `lus` for `tar`, `add` for `mul`, and

617for expressions, we just need to swap `lus` for `tar`, `add` for `mul`, and

612−`;~(pose factor turm)` instead of `;(pose turm expr)`.

618+`;~(pose factor turm)` instead of `;~(pose turm expr)`.

613

619

614```hoon

620```hoon

615++ expr

621++ expr