189164a · urbit.org · git.iko.soy

git.iko.soy/urbit.org

on master help sign in

Merge pull request #1306 from urbit/parsing-edits

189164atinnus-napbus5 years ago

content/docs/hoon/guides/parsing.md+91−59

compare with master

unifiedsplit

@@ -70,7 +70,7 @@ Combinators come in a few shapes and sizes, and typical operations they may

70perform would be to repeat the same parsing operation until the string is

71consumed, try a few different parsing operations until one of them works,

72or perform a sequence of parsing operations. We will see how all of this is done

73−with Hoon in the [Parsing in Hoon](#parsing-in-hoon) section.

73+with Hoon in the next section.

75# Parsing in Hoon

@@ -157,7 +157,7 @@ Glyphs)](/docs/hoon/reference/stdlib/4h), [4i: Parsing (Useful

157Idioms)](/docs/hoon/reference/stdlib/4i), [4j: Parsing (Bases and Base

158Digits)](/docs/hoon/reference/stdlib/4j), [4l: Atom Parsing](/docs/hoon/reference/stdlib/4l).

159

160−### `+just`

160+### [`+just`](/docs/hoon/reference/stdlib/4f/#just)

161

162The most basic rule builder, `+just` takes in a single `char` and produces a

163`rule` that attempts to match that `char` to the first character in the `tape`

@@ -190,7 +190,7 @@ parsing has failed.

190Later we will use [+star](#star) to string together a sequence of `+just`s in

191order to parse multiple characters at once.

192

193−### `+jest`

193+### [`+jest`](/docs/hoon/reference/stdlib/4f/#jest)

194

195`+jest` is a `rule` builder used to match a `cord`. It takes an input `cord` and

196produces a `rule` that attempts to match that `cord` against the beginning of

@@ -236,7 +236,7 @@ beginning the parse failed. We will see in [parser

236combinators](#parser-combinators) how to modify this `rule` so that it

237finds `bc` successfully.

238

239−### `+shim`

239+### [`+shim`](/docs/hoon/reference/stdlib/4f/#shim)

240

241`+shim` is used to parse characters within a given range. It takes in two atoms

242and returns a `rule`.

@@ -246,7 +246,17 @@ and returns a `rule`.

246[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]

247```

248

249−### `+cold`

249+### [`+next`](/docs/hoon/reference/stdlib/4f/#next)

250+

251+`+next` is a simple `rule` that takes in the next character and returns it as the

252+parsing result.

253+

254+```

255+> (next [[1 1] "abc"])

256+[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]

257+```

258+

259+### [`+cold`](/docs/hoon/reference/stdlib/4f/#cold)

250

260

251`+cold` is a `rule` builder that takes in a constant noun we'll call `cus` and a

261`+cold` is a `rule` builder that takes in a constant noun we'll call `cus` and a

252`rule` we'll call `sef`. It returns a `rule` identical to the `sef` except it

262`rule` we'll call `sef`. It returns a `rule` identical to the `sef` except it

@@ -264,7 +274,7 @@ One common scenario where `+cold` sees play is when writing [command line

264interface (CLI) apps](/docs/hoon/guides/cli-tutorial). We usher the

274interface (CLI) apps](/docs/hoon/guides/cli-tutorial). We usher the

265reader there to find an example where `+cold` is used.

275reader there to find an example where `+cold` is used.

266

276

267−### `+knee`

277+### [`+knee`](/docs/hoon/reference/stdlib/4f/#knee)

268

278

269Another important function in the parser builder library is `+knee`, used for building

279Another important function in the parser builder library is `+knee`, used for building

270recursive parsers. We delay discussion of `+knee` to the

280recursive parsers. We delay discussion of `+knee` to the

@@ -284,11 +294,11 @@ alongside a `rule`, and attempt to parse the input with the `rule`. If the

284parse succeeds, it returns the result. There are crashing and unitized versions

294parse succeeds, it returns the result. There are crashing and unitized versions

285of each caller, corresponding to what happens when a parse fails.

295of each caller, corresponding to what happens when a parse fails.

286

296

287−For additional information including additional examples see [4g: Parsing (Outside Caller)](/docs/hoon/reference/stdlib/4g).

297+For additional information including examples see [4g: Parsing (Outside Caller)](/docs/hoon/reference/stdlib/4g).

288

298

289### Parsing `tape`s

299### Parsing `tape`s

290

300

291−`+scan` takes in a `tape` and a `rule` and attempts to parse the `tape` with the

301+[`+scan`](/docs/hoon/reference/stdlib/4g/#scan) takes in a `tape` and a `rule` and attempts to parse the `tape` with the

292`rule`.

302`rule`.

293

303

294```

304```

@@ -299,7 +309,7 @@ For additional information including additional examples see [4g: Parsing (Outsi

299'syntax-error'

309'syntax-error'

300```

310```

301

311

302−`+rust` is the unitized version of `+scan`.

312+[`+rust`](/docs/hoon/reference/stdlib/4g/#rust) is the unitized version of `+scan`.

303

313

304```

314```

305> (rust "a" (just 'a'))

315> (rust "a" (just 'a'))

@@ -309,25 +319,26 @@ For additional information including additional examples see [4g: Parsing (Outsi

309```

319```

310

320

311For the remainder of this tutorial we will make use of `+scan` so that we do not

321For the remainder of this tutorial we will make use of `+scan` so that we do not

312−need to deal directly with `nail`s except where it is ilustrative to do so.

322+need to deal directly with `nail`s except where it is illustrative to do so.

313

323

314### Parsing atoms

324### Parsing atoms

315

325

316[Recall](/docs/hoon/hoon-school/lists) that `cord`s are atoms with the aura

326[Recall](/docs/hoon/hoon-school/lists) that `cord`s are atoms with the aura

317`@t` and are typically used to represent strings internally as data, as atoms

327`@t` and are typically used to represent strings internally as data, as atoms

318are faster for the computer to work with than `tape`s, which are `list`s of

328are faster for the computer to work with than `tape`s, which are `list`s of

319−`@tD` atoms. `+rash` and `+rush` are for parsing atoms, with `+rash` being

329+`@tD` atoms. [`+rash`](/docs/hoon/reference/stdlib/4g/#rash) and [`+rush`](/docs/hoon/reference/stdlib/4g/#rush) are for parsing atoms, with `+rash` being

320−analogous to `+scan` and `+rush` being analogous to `rust`. Under the hood, `+rash`

330+analogous to `+scan` and `+rush` being analogous to `+rust`. Under the hood, `+rash`

321calls `+scan` after converting the input atom to a `tape`, and `+rush` does

331calls `+scan` after converting the input atom to a `tape`, and `+rush` does

322similary for `+rust`.

332similary for `+rust`.

323

333

324## Parser modifiers

334## Parser modifiers

325

335

326−The standard library provides a number of gates that take a `rule` and

336+The standard library provides a number of gates that take a `rule` and produce a

327−produce a new modified `rule` according to some process. We call these _parser modifiers_. These are

337+new modified `rule` according to some process. We call these _parser modifiers_.

338+These are documented among the [parser

328−documented among the [parser builders](/docs/hoon/reference/stdlib/4f).

339+builders](/docs/hoon/reference/stdlib/4f).

329

340

330−### `+ifix`

341+### [`+ifix`](/docs/hoon/reference/stdlib/4f/#ifix)

331

342

332`+ifix` modifies a `rule` so that it matches that `rule` only when it is

343`+ifix` modifies a `rule` so that it matches that `rule` only when it is

333surrounded on both sides by text that matches a pair of `rule`s, which is discarded.

344surrounded on both sides by text that matches a pair of `rule`s, which is discarded.

@@ -341,7 +352,7 @@ surrounded on both sides by text that matches a pair of `rule`s, which is discar

341ASCII glyphs have counterparts of this sort, documented

352ASCII glyphs have counterparts of this sort, documented

342[here](/docs/hoon/reference/stdlib/4h).

353[here](/docs/hoon/reference/stdlib/4h).

343

354

344−### `+star` {#star}

355+### [`+star`](/docs/hoon/reference/stdlib/4f/#star) {#star}

345

356

346`+star` is used to apply a `rule` repeatedly. Recall that `+just` only parses

357`+star` is used to apply a `rule` repeatedly. Recall that `+just` only parses

347the first character in the input `tape.`

358the first character in the input `tape.`

@@ -365,11 +376,32 @@ and we note that the parsing ceases when it fails.

365[p=[p=1 q=4] q=[~ [p=[i='a' t=<|a a|>] q=[p=[p=1 q=4] q="b"]]]]

376[p=[p=1 q=4] q=[~ [p=[i='a' t=<|a a|>] q=[p=[p=1 q=4] q="b"]]]]

366```

377```

367

378

379+We can combine `+star` with `+next` to just return the whole input:

380+

381+```

382+> ((star next) [[1 1] "aaabc"])

383+[p=[p=1 q=6] q=[~ [p=[i='a' t=<|a a b c|>] q=[p=[p=1 q=6] q=""]]]]

384+```

385+

386+### [`+cook`](/docs/hoon/reference/stdlib/4f/#cook)

387+

388+`+cook` takes a `rule` and a gate and produces a modified version of the `rule`

389+that passes the result of a successful parse through the given gate.

390+

391+Let's modify the rule `(just 'a')` so that it when it successfully parses `a`,

392+it returns the following letter `b` as the result.

393+

394+```

395+((cook |=(a=@ `@t`+(a)) (just 'a')) [[1 1] "abc"])

396+[p=[p=1 q=2] q=[~ u=[p='b' q=[p=[p=1 q=2] q="bc"]]]]

397+```

398+

399+

368## Parser combinators

400## Parser combinators

369

401

370Building complex parsers from simpler parsers is accomplished in Hoon with the

402Building complex parsers from simpler parsers is accomplished in Hoon with the

371use of two tools: the monadic applicator rune

403use of two tools: the monadic applicator rune

372−[`;~`](/docs/hoon/reference/rune/mic#micsig) and [parsing

404+[`;~`](/docs/hoon/reference/rune/mic/#-micsig) and [parsing

373combinators](/docs/hoon/reference/stdlib/4e). First we introduce a few

405combinators](/docs/hoon/reference/stdlib/4e). First we introduce a few

374combinators, then we examine more closely how `;~` is used to chain them together.

406combinators, then we examine more closely how `;~` is used to chain them together.

375

407

@@ -380,11 +412,11 @@ The syntax to combine `rule`s is

380```

412```

381

413

382The `rule`s are composed together using the combinator as an

414The `rule`s are composed together using the combinator as an

383−intermediate function, which takes product of a `rule` (an `edge`) and a `rule` and turns

415+intermediate function, which takes the product of a `rule` (an `edge`) and a `rule` and turns

384it into a sample (a `nail`) for the next `rule` to handle. We elaborate on this

416it into a sample (a `nail`) for the next `rule` to handle. We elaborate on this

385behavior [below](#micsig).

417behavior [below](#micsig).

386

418

387−### `+plug`

419+### [`+plug`](/docs/hoon/reference/stdlib/4e/#plug)

388

420

389`+plug` simply takes the `nail` in the `edge` produced by one rule and passes it

421`+plug` simply takes the `nail` in the `edge` produced by one rule and passes it

390to the next `rule`, forming a cell of the results as it proceeds.

422to the next `rule`, forming a cell of the results as it proceeds.

@@ -394,7 +426,7 @@ to the next `rule`, forming a cell of the results as it proceeds.

394['star' 'ship']

426['star' 'ship']

395```

427```

396

428

397−### `+pose`

429+### [`+pose`](/docs/hoon/reference/stdlib/4e/#pose)

398

430

399`+pose` tries each `rule` you hand it successively until it finds one that

431`+pose` tries each `rule` you hand it successively until it finds one that

400works.

432works.

@@ -406,7 +438,7 @@ works.

406'b'

438'b'

407```

439```

408

440

409−### `+glue`

441+### [`+glue`](/docs/hoon/reference/stdlib/4e/#glue)

410

442

411`+glue` parses a delimiter in between each `rule` and forms a cell of the

443`+glue` parses a delimiter in between each `rule` and forms a cell of the

412results of each `rule`.

444results of each `rule`.

@@ -417,15 +449,17 @@ results of each `rule`.

417> (scan "a,b,a" ;~((glue com) (just 'a') (just 'b')))

449> (scan "a,b,a" ;~((glue com) (just 'a') (just 'b')))

418{1 4}

450{1 4}

419syntax error

451syntax error

452+> (scan "a,b,a" ;~((glue com) (just 'a') (just 'b') (just 'a')))

453+['a' 'b' 'a']

420```

454```

421

455

422−### `;~` {#micsig}

456+### [`;~`](/docs/hoon/reference/rune/mic/#-micsig) {#micsig}

423

457

424Understanding the rune `;~` is essential to building parsers with Hoon. Let's

458Understanding the rune `;~` is essential to building parsers with Hoon. Let's

425take this opportunity to think about it carefully.

459take this opportunity to think about it carefully.

426

460

427−The `rule` created by `;~(combinator [list rule])` may be understood

461+The `rule` created by `;~(combinator (list rule))` may be understood

428−inductively. To do this, let's consider the base case where our `[list rule]` has only a

462+inductively. To do this, let's consider the base case where our `(list rule)` has only a

429single entry.

463single entry.

430

464

431```

465```

@@ -433,9 +467,9 @@ single entry.

433'star'

467'star'

434```

468```

435

469

436−Our output is identical to that given by `scan "star" (jest 'star')`. This is to

470+Our output is identical to that given by `(scan "star" (jest 'star'))`. This is to

437be expected. The combinator `+plug` is specifically used for chaining together

471be expected. The combinator `+plug` is specifically used for chaining together

438−`rule`s in the `[list rule]`, but if there is only one `rule`, there is nothing

472+`rule`s in the `(list rule)`, but if there is only one `rule`, there is nothing

439to chain. Thus, swapping out `+plug` for another combinator makes no difference here:

473to chain. Thus, swapping out `+plug` for another combinator makes no difference here:

440

474

441```

475```

@@ -445,7 +479,7 @@ to chain. Thus, swapping out `+plug` for another combinator makes no difference

445'star'

479'star'

446```

480```

447

481

448−`;~` and the combinator only begin to play a role once the `[list rule]` has at

482+`;~` and the combinator only begin to play a role once the `(list rule)` has at

449least two elements. So let's look at an example done with `+plug`, the simplest

483least two elements. So let's look at an example done with `+plug`, the simplest

450combinator.

484combinator.

451

485

@@ -470,8 +504,9 @@ syntax error

470## Parsing numbers

504## Parsing numbers

471

505

472Functions for parsing numbers are documented in [4j: Parsing (Bases and Base

506Functions for parsing numbers are documented in [4j: Parsing (Bases and Base

473−Digits)](/docs/hoon/reference/stdlib/4j). In particular, `dem` is a `rule`

507+Digits)](/docs/hoon/reference/stdlib/4j). In particular,

508+[`dem`](/docs/hoon/reference/stdlib/4i/#dem) is a `rule` for parsing decimal

474−for parsing decimal numbers.

509+numbers.

475

510

476```

511```

477> (scan "42" dem)

512> (scan "42" dem)

@@ -485,18 +520,23 @@ for parsing decimal numbers.

485Naively attempting to write a recursive `rule`, i.e. like

520Naively attempting to write a recursive `rule`, i.e. like

486

521

487```

522```

488−> |-(;~(pose den ;~(pose $ (easy ~))))

523+> |-(;~(plug prn ;~(pose $ (easy ~))))

489```

524```

490

525

491−results in an error.

526+results in an error:

492−

493```

527```

494-find.,.+6

528-find.,.+6

495-find.,.+6

529-find.,.+6

496rest-loop

530rest-loop

497```

531```

498

532

533+Here, [`+prn`](/docs/hoon/reference/stdlib/4i/#prn) is a `rule` used

534+to parse any printable character, and

535+[`+easy`](/docs/hoon/reference/stdlib/4f/#easy) is a `rule` that always returns a

536+constant (`~` in this case) regardless of the input.

537+

499−Thus some special sauce is required, the `+knee` function.

538+Thus some special sauce is required, the

539+[`+knee`](/docs/hoon/reference/stdlib/4f/#knee) function.

500

540

501`+knee` takes in a noun that is the default value of the parser, typically given

541`+knee` takes in a noun that is the default value of the parser, typically given

502as the bunt value of the type that the `rule` produces, as well as a gate that

542as the bunt value of the type that the `rule` produces, as well as a gate that

@@ -505,19 +545,11 @@ in the `rule` in a manner acceptable to the compiler. Thus the preferred manner

505to write the above `rule` is as follows:

545to write the above `rule` is as follows:

506

546

507```hoon

547```hoon

508−++ pars

509− |-

510− ;~ plug prn

511− ;~ pose

512− (knee *tape |.(^$))

548+|-(;~(plug prn ;~(pose (knee *tape |.(^$)) (easy ~))))

513− (easy ~)

514− ==

515− ==

516```

549```

517

550

518You may want to utilize the `~+` rune when writing recursive parsers to cache

551You may want to utilize the `~+` rune when writing recursive parsers to cache

519−the parser to improve performance. In [parsing arithmetic

552+the parser to improve performance. In the following section, we will be writing a recursive

520−expressions](#parsing-arithmetic-expressions) we will be writing a recursive

521parser making use of `+knee` and `~+` throughout.

553parser making use of `+knee` and `~+` throughout.

522

554

523# Parsing arithmetic expressions

555# Parsing arithmetic expressions

@@ -536,15 +568,15 @@ To build a parser it is a helpful exercise to first describe its

536formal mathematical definition, but we will manage to get by here describing the grammar

568formal mathematical definition, but we will manage to get by here describing the grammar

537for arithmetic expressions informally.

569for arithmetic expressions informally.

538

570

539−First lets look at the code we're going to use, and then dive into explaining

571+First let's look at the code we're going to use, and then dive into explaining

540it. If you'd like to follow along, save the following as `expr-parse.hoon` in

572it. If you'd like to follow along, save the following as `expr-parse.hoon` in

541your `gen/` folder.

573your `gen/` folder.

542

574

543```hoon

575```hoon

544:: expr-parse: parse arithmetic expressions

576:: expr-parse: parse arithmetic expressions

545::

577::

546−|= xprs=tape

578+|= math=tape

547−|^ (scan xprs expr)

579+|^ (scan math expr)

548++ factor

580++ factor

549 %+ knee *@ud

581 %+ knee *@ud

550 |. ~+

582 |. ~+

@@ -552,19 +584,19 @@ your `gen/` folder.

552 dem

584 dem

553 (ifix [pal par] expr)

585 (ifix [pal par] expr)

554 ==

586 ==

555−++ turm

587+++ term

556 %+ knee *@ud

588 %+ knee *@ud

557 |. ~+

589 |. ~+

558 ;~ pose

590 ;~ pose

559− ((slug mul) tar ;~(pose factor turm))

591+ ((slug mul) tar ;~(pose factor term))

560 factor

592 factor

561 ==

593 ==

562++ expr

594++ expr

563 %+ knee *@ud

595 %+ knee *@ud

564 |. ~+

596 |. ~+

565 ;~ pose

597 ;~ pose

566− ((slug add) lus ;~(pose turm expr))

598+ ((slug add) lus ;~(pose term expr))

567− turm

599+ term

568 ==

600 ==

569--

601--

570```

602```

@@ -627,16 +659,16 @@ them by folding with a binary gate. In this case, our delimiter is `+` and our

627binary gate is `+add`. That is to say, we will split the input string into terms

659binary gate is `+add`. That is to say, we will split the input string into terms

628and expressions separated by luses, parse each term and expression until they

660and expressions separated by luses, parse each term and expression until they

629reduce to a `@ud`, and then add them together. This is accomplished with the

661reduce to a `@ud`, and then add them together. This is accomplished with the

630−rule `((slug add) lus ;~(pose turm expr))`.

662+`rule` `((slug add) lus ;~(pose term expr))`.

631

663

632−If the above `rule` does not parse the expression, it must be a `turm`, so the

664+If the above `rule` does not parse the expression, it must be a `term`, so the

633−`tape` is automatically passed to `+turm` to be evaluated. Again we use `;~` and `pose` to

665+`tape` is automatically passed to `+term` to be evaluated. Again we use `;~` and `pose` to

634accomplish this:

666accomplish this:

635

667

636```hoon

668```hoon

637;~ pose

669;~ pose

638− ((slug add) lus ;~(pose turm expr))

670+ ((slug add) lus ;~(pose term expr))

639− turm

671+ term

640==

672==

641```

673```

642

674

@@ -647,15 +679,15 @@ the same reasons.

647

679

648A _term_ is either a factor times a term or a factor. This is handled similarly

680A _term_ is either a factor times a term or a factor. This is handled similarly

649for expressions, we just need to swap `lus` for `tar`, `add` for `mul`, and

681for expressions, we just need to swap `lus` for `tar`, `add` for `mul`, and

650−`;~(pose factor turm)` instead of `;~(pose turm expr)`.

682+`;~(pose factor term)` instead of `;~(pose term expr)`.

651

683

652```hoon

684```hoon

653++ expr

685++ expr

654 %+ knee *@ud

686 %+ knee *@ud

655 |. ~+

687 |. ~+

656 ;~ pose

688 ;~ pose

657− ((slug add) lus ;~(pose turm expr))

689+ ((slug add) lus ;~(pose term expr))

658− turm

690+ term

659 ==

691 ==

660```

692```

661

693