3336849 · urbit.org · git.iko.soy

git.iko.soy/urbit.org

on master help sign in

editing pass, cold and ;~`

3336849JONATHAN PAPROCKI6 years ago

tutorials/arvo/parsing.md+76−27

compare with master

unifiedsplit

@@ -15,26 +15,26 @@ how it is implemented in Hoon.

16A program which takes a raw sequence of characters as an input and produces a data

17structure as an output is known as a _parser_. The data structure produced

18−depends on the use case, but often it may be represented as a tree, and the

18+depends on the use case, but often it may be represented as a tree and the

19output is thought of as a structural representation of the input. Parsers are ubiquitous in

20−computing, commonly used in applications such as reading files, compiling

20+computing, commonly used for to perform tasks such as reading files, compiling

21source code, or understanding commands input in a command line interface.

23Parsing a string is rarely done all at once. Instead, it is usually done

24character-by-character, and the return contains the data structure representing

25what has been parsed thus far as well as the remainder of the string to be

26−parsed. They also need to be able to fail in case the input is improperly formed.

26+parsed. They also need to be able to fail in case the input is improperly

27+formed. We will see each of these standard practices implemented in Hoon below.

28## Functional parsers

29## Functional parsers

30−How parsers are built varies greatly depending on what sort of programming

31+How parsers are built varies substantially depending on what sort of programming

31language it is written in. As Hoon is a functional programming language, we will

32language it is written in. As Hoon is a functional programming language, we will

32be focused on understanding _functional parsers_, also known as _combinator

33be focused on understanding _functional parsers_, also known as _combinator

33parsers_.

34parsers_.

35−Functional parsers are built piece by piece from simple irreducible components

36+Complex functional parsers are built piece by piece from simpler parsers

36−that are plugged into one another in various ways to form more complex

37+that are plugged into one another in various ways to perform the desired task.

37−parsers.

39The basic building blocks, or primitives, are parsers that read only a

40single character. There are frequently a few types of possible input characters,

@@ -71,11 +71,13 @@ with Hoon in the [Parsing in Hoon](#parsing-in-hoon) section.

72In this section we will cover the basic types, parser functions, and parser

73combinators that are utilized for parsing in Hoon. This is not a complete guide

74−to every parsing-related functionality in the standard library, but ought to be

74+to every parsing-related functionality in the standard library (of which there

75+are quite a few), but ought to be

75sufficient to get started with parsing in Hoon and be equipped to discover the

76sufficient to get started with parsing in Hoon and be equipped to discover the

76remainder yourself.

77remainder yourself.

78−We encourage you to follow along with this section by inputting the commands in dojo.

79+We encourage you to follow along with this section by inputting the relevant

80+commands in dojo as they are presented.

80## Basic types

82## Basic types

@@ -177,7 +179,8 @@ Now we have that `p.edg` is the same as the input `hair`, `[1 1]`, meaning the

177parser has not advanced since the parse failed. `q.edg` is null, indicating that

179parser has not advanced since the parse failed. `q.edg` is null, indicating that

178the parse has failed.

180the parse has failed.

179

181

180−Later we will use [+star](#star) to string together a sequence of `+just`s in order to parse an entire string.

182+Later we will use [+star](#star) to string together a sequence of `+just`s in

183+order to parse multiple characters at once.

181

184

182### `+jest`

185### `+jest`

183

186

@@ -229,11 +232,30 @@ and returns a `rule`.

229[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]

232[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]

230```

233```

231

234

235+### `+cold`

236+

237+`+cold` is a `rule` builder that takes in a constant noun we'll call `cus` and a

238+`rule` we'll call `sef` and returns a `rule` identical to the `sef` except it

239+returns `cus` when it parses successfully.

240+

241+Here we see that `p.q` of the `edge` returned by the `rule` created with `+cold`

242+is `%foo`.

243+```

244+> ((cold %foo (just 'a')) [[1 1] "abc"])

245+[p=[p=1 q=2] q=[~ u=[p=%foo q=[p=[p=1 q=2] q="bc"]]]]

246+```

247+

248+One common scenario where `+cold` sees play is when writing [command line

249+interface (CLI) apps](@/docs/tutorials/arvo/cli-tutorial.md). We usher the

250+reader there to find an example where `+cold` is used.

251+

252+

232### `+knee`

253### `+knee`

233

254

234Another important function in the parser builder library is `+knee`, used for building

255Another important function in the parser builder library is `+knee`, used for building

235recursive parsers. We delay discussion of `+knee` to the

256recursive parsers. We delay discussion of `+knee` to the

236−[section below](#recursive-parsers) as more context is needed to explain it properly.

257+[section below](#recursive-parsers) as more context is needed to explain it

258+properly.

237

259

238

260

239## Outside callers

261## Outside callers

@@ -252,7 +274,7 @@ Here we cover the basics, for complete information including additional examples

252

274

253### Parsing `tape`s

275### Parsing `tape`s

254

276

255−`+scan` takes in a `tape` and a `rule` and attempts to parse the `tape` with the rule.

277+`+scan` takes in a `tape` and a `rule` and attempts to parse the `tape` with the `rule`.

256

278

257```

279```

258> (scan "hello" (jest 'hello'))

280> (scan "hello" (jest 'hello'))

@@ -273,7 +295,7 @@ which is null if the parse failed.

273```

295```

274

296

275For the remainder of this tutorial we will make use of `+scan` so that we do not

297For the remainder of this tutorial we will make use of `+scan` so that we do not

276−need to deal directly with `nail`s unles it would be ilustrative to do so.

298+need to deal directly with `nail`s except where it is ilustrative to do so.

277

299

278## Parser modifiers

300## Parser modifiers

279

301

@@ -331,7 +353,8 @@ The syntax to combine `rule`s is

331```

353```

332The `rule`s are composed together using the combinator as an

354The `rule`s are composed together using the combinator as an

333intermediate function, which takes product of a `rule` (an `edge`) and a `rule` and turns

355intermediate function, which takes product of a `rule` (an `edge`) and a `rule` and turns

334−it into a sample (a `nail`) for the next `rule` to handle.

356+it into a sample (a `nail`) for the next `rule` to handle. We elaborate on this

357+behavior [below](#micsig).

335

358

336### `+plug`

359### `+plug`

337

360

@@ -368,17 +391,47 @@ results of each `rule`.

368syntax error

391syntax error

369```

392```

370

393

371−### `;~`

394+### `;~` {#micsig}

372

395

373Understanding the rune `;~` is essential to building parsers with Hoon. Let's

396Understanding the rune `;~` is essential to building parsers with Hoon. Let's

374take this opportunity to think about it carefully.

397take this opportunity to think about it carefully.

375

398

376−The `rule` created by `;~(combinator [list rule])`

399+The `rule` created by `;~(combinator [list rule])` may be understood

400+inductively. To this, let's consider the base case where our `[list rule]` has only a

401+single entry.

377```

402```

378> (scan "star" ;~(plug (jest 'star')))

403> (scan "star" ;~(plug (jest 'star')))

379'star'

404'star'

380```

405```

381−

406+Our output is identical to that given by `scan "star" (jest 'star')`. This is to

407+be expected. The combinator `+plug` is specifically used for chaining together

408+`rule`s in the `[list rule]`, but if there is only one `rule`, there is nothing

409+to chain. Thus, swapping out `+plug` for another combinator makes no difference here:

410+```

411+> (scan "star" ;~(pose (jest 'star')))

412+'star'

413+> (scan "star" ;~((glue com) (jest 'star')))

414+'star'

415+```

416+`;~` and the combinator only begin to play a role once the `[list rule]` has at

417+least two elements. So let's look at an example done with `+plug`, the simplest

418+combinator.

419+```

420+> (scan "star" ;~(plug (jest 'st') (jest 'ar')))

421+['st' 'ar']

422+```

423+Our return suggests that we first parsed `"star"` with the `rule` `(jest 'st')` and passed

424+the resulting `edge` to `(jest 'ar')` - in other words, we called `+plug` on `(jest

425+'st')` and the `edge` returned once it had been used to parse `"star"`. Thus

426+`+plug` was the glue that allowed us to join the two `rule`s, and `;~` performed

427+the gluing operation. And so, swapping `+plug` for `+pose` results in a crash,

428+which clues us into the fact that the combinator now has an effect since there

429+is more than one `rule`.

430+```

431+`> (scan "star" ;~(pose (jest 'st') (jest 'ar')))

432+{1 3}

433+syntax error

434+```

382

435

383## Parsing atoms

436## Parsing atoms

384

437

@@ -403,11 +456,7 @@ for parsing decimal numbers.

40343

45643

404```

457```

405

458

406−## Other parsing tidbits

407−

408−Need to mention `cold`, `like`

409−

410−### Recursive parsers

459+## Recursive parsers

411

460

412Naively attempting to write a recursive `rule`, i.e. like

461Naively attempting to write a recursive `rule`, i.e. like

413```

462```

@@ -436,7 +485,7 @@ expressions](#parsing-arithmetic-expressions) we will be writing a recursive

436parser making use of `+knee` and `~+` throughout.

485parser making use of `+knee` and `~+` throughout.

437

486

438

487

439−## Parsing arithmetic expressions

488+# Parsing arithmetic expressions

440

489

441In this section we will be applying what we have learned to write a parser for

490In this section we will be applying what we have learned to write a parser for

442arithmetic expressions in Hoon. That is, we will make a `rule` that takes in

491arithmetic expressions in Hoon. That is, we will make a `rule` that takes in

@@ -463,6 +512,9 @@ your `gen/` folder.

463(print xprs)

512(print xprs)

464|%

513|%

465++ int @ud

514++ int @ud

515+++ print

516+ |= [message=tape]

517+ (scan message expr)

466++ factor

518++ factor

467 %+ knee *int

519 %+ knee *int

468 |. ~+ ;~ pose

520 |. ~+ ;~ pose

@@ -481,9 +533,6 @@ your `gen/` folder.

481 ((slug add) lus ;~(pose turm expr))

533 ((slug add) lus ;~(pose turm expr))

482 turm

534 turm

483 ==

535 ==

484−++ print

485− |= [message=tape]

486− (scan message expr)

487--

536--

488```

537```

489

538

@@ -508,7 +557,7 @@ into Hoon terms, a decimal number is parsed by the `rule` `+dem` and an

508expression is parsed by removing the surrounding parentheses and then passing

557expression is parsed by removing the surrounding parentheses and then passing

509the result to the expression parser arm `+expr`, given by the `rule` `(ifix [lit

558the result to the expression parser arm `+expr`, given by the `rule` `(ifix [lit

510rit] expr)`. Since we want to parse our expression with one or the other, we

559rit] expr)`. Since we want to parse our expression with one or the other, we

511−chain these two `rule`s together using the monadic composition rune `;~` along

560+chain these two `rule`s together using the monadic applicator rune `;~` along

512with `+pose`, which says to try each rule in succession until one of them works.

561with `+pose`, which says to try each rule in succession until one of them works.

513

562

514Since expressions ultimately reduce to factors, we are actually building a

563Since expressions ultimately reduce to factors, we are actually building a