3A map of `jibb-dev/lib/jibb.hoon`, the compiler as milestone 1 left it: the layout pass, the parser, and the printers the golden tests compare. The file explains itself in design notes at its top, numbered 1 to 4; this page says where things are and points there for the reasoning.
5The plan's Compiler architecture section gives the whole pipeline. Here it is shortened, with the milestone that builds each step; milestone 1 built the first line:
9 → layout pass → parser → a syntax tree per module milestone 1
10 → module graph: imports resolved, Hoon types read from Clay milestone 2
11 → type checker, one module at a time, in dependency order milestone 2
12 → core language milestone 3
14 → JavaScript milestone 6
19- **One core, no Ford runes.** `urbit eval` runs the library as `=> <this file> <expression>`, and a ship builds it as `/lib/jibb.hoon`. So it can't import with `/-` or `/+`, and under `urbit eval` it can't scry. Anything from Clay, such as an imported module's text or a `/sur` file's type, is best passed in by the caller, which keeps every stage testable under eval.
20- **The standard library is the subject.** Under `urbit eval`, as on a ship, the library is built against `..zuse`: `hoon.hoon`, lull and zuse. It reads literals with Hoon's own parser, `nuck:so`. Hoon's type engine, `ut`, is there for milestone 2, and under eval `ride` compiles Hoon source, such as a `/sur` file without Ford runes, into a type, which tests can use in place of Clay's builds.
21- **The doc wins.** Where the file and the doc disagree, the doc is right, and the file's notes quote the doc's words for each rule they implement.
25From the header of the file:
27| Arm | Takes | Gives |
29| `+layout` | `[file=path src=@t]` | `(list token)`: never fails; an error is the last token |
30| `+layout-error` | `[file=path toks=(list token)]` | `(unit error)`: the error token's error, if any |
31| `+parse` | `[file=path src=@t]` | `(each module error)` |
32| `+parse-tokens` | `[file=path toks=(list token)]` | `(each module error)` |
33| `+golden` | `[file=path src=@t]` | `[tokens=@t tree=@t]`: both stages printed, or the rendered error |
34| `+show-tokens` | `(list token)` | `@t` |
35| `+show-module` | `module` | `@t` |
36| `+show-error` | `[src=@t error]` | `@t` |
38A file is a Clay path whose last segment is the extension, as `/ui/basket/jibb`.
42The core is split into chapters, `+| %name`, in this order:
46| `%positions` | `loc`, `span` and `error` |
47| `%tokens` | `token`, `lexeme`, `glyph` and `lit`: what the layout pass emits |
48| `%tree` | The syntax tree: `module`, `import`, `def`, `expr`, `piece`, `qseg`, `item`, `edit`, `arm`, `sample`, `field`, `pattern`, `shape` and `pseg` |
49| `%api` | The arms above that aren't printers |
50| `%lexer` | The layout pass's door, `+lexer`, and its helpers |
51| `%printers` | `+show-tokens`, `+show-module`, `+show-error` |
52| `%parser-helpers` | Error building (`+fail`, `+fail-in`, `+fail-space`, `+fail-line`, `+moved`, …) and the `+then` combinator |
53| `%parser` | The parser's door, `+parser`: one arm per EBNF rule, as `+read-module`, `+read-expr`, `+read-mold` |
54| `%span-check` | `+check-spans`, which `+golden` runs on every tree, and `+check-spans-test` |
60| 1. Positions | Lines and columns count from 1, columns in characters; every token and node has a span, start and end, and the rules a node's span follows |
61| 2. The layout pass | 2.0 where errors point, then lines, characters, blank lines, levels (INDENT, NEWLINE, DEDENT), the end of the file, tokens and tightness, paths, literals (read with `nuck:so`), strings, and what fails here |
62| 3. The parser | Why recursive descent (the grammar is LL(2) after the layout pass), the EBNF rule by rule with the node each gives, what the EBNF leaves out and how it's filled in, spacing, and where each error points, with its message |
63| 4. Printed formats | The token, tree and error formats the golden files hold |
67Every node is `[%tag at=span …]`, so `+span-of` finds its span in one place; the few that aren't say so in the `%tree` chapter. The kinds of node:
69- **Module**: imports (`%sur` for `/-`, `%mark` for `/%`, `%jibb` for `/=`), then definitions (`%mold` for `+$`, `%value` for `++`).
70- **Expressions** (`expr`): `%lit`, `%str`, `%path`, `%qath`, `%wing`, `%change`, `%pair`, `%unit`, `%group`, `%list`, `%tuple`, `%gate-wide`, `%ctor`, `%call`, `%gate`, `%cast`, `%trace`, `%branch`, `%crash`, and `%body` for an indented body, with its local definitions if it has any.
71- **Molds as written** (`shape`): `%aura`, `%noun`, `%void`, `%flag`, `%const`, `%ref`, `%tvar`, `%path`, `%apply`, `%cell`, `%union`, `%fn`, `%generic`.
72- **Samples, fields and patterns**: a sample is `%bind`, `%face`, `%pun`, `%cell` or `%bare`, since a bare name binds anywhere in a sample; a pattern is `%lit`, `%str`, `%any`, `%bind` or `%cell`.
73- **Literals** (`lit`) keep the form they matched, their aura and value, and their text as written, which the printers show.
75The tree keeps everything the checker needs to report an error at the code as written: open and closed brackets give the same node, but every node keeps its span.
77## Positions and errors
79- A span runs from a node's first character to just past its last. A node ends where its last token ends, never at a layout token, and holds its children's spans in order. `+check-spans` tests both rules on every tree the golden tests print.
80- An error is `[file=path line=@ud col=@ud msg=tape]`: a start, no end yet. Milestone 2's type errors need more, since the plan asks that "each error names both conflicting sites, with excerpts". Spans keep their ends for that.
81- Every error points at the first character where the text stops being the start of any valid module (note 2.0). The layout pass never fails: it ends its tokens with an error token, and the parser reports whichever error comes first, its own or that one.
82- The parser's messages read "expected <what>, found <token>", with an optional "; <hint>". The layout pass's state the rule broken, as "a tab: Jibb uses spaces only" or "`|%` is left out: modules are records". None ends with a period, and all start lowercase unless they open with a name, as "Jibb writes no @c literals". `+show-error` prints `<file>:<line>:<col>: <message>`, the line, and a caret under the column.
86Note 4 has examples of each.
88- **Tokens**: one per line, `<line>:<col>`, then `tight` if the token follows the previous one on its line with no space between, then its kind and text.
89- **Trees**: S-expressions, `(kind line:col fields kids)`, with each node's start; a node whose kids are all leaves prints on one line. Past 64 levels the indent stops growing, so the printed tree stays linear in the text's size.
90- **Errors**: as above. Control characters print as spaces in the excerpt, so the caret stays under its column.
94`+golden` is the only arm the golden tests call: `tools/golden` runs it under `urbit eval` on every case and compares `tokens.txt` and `tree.txt`; `tools/ship gen-tests` writes a `-test` arm per case that does the same on the ship. Adding a stage, such as the checker's output, means adding a field to `+golden`'s product and teaching both runners, the stub libraries in `tools/testdata` and their self-tests about it.
98Starting `urbit eval` and building the library takes about 8 s. A 2,000-line module of 400 definitions then parses in about a second. Loops over definitions, arms, lines and items are tail calls, so a long module grows no stack; nesting costs a few stack frames a level, and 100,000 nested brackets still parse (note 3).
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list