@@ -0,0 +1,1043 @@
1+:- :~ navhome/'/docs/'
2+ sort/'4'
3+ next/'true'
4+ title/'API overview by prefix'
5+ ==
6+;>
7+
8+## u3: API overview by prefix
9+
10+Let's run through the `u3` modules one by one. All public
11+functions are commented, but the comments may be cryptic.
12+
13+### u3m: main control
14+
15+To start `u3`, run
16+
17+```
18+/* u3m_boot(): start the u3 system.
19+*/
20+ void
21+ u3m_boot(c3_o nuu_o, c3_o bug_o, c3_c* dir_c);
22+```
23+
24+`nuu_o` is `c3y` (yes, `0`) if you're creating a new pier,
25+`c3n` (no, `1`) if you're loading an existing one. `bug_o`
26+is `c3y` if you want to test the garbage-collector, `c3n`
27+otherwise. `dir_c` is the directory for the pier files.
28+
29+`u3m_boot()` expects an `urbit.pill` file to load the kernel
30+from. It will try first `$dir/.urb.urbit.pill`, then `U3_LIB`.
31+
32+Any significant computation with nouns, certainly anything Turing
33+complete, should be run (a) virtualized and (b) in an inner road.
34+These are slightly different things, but at the highest level we
35+bundle them together for your convenience, in `u3m_soft()`:
36+
37+```
38+/* u3m_soft(): system soft wrapper. unifies unix and nock errors.
39+**
40+** Produces [%$ result] or [%error (list tank)].
41+*/
42+ u3_noun
43+ u3m_soft(c3_w sec_w, u3_funk fun_f, u3_noun arg);
44+```
45+
46+`sec_w` is the number of seconds to time out the computation.
47+`fun_f` is a C function accepting `arg`.
48+
49+The result of `u3m_soft()` is a cell whose head is an atom. If
50+the head is `%$` - ie, `0` - the tail is the result of
51+`fun_f(arg)`. Otherwise, the head is a `term` (an atom which is
52+an LSB first string), and the tail is a `(list tank)` (a list of
53+`tank` printables - see `++tank` in `hoon.hoon`). Error terms
54+should be the same as the exception terms above.
55+
56+If you're confident that your computation won't fail, you can
57+use `u3m_soft_sure()`, `u3m_soft_slam()`, or `u3m_soft_nock()`
58+for C functions, Hoon function calls, and Nock invocations.
59+Caution - this returns just the result, and asserts globally.
60+
61+All the `u3m_soft` functions above work *only on the surface*.
62+Within the surface, virtualize with `u3m_soft_run()`. Note that
63+this takes a `fly` (a namespace gate), thus activating the `11`
64+super-operator in the nock virtualizer, `++mock`. When actually
65+using the `fly`, call `u3m_soft_esc()`. Don't do either unless
66+you know what you're doing!
67+
68+For descending into a subroad *without* Nock virtualization,
69+use `u3m_hate()` and `u3m_love` respectively. Hating enters
70+a subroad; loving leaves it, copying out a product noun.
71+
72+Other miscellaneous tools in `u3m`: `u3m_file()` loads a Unix
73+file as a Nock atom; `u3m_water()` measures the boundaries of
74+the loom in current use (ie, watermarks); and a variety of
75+prettyprinting routines, none perfect, are available, mainly for
76+debugging printfs: `u3m_pretty()`, `u3m_p()`, `u3m_tape()` and
77+`u3m_wall()`.
78+
79+It's sometimes nice to run a mark-and-sweep garbage collector,
80+`u3m_grab()`, which collects the world from a list of roots,
81+and asserts if it finds any leaks or incorrect refcounts. This
82+tool is for debugging and long-term maintenance only; refcounts
83+should never err.
84+
85+### u3j: jets
86+
87+The jet system, `u3j`, is what makes `u3` and `nock` in any sense
88+a useful computing environment. Except perhaps `u3a` (there is
89+really no such thing as a trivial allocator, though `u3a` is
90+dumber than most) - `u3j` is the most interesting code in `u3`.
91+
92+Let's consider the minor miracle of driver-to-battery binding
93+which lets `u3j` work - and decrement not be `O(n)` - without
94+violating the precisely defined semantics of pure Nock, *ever*.
95+
96+It's easy to assume that jets represent an architectural coupling
97+between Hoon language semantics and Nock interpreter internals.
98+Indeed such a coupling would be wholly wrongtious and un-Urbit.
99+But the jet system is not Hoon-specific. It is specific to nock
100+runtime systems that use a design pattern we call a `core`.
101+
102+#### u3j: core structure
103+
104+A core is no more than a cell `[code data]`, in which a `code` is
105+either a Nock formula or a cell of `code`s, and `data` is anything.
106+In a proper core, the subject each formula expects is the core
107+itself.
108+
109+Except for the arbitrary decision to make a core `[code data]`,
110+(or as we sometimes say, `[battery payload]`), instead of `[data
111+code]`, any high-level language transforming itself to Nock would
112+use this design.
113+
114+So jets are in fact fully general. Broadly speaking, the jet
115+system works by matching a C *driver* to a battery. When the
116+battery is invoked with Nock operator `9`, it must be found in
117+associative memory and linked to its driver. Then we link the
118+formula axis of the operation (`a` in `[9 a b]`) to a specific
119+function in the driver.
120+
121+To validate this jet binding, we need to know two things. One,
122+we need to know the C function actually is a perfect semantic
123+match for the Nock formula. This can be developed with driver
124+test flags, which work, and locked down with a secure formula
125+hash in the driver, which we haven't bothered with just yet.
126+(You could also try to develop a formal method for verifying
127+that C functions and Nock formulas are equivalent, but this is
128+a research problem for the future.)
129+
130+Two, we need to validate that the payload is appropriate for the
131+battery. We should note that jets are a Nock feature and have no
132+reference to Hoon. A driver which relies on the Hoon type system
133+to only pair it with valid payloads is a broken driver, and
134+breaks the Nock compliance of the system as a whole. So don't.
135+
136+Now, a casual observer might look at `[battery payload]` and
137+expect the simplest case of it to be `[formula subject]`. That
138+is: to execute a simple core whose battery is a single formula,
139+we compute
140+
141+```
142+nock(+.a -.a)
143+```
144+
145+Then, naturally, when we go from Hoon or a high-level language
146+containing functions down to Nock, `[function arguments]` turns
147+into `[formula subject]`. This seems like an obvious design, and
148+we mention it only because it is *completely wrong*.
149+
150+Rather, to execute a one-armed core like the above, we run
151+
152+```
153+nock(a -.a)
154+```
155+
156+and the normal structure of a `gate`, which is simply Urbitese
157+for "function," is:
158+
159+```
160+[formula [sample context]]
161+```
162+
163+where `sample` is Urbitese for "arguments" - and `context`, any
164+Lisper will at once recognize, is Urbitese for "environment."
165+
166+To `slam` or call the gate, we simply replace the default sample
167+with the caller's data, then nock the formula on the entire gate.
168+
169+What's in the context? Unlike in most dynamic languages, it is
170+not some secret system-level bag of tricks. Almost always it is
171+another core. This onion continues until at the bottom, there is
172+an atomic constant, conventionally is the kernel version number.
173+
174+Thus a (highly desirable) `static` core is one of the form
175+
176+```
177+[battery constant]
178+[battery static-core]
179+```
180+
181+ie, a solid stack of nested libraries without any dynamic data.
182+The typical gate will thus be, for example,
183+
184+```
185+[formula [sample [battery battery battery constant]]]
186+```
187+
188+but we would be most foolish to restrict the jet mechanism to
189+cores of this particular structure. We cannot constrain a
190+payload to be `[sample static-core]`, or even `[sample core]`.
191+Any such constraint would not be rich enough to handle Hoon,
192+let alone other languages.
193+
194+#### u3j: jet state
195+
196+There are two fundamental rules of computer science: (1) every
197+system is best understood through its state; (2) less state is
198+better than more state. Sadly, a pier has three different jet
199+state systems: `cold`, `warm` and `hot`. It needs all of them.
200+
201+Hot state is associated with this particular Unix process. The
202+persistent pier is portable not just between process and process,
203+but machine and machine or OS and OS. The set of jets loaded
204+into a pier may itself change (in theory, though not in the
205+present implementation) during the lifetime of the process. Hot
206+state is a pure C data structure.
207+
208+Cold state is associated with the logical execution history of
209+the pier. It consists entirely of nouns and ignores restarts.
210+
211+Warm state contains all dependencies between cold and hot
212+state. It consists of C structures allocated on the loom.
213+
214+Warm state is purely a function of cold and hot states, and
215+we can wipe and regenerate it at any time. On any restart where
216+the hot state might have changed, we clear the warm state
217+with `u3j_ream()`.
218+
219+There is only one hot state, the global jet dashboard
220+`u3j_Dash` or `u3D` for short. In the present implementation,
221+u3D is a static structure not modified at runtime, except for
222+numbering itself on process initialization. This structure -
223+which embeds function pointers to all the jets - is defined
224+in `j/tree.c`. The data structures:
225+
226+```
227+ /* u3j_harm: driver arm.
228+ */
229+ typedef struct _u3j_harm {
230+ c3_c* fcs_c; // `.axe` or name
231+ u3_noun (*fun_f)(u3_noun); // compute or 0 / semitransfer
232+ c3_o ice; // perfect (don't test)
233+ c3_o tot; // total (never punts)
234+ c3_o liv; // live (enabled)
235+ } u3j_harm;
236+
237+ /* u3j_core: C core driver.
238+ */
239+ typedef struct _u3j_core {
240+ c3_c* cos_c; // control string
241+ struct _u3j_harm* arm_u; // blank-terminated static list
242+ struct _u3j_core* dev_u; // blank-terminated static list
243+ struct _u3j_core* par_u; // dynamic parent pointer
244+ c3_l jax_l; // dynamic jet index
245+ } u3j_core;
246+
247+ /* u3e_dash, u3_Dash, u3D: jet dashboard singleton
248+ */
249+ typedef struct _u3e_dash {
250+ u3j_core* dev_u; // null-terminated static list
251+ c3_l len_l; // ray_u filled length
252+ c3_l all_l; // ray_u allocated length
253+ u3j_core* ray_u; // dynamic driver array
254+ } u3j_dash;
255+```
256+
257+Warm and cold state is *per road*. In other words, as we nest
258+roads, we also nest jet state. The jet state in the road is:
259+
260+```
261+ struct { // jet dashboard
262+ u3p(u3h_root) har_p; // warm state
263+ u3_noun das; // cold state
264+ } jed;
265+```
266+
267+In case you understand Hoon, `das` (cold state) is a `++dash`,
268+and `har_p` (warm state) is a map from battery to `++calx`:
269+
270+```
271+ ++ bane ,@tas :: battery name
272+ ++ bash ,@uvH :: label hash
273+ ++ bosh ,@uvH :: local battery hash
274+ ++ batt ,* :: battery
275+ ++ calf ::
276+ $: jax=,@ud :: hot core index
277+ hap=(map ,@ud ,@ud) :: axis/hot arm index
278+ lab=path :: label as path
279+ jit=* :: arbitrary data
280+ == ::
281+ ++ calx (trel calf (pair bash cope) club) :: cached by battery
282+ ++ clog (pair cope (map batt club)) :: identity record
283+ ++ club (pair corp (map term nock)) :: battery pattern
284+ ++ cope (trel bane axis (each bash noun)) :: core pattern
285+ ++ core ,* :: core
286+ ++ corp (each core batt) :: parent or static
287+ ++ dash (map bash clog) :: jet system
288+```
289+
290+The driver index `jax` in a `++calx` is an index into `ray_u` in the
291+dashboard - ie, a pointer into hot state. This is why the warm
292+state has to be reset when we reload the pier in a new process.
293+
294+Why is jet state nested? Nock of course is a functional system,
295+so as we compute we don't explicitly create state. Jet state is
296+an exception to this principle (which works only because it can't
297+be semantically detected from Nock/Hoon) - but it can't violate
298+the fundamental rules of the allocation system.
299+
300+For instance, when we're on an inner road, we can't allocate on
301+an outer road, or point from an outer road to an inner. So if we
302+learn something - like a mapping from battery to jet - in the
303+inner road, we have to keep it in the inner road.
304+
305+Mitigating this problem, when we leave an inner road (with
306+`u3m_love()`), we call `u3j_reap()` to promote jet information in
307+the dying road. Reaping promotes anything we've learned about
308+any battery that either (a) already existed in the outer road, or
309+(b) is being saved to the outer road.
310+
311+#### u3j: jet binding
312+
313+Jet binding starts with a `%fast` hint. (In Hoon, this is
314+produced by the runes `~%`, for the general case, or `~/`
315+for simple functions.) To bind a jet, execute a formula of the
316+form:
317+
318+```
319+[10 [%fast clue-formula] core-formula]
320+```
321+
322+`core-formula` assembles the core to be jet-propelled.
323+`clue-formula` produces the hint information, or `++clue`
324+above, which we want to annotate it with.
325+
326+A clue is a triple of name, parent, and hooks:
327+
328+```
329+++ clue (trel chum nock (list (pair term nock)))
330+```
331+
332+The name, or `++chum`, has a bunch of historical structure which
333+we don't need (cleaning these things up is tricky), but just gets
334+flattened into a term.
335+
336+The parent axis is a nock formula, but always reduces to a simple
337+axis, which is the address of this core's *parent*. Consider
338+again an ordinary gate
339+
340+```
341+[formula [sample context]]
342+```
343+
344+Typically the `context` is itself a library core, which itself
345+has a jet binding. If so, the parent axis of this gate is `7`.
346+
347+If the parent is already bound - and the parent *must* be already
348+bound, in this road or a road containing it - we can hook this core
349+bottom-up into a tree hierarchy. Normally the child core is
350+produced by an arm of the parent core, so this is not a problem -
351+we wouldn't have the child if we hadn't already made the parent.
352+
353+The clue also contains a list of *hooks*, named nock formulas on
354+the core. Usually these are arms, but they need not be. The
355+point is that we often want to call a core from C, in a situation
356+where we have no type or other source information. A common case
357+of this is a complex system in which we're mixing functions which
358+are jet-propelled with functions that aren't.
359+
360+In any case, all the information in the `%fast` hint goes to
361+`u3j_mine()`, which registers the battery in cold state (`das` in
362+`jed` in `u3R`), then warm state (`har_p` in `jed`).
363+
364+It's essential to understand that the `%fast` hint has to be,
365+well, fast - because we apply it whenever we build a core. For
366+instance, if the core is a Hoon gate - a function - we will call
367+`u3j_mine` every time the function is called.
368+
369+### u3j: the cold jet dashboard
370+
371+For even more fun, the jet tree is not actually a tree of
372+batteries. It's a tree of battery *labels*, where a label is
373+an [axis term] path from the root of the tree. (At the root,
374+if the core pattern is always followed properly, is a core whose
375+payload is an atomic constant, conventionally the Hoon version.)
376+
377+Under each of these labels, it's normal to have an arbitrary
378+number of different Nock batteries (not just multiple copies
379+of the same noun, a situation we *do* strive to avoid). For
380+instance, one might be compiled with debugging hints, one not.
381+
382+We might even have changed the semantics of the battery without
383+changing the label - so long as those semantics don't invalidate
384+any attached driver.
385+
386+For instance, it's normal to have two equivalent Nock batteries
387+at the same time in one pier: one battery compiled with
388+debugging hints, one not.
389+
390+Rather, the jet tree is a semantic hierarchy. The root of the
391+hierarchy is a constant, by convention the Hoon kernel version
392+because any normal jet-propelled core has, at the bottom of its
393+onion of libraries, the standard kernel. Thus if the core is
394+
395+```
396+[foo-battery [bar-battery [moo-battery 164]]]
397+```
398+
399+we can reverse the nesting to construct a hierarchical core
400+path. The static core
401+
402+```
403+164/moo/bar/foo
404+```
405+
406+extends the static core `164/moo/bar` by wrapping the `foo`
407+battery (ie, in Hoon, `|%`) around it. With the core above,
408+you can compute `foo` stuff, `bar` stuff, and `moo` stuff.
409+Rocket science, not.
410+
411+Not all cores are static, of course - they may contain live data,
412+like the sample in a gate (ie, argument to a function). Once
413+again, it's important to remember that we track jet bindings not
414+by the core, which may not be static, but by the battery, which
415+is always static.
416+
417+(And if you're wondering how we can use a deep noun like a Nock
418+formula or battery as a key in a key-value table, remember
419+`mug_w`, the lazily computed short hash, in all boxed nouns.)
420+
421+In any case, `das`, the dashboard, is a map from `bash` to jet
422+location record (`++clog`). A `clog` in turn contains two kinds
423+of information: the `++cope`, or per-location noun; and a map of
424+batteries to a per-battery `++club`.
425+
426+The `cope` is a triple of `++bane` (battery name, right now just
427+a `term`); `++axis`, the axis, within *this* core, of the parent;
428+and `(each bash noun)`, which is either `[0 bash]` if the parent
429+is another core, or `[1 noun]`, for the constant noun (like
430+`164`) if there is no parent core.
431+
432+A `bash` is just the noun hash (`++sham`) of a `cope`, which
433+uniquely expresses the battery's hierarchical location without
434+depending on the actual formulas.
435+
436+The `club` contains a `++corp`, which we use to actually validate
437+the core. Obviously jet execution has to be perfectly compatible
438+with Nock. We search on the battery, but getting the battery
439+right is not enough - a typical battery is dependent on its
440+context. For example, your jet-propelled library function is
441+very likely to call `++dec` or other advanced kernel technology.
442+If you've replaced the kernel in your context with something
443+else, we need to detect this and not run the jet.
444+
445+There are two cases for a jet-propelled core - either the entire
446+core is a static constant, or it isn't. Hence the definition
447+of `corp`:
448+
449+```
450+++ corp (each core batt) :: parent or static
451+```
452+
453+Ie, a `corp` is `[0 core]` or `[1 batt]`. If it's static -
454+meaning that the jet only works with one specific core, ie, the
455+parent axis of each location in the hierarchy is `3` - we can
456+validate with a single comparison. Otherwise, we have to recurse
457+upward by checking the parent.
458+
459+Note that there is at present no way to force a jet to depend on
460+static *data*.
461+
462+### u3j: the warm jet dashboard
463+
464+We don't use the cold state to match jets as we call them. We
465+use the cold state to register jets as we find them, and also to
466+rebuild the warm state after the hot state is reset.
467+
468+What we actually use at runtime is the warm state, `jed->har_p`,
469+which is a `u3h` (built-in hashtable), allocated on the loom,
470+from battery to `++calx`.
471+
472+A `calx` is a triple of a `++calf`, a `[bash cope]` cell, and a
473+`club`. The latter two are all straight from cold state.
474+
475+The `calf` contains warm data dependent on hot state. It's a
476+quadruple: of `jax`, the hot driver index (in `ray_u` in
477+`u3j_dash`); `hap`, a table from arm axis (ie, the axis of each
478+formula within the battery) to driver arm index (into `arm_u` in
479+`u3j_core`); `lab`, the complete label path; and `jit`, any
480+other dynamic data that may speed up execution.
481+
482+We construct `hap`, when we create the calx, by iterating through
483+the arms registered in the `u3j_core`. Note the way a `u3j_harm`
484+declares itself, with the string `fcs_c` which can contain either
485+an axis or a name. Most jetted cores are of course gates, which
486+have one formula at one axis within the core: `fcs_c` is `".3"`.
487+
488+But we do often have fast cores with more complex arm structure,
489+and it would be sad to have to manage their axes by hand. To use
490+an `fcs_c` with a named arm, it's sufficient to make sure the
491+name is bound to a formula `[0 axis]` in the hook table.
492+
493+`jit`, as its name suggests, is a stub where any sort of
494+optimization data computed on battery registration might go. To
495+use it, fill in the `_cj_jit()` function.
496+
497+### u3j: the hot dashboard
498+
499+Now it should be easy to see how we actually invoke jets. Every
500+time we run a nock `9` instruction (pretty often, obviously), we
501+have a core and an axis. We pass these to `u3j_kick()`, which
502+will try to execute them.
503+
504+Because nouns with a reference count of 1 are precious,
505+`u3j_kick()` has a tricky reference control definition. It
506+reserves the right to return `u3_none` in the case where there is
507+no driver, or the driver does not apply for this case; in this
508+case, it retains argument `cor`. If it succeeds, though, it
509+transfers `cor`.
510+
511+`u3j_kick()` searches for the battery (always the head of the
512+core, of course) in the hot dashboard. If the battery is
513+registered, it searches for the axis in `hap` in the `calx`.
514+If it exists, the core matches a driver and the driver jets this
515+arm. If not, we return `u3_none`.
516+
517+Otherwise, we call `fun_f` in our `u3j_harm`. This obeys the
518+same protocol as `u3j_kick()`; it can refuse to function by
519+returning `u3_none`, or consume the noun.
520+
521+Besides the actual function pointer `fun_f`, we have some flags
522+in the `u3j_harm` which tell us how to call the arm function.
523+
524+If `ice` is yes (`&`, `0`), the jet is known to be perfect and we
525+can just trust the product of `fun_f`. Otherwise, we need to run
526+*both* the Nock arm and `fun_f`, and compare their results.
527+
528+(Note that while executing the C side of this test, we have to
529+set `ice` to yes; on the Nock side, we have to set `liv` to no.
530+Otherwise, many non-exponential functions become exponential.
531+When auto-testing jets in this way, the principle is that the
532+test is on the outermost layer of recursion.)
533+
534+(Note also that anyone who multi-threads this execution
535+environment has a slight locking problem with these flags if arm
536+testing is multi-threaded.)
537+
538+If `tot` is yes, (`&`, `0`), the arm function is *total* and has
539+to return properly (though it can still return *u3_none*).
540+Otherwise, it is *partial* and can `u3_cm_bail()` out with
541+c3__punt. This feature has a cost: the jet runs in a subroad.
542+
543+Finally, if `liv` is no (`|`, 1), the jet is off and doesn't run.
544+
545+It should be easy to see how the tree of cores gets declared -
546+precisely, in `j/dash.c`. We declare the hierarchy as a tree
547+of `u3j_core` structures, each of which comes with a static list
548+of arms `arm_u` and sub-cores `dev_u`.
549+
550+In `u3j_boot()`, we traverse the hierarchy, fill in parent
551+pointers `par_u`, and enumerate all `u3j_core` structures
552+into a single flat array `u3j_dash.ray_u`. Our hot state
553+then appears ready for action.
554+
555+### u3j: jet functions
556+
557+At present, all drivers are compiled statically into `u3`. This is
558+not a long-term permanent solution or anything. However, it will
559+always be the case with a certain amount of core functionality.
560+
561+For instance, there are some jet functions that we need to call
562+as part of loading the Arvo kernel - like `++cue` to unpack a
563+noun from an atom. And obviously it makes sense, when jets are
564+significant enough to compile into `u3`, to export their symbols
565+in headers and the linker.
566+
567+There are three interface prefixes for standard jet functions:
568+`u3k`, `u3q`, and `u3w`. All jets have `u3w` interfaces; most
569+have `u3q`; some have `u3k`. Of course the actual logic is
570+shared.
571+
572+`u3w` interfaces use the same protocol as `fun_f` above: the
573+caller passes the entire core, which is retained if the function
574+returns `u3_none`, transferred otherwise. Why? Again, use
575+counts of 1 are special and precious for performance hackers.
576+
577+`u3q` interfaces break the core into C arguments, *retain* noun
578+arguments, and *transfer* noun returns. `u3k` interfaces are the
579+same, except with more use of `u3_none` and other simple C
580+variations on the Hoon original, but *transfer* both arguments
581+and returns. Generally, `u3k` are most convenient for new code.
582+
583+Following `u3k/q/w` is `[a-f]`, corresponding to the 6 logical
584+tiers of the kernel, or `g` for user-level jets. Another letter
585+is added for functions within subcores. The filename, under
586+`j/`, follows the tier and the function name.
587+
588+For instance, `++add` is `u3wa_add(cor)`, `u3qa_add(a, b)`, or
589+`u3ka_add(a, b)`, in `j/a/add.c`. `++get` in `++by` is
590+`u3wdb_get(cor)`, `u3kdb_get(a, b)`, etc, in `j/d/by_get.c`.
591+
592+For historical reasons, all internal jet code in `j/[a-f]`
593+*retains* noun arguments, and *transfers* noun results. Please
594+do not do this in new `g` jets! The new standard protocol is to
595+transfer both arguments and results.
596+
597+### u3a: allocation functions
598+
599+`u3a` allocates on the current road (u3R). Its internal
600+structures are uninteresting and typical of a naive allocator.
601+
602+The two most-used `u3a` functions are `u3a_gain()` to add a
603+reference count, and `u3a_lose()` to release one (and free the
604+noun, if the use count is zero). For convenience, `u3a_gain()`
605+returns its argument. The pair are generally abbreviated with
606+the macros `u3k()` and `u3z()` respectively.
607+
608+Normally we create nouns through `u3i` functions, and don't call
609+the `u3a` allocators directly. But if you do:
610+
611+One, there are *two* sets of allocators: the word-aligned
612+allocators and the fully-aligned (ie, malloc compatible)
613+allocators. For instance, on a typical OS X setup, malloc
614+produces 16-byte aligned results - needed for some SSE
615+instructions.
616+
617+These allocators are *not compatible*. For 32-bit alignment
618+as used in nouns, call
619+
620+```
621+ /* u3a_walloc(): allocate storage measured in words.
622+ */
623+ void*
624+ u3a_walloc(c3_w len_w);
625+
626+ /* u3a_wfree(): free storage.
627+ */
628+ void
629+ u3a_wfree(void* lag_v);
630+
631+ /* u3a_wealloc(): word realloc.
632+ */
633+ void*
634+ u3a_wealloc(void* lag_v, c3_w len_w);
635+```
636+
637+For full alignment, call:
638+
639+```
640+ /* u3a_malloc(): aligned storage measured in bytes.
641+ */
642+ void*
643+ u3a_malloc(size_t len_i);
644+
645+ /* u3a_realloc(): aligned realloc in bytes.
646+ */
647+ void*
648+ u3a_realloc(void* lag_v, size_t len_i);
649+
650+ /* u3a_realloc2(): gmp-shaped realloc.
651+ */
652+ void*
653+ u3a_realloc2(void* lag_v, size_t old_i, size_t new_i);
654+
655+ /* u3a_free(): free for aligned malloc.
656+ */
657+ void
658+ u3a_free(void* tox_v);
659+
660+ /* u3a_free2(): gmp-shaped free.
661+ */
662+ void
663+ u3a_free2(void* tox_v, size_t siz_i);
664+```
665+
666+There are also a set of special-purpose allocators for building
667+atoms. When building atoms, please remember that it's incorrect
668+to have a high 0 word - the word length in the atom structure
669+must be strictly correct.
670+
671+Of course, we don't always know how large our atom will be.
672+Therefore, the standard way of building large atoms is to
673+allocate a block of raw space with `u3a_slab()`, then chop off
674+the end with `u3a_malt()` (which does the measuring itself)
675+or `u3a_mint()` in case you've measured it yourself.
676+
677+Once again, *do not call `malloc()`* (or C++ `new`) within any
678+code that may be run within a jet. This will cause rare sporadic
679+corruption when we interrupt execution within a `malloc()`. We'd
680+just override the symbol, but `libuv` uses `malloc()` across
681+threads within its own synchronization primitives - for this to
682+work with `u3a_malloc()`, we'd have to introduce our own locks on
683+the surface-level road (which might be a viable solution).
684+
685+### u3n: nock execution
686+
687+The `u3n` routines execute Nock itself. On the inside, they have
688+a surprising resemblance to the spec proper (the only interesting
689+detail is how we handle tail-call elimination) and are, as one
690+would expect, quite slow. (There is no such thing as a fast tree
691+interpreter.)
692+
693+There is only one Nock, but there are lots of ways to call it.
694+(Remember that all `u3n` functions *transfer* C arguments and
695+returns.)
696+
697+The simplest interpreter, `u3n_nock_on(u3_noun bus, u3_noun fol)`
698+invokes Nock on `bus` (the subject) and `fol` (the formula).
699+(Why is it`[subject formula]`, not `[formula subject]`? The same
700+reason `0` is true and `1` is false.)
701+
702+A close relative is `u3n_slam_on(u3_noun gat, u3_noun sam)`,
703+which slams a *gate* (`gat`) on a sample (`sam`). (In a normal
704+programming language which didn't talk funny and was retarded,
705+`u3n_slam_on()` would call a function on an argument.) We could
706+write it most simply as:
707+
708+```
709+ u3_noun
710+ u3n_slam_on(u3_noun gat, u3_noun sam)
711+ {
712+ u3_noun pro = u3n_nock_on
713+ (u3nc(u3k(u3h(gat)),
714+ u3nc(sam, u3k(u3t(u3t(gat))))),
715+ u3k(u3h(gat)));
716+ u3z(gat);
717+ return pro;
718+ }
719+```
720+
721+Simpler is `u3n_kick_on(u3_noun gat)`, which slams a gate (or,
722+more generally, a *trap* - because sample structure is not even
723+needed here) without changing its sample:
724+
725+```
726+ u3_noun
727+ u3n_kick_on(u3_noun gat, u3_noun sam)
728+ {
729+ return u3n_nock_on(gat, u3k(u3h(gat)));
730+ }
731+```
732+
733+The `_on` functions in `u3n` are all defined as pure Nock. But
734+actually, even though we say we don't extend Nock, we do. But we
735+don't. But we do.
736+
737+Note that `u3` has a well-developed error handling system -
738+`u3m_bail()` to throw an exception, `u3m_soft_*` to catch one.
739+But Nock has no exception model at all. That's okay - all it
740+means if that if an `_on` function bails, the exception is an
741+exception in the caller.
742+
743+However, `u3`'s exception handling happens to match a convenient
744+virtual super-Nock in `hoon.hoon`, the infamous `++mock`. Of
745+course, Nock is slow, and `mock` is Nock in Nock, so it is
746+(logically) super-slow. Then again, so is decrement.
747+
748+With the power of `u3`, we nest arbitrary layers of `mock`
749+without any particular performance cost. Moreover, we simply
750+treat Nock proper as a special case of `mock`. (More precisely,
751+the internal VM loop is `++mink` and the error compiler is
752+`++mook`. But we call the whole sandbox system `mock`.)
753+
754+The nice thing about `mock` functions is that (by executing
755+within `u3m_soft_run()`, which as you may recall uses a nested
756+road) they provide both exceptions and the namespace operator -
757+`.^` in Hoon, which becomes operator `11` in `mock`.
758+
759+`11` requires a namespace function, or `fly`, which produces a
760+`++unit` - `~` (`0`) for no binding, or `[0 value]`. The sample
761+to a `fly` is a `++path`, just a list of text `span`.
762+
763+`mock` functions produce a `++toon`. Fully elaborated:
764+
765+```
766+ ++ noun ,* :: any noun
767+ ++ path (list ,@ta) :: namespace path
768+ ++ span ,@ta :: text-atom (ASCII)
769+ ++ toon $% [%0 p=noun] :: success
770+ [%1 p=(list path)] :: blocking paths
771+ [%2 p=(list tank)] :: stack trace
772+ == ::
773+ ++ tank :: printable
774+ $% [%leaf p=tape] :: flat text
775+ $: %palm :: backstep list
776+ p=[p=tape q=tape r=tape s=tape] :: mid cap open close
777+ q=(list tank) :: contents
778+ == ::
779+ $: %rose :: straight list
780+ p=[p=tape q=tape r=tape] :: mid open close
781+ q=(list tank) :: contents
782+ == ::
783+ ==
784+```
785+
786+(Note that `tank` is overdesigned and due for replacement.)
787+
788+What does a `toon` mean? Either your computation succeded (`[0
789+noun]`, or could not finish because it blocked on one or more
790+global paths (`[1 (list path)]`), or it exited with a stack trace
791+(`[2 (list tank)]`).
792+
793+Note that of all the `u3` exceptions, only `%exit` is produced
794+deterministically by the Nock definition. Therefore, only
795+`%exit` produces a `2` result. Any other argument to
796+`u3m_bail()` will unwind the virtualization stack all the way to
797+the top - or to be more exact, to `u3m_soft_top()`.
798+
799+In any case, the simplest `mock` functions are `u3n_nock_un()`
800+and `u3n_slam_un()`. These provide exception control without
801+any namespace change, as you can see by the code:
802+
803+```
804+ /* u3n_nock_un(): produce .*(bus fol), as ++toon.
805+ */
806+ u3_noun
807+ u3n_nock_un(u3_noun bus, u3_noun fol)
808+ {
809+ u3_noun fly = u3nt(u3nt(11, 0, 6), 0, 0); // |=(a=* .^(a))
810+
811+ return u3n_nock_in(fly, bus, fol);
812+ }
813+
814+ /* u3n_slam_un(): produce (gat sam), as ++toon.
815+ */
816+ u3_noun
817+ u3n_slam_un(u3_noun gat, u3_noun sam)
818+ {
819+ u3_noun fly = u3nt(u3nt(11, 0, 6), 0, 0); // |=(a=* .^(a))
820+
821+ return u3n_slam_in(fly, gat, sam);
822+ }
823+```
824+
825+The `fly` is added as the first argument to `u3n_nock_in()` and
826+`u3n_slam_in()`. Of course, logically, `fly` executes in the
827+caller's exception layer. (Maintaining this illusion is slightly
828+nontrivial.) Finally, `u3n_nock_an()` is a sandbox with a null
829+namespace.
830+
831+### u3e: persistence
832+
833+The only `u3e` function you should need to call is `u3e_save()`,
834+which saves the loom. As it can be restored on any platform,
835+please make sure you don't have any state in the loom that is
836+bound to your process or architecture - except for exceptions
837+like the warm jet state, which is actively purged on reboot.
838+
839+### u3r: reading nouns (weak)
840+
841+As befits accessors they don't make anything, `u3r` noun reading
842+functions always retain their arguments and their returns. They
843+never bail; rather, when they don't work, they return a `u3_weak`
844+result.
845+
846+Most of these functions are straightforward and do only what
847+their comments say. A few are interesting enough to discuss.
848+
849+`u3r_at()` is the familiar tree fragment function, `/` from the
850+Nock spec. For taking complex nouns apart, `u3r_mean()` is a
851+relatively funky way of deconstructing nouns with a varargs list
852+of `axis`, `u3_noun *`. For cells, triples, etc, decompose with
853+`u3r_cell()`, `u3r_trel()`, etc. For the tagged equivalents, use
854+`u3r_pq()` and friends.
855+
856+`u3r_sing(u3_noun a, u3_noun b)` (true if `a` and `b` are a
857+*single* noun) are interesting because it uses mugs to help it
858+out. Clearly, different nouns may have the same mug, but the
859+same nouns cannot have a different mug. It's important to
860+understand the performance characteristics of `u3r_sing()`:
861+the worst possible case is a comparison of duplicate nouns,
862+which have the same value but were created separately. In this
863+case, the tree is traversed
864+
865+`u3r_sung()` is a deeply funky and frightening version of
866+`u3r_sing()` that unifies pointers to the duplicate nouns it
867+finds, freeing the second copy. Obviously, do not use
868+`u3r_sung()` when you have live, but not reference counted, noun
869+references from C - if they match a noun with a refcount of 1
870+that gets freed, bad things happen.
871+
872+It's important to remember that `u3r_mug()`, which produces a
873+31-bit, nonzero insecure hash, uses the `mug_w` slot in any boxed
874+noun as a lazy cache. There are a number of variants of
875+`u3r_mug()` that can get you out of building unneeded nouns.
876+
877+### u3x: reading nouns (bail)
878+
879+`u3x` functions are like `u3r` functions, but instead of
880+returning `u3_none` when (for instance) we try to take the head
881+of an atom, they bail with `%exit`. In other words, they do what
882+the same operation would do in Nock.
883+
884+### u3h: hash tables.
885+
886+We can of course use the Hoon `map` structure as an associative
887+array. This is a balanced treap and reasonably fast. However,
888+it's considerably inferior to a custom structure like an HAMT
889+(hash array-mapped trie). We use `u3_post` to allocate HAMT
890+structures on the loom.
891+
892+(Our HAMT implements the classic Bagwell algorithm which depends
893+on the `gcc` standard directive `__builtin_popcount()`. On a CPU
894+which doesn't support popcount or an equivalent instruction, some
895+other design would probably be preferable.)
896+
897+There's no particular rocket science in the API. `u3h_new()`
898+creates a hashtable; `u3h_free()` destroys one; `u3h_put()`
899+inserts, `u3h_get()` retrieves. You can transform values in a
900+hashtable with `u3h_walk()`.
901+
902+The only funky function is `u3h_gut()`, which unifies keys with
903+`u3r_sung()`. As with all cases of `u3r_sung()`, this must be
904+used with extreme caution.
905+
906+### u3z: memoization
907+
908+Connected to the `~+` rune in Hoon, via the Nock `%memo` hint,
909+the memoization facility is a general-purpose cache.
910+
911+(It's also used for partial memoization - a feature that'll
912+probably be removed, in which conservative worklist algorithms
913+(which would otherwise be exponential) memoize everything in the
914+subject *except* the worklist. This is used heavily in the Hoon
915+compiler jets (j/f/*.c). Unfortunately, it's probably not
916+possible to make this work perfectly in that it can't be abused
917+to violate Nock, so we'll probably remove it at a later date,
918+instead making `++ut` keep its own monadic cache.)
919+
920+Each `u3z` function comes with a `c3_m` mote which disambiguates
921+the function mapping key to value. For Nock itself, use 0. For
922+extra speed, small tuples are split out in C; thus, find with
923+
924+```
925+ u3_weak u3z_find(c3_m, u3_noun);
926+ u3_weak u3z_find_2(c3_m, u3_noun, u3_noun);
927+ u3_weak u3z_find_3(c3_m, u3_noun, u3_noun, u3_noun);
928+ u3_weak u3z_find_4(c3_m, u3_noun, u3_noun, u3_noun, u3_noun);
929+```
930+
931+and save with
932+
933+```
934+ u3_noun u3z_save(c3_m, u3_noun, u3_noun);
935+ u3_noun u3z_save_2(c3_m, u3_noun, u3_noun, u3_noun);
936+ u3_noun u3z_save_3(c3_m, u3_noun, u3_noun, u3_noun, u3_noun);
937+ u3_noun u3z_save_4(c3_m, u3_noun, u3_noun, u3_noun, u3_noun, u3_noun);
938+```
939+
940+where the value is the last argument. To eliminate duplicate
941+nouns, there is also
942+
943+```
944+ u3_noun
945+ u3z_uniq(u3_noun);
946+```
947+
948+`u3z` functions retain keys and transfer values.
949+
950+The `u3z` cache, built on `u3h` hashes, is part of the current
951+road, and goes away when it goes away. (In future, we may wish
952+to promote keys/values which outlive the road, as we do with jet
953+state.) There is no cache reclamation at present, so be careful.
954+
955+### u3t: tracing and profiling.
956+
957+TBD.
958+
959+### u3v: the Arvo kernel
960+
961+An Arvo kernel - or at least, a core that compiles with the Arvo
962+interface - is part of the global `u3` state. What is an Arvo
963+core? Slightly pseudocoded:
964+
965+```
966+ ++ arvo
967+ |%
968+ ++ come |/ {yen/@ ova/(list ovum) nyf/pone} :: 11
969+ ^- {(list ovum) _+>}
970+ !!
971+ ++ keep |/ {now/@da hap/path} :: 4
972+ ^- (unit ,@da)
973+ !!
974+ ++ load |/ {yen/@ ova/(list ovum) nyf/pane} :: 86
975+ ^- {(list ovum) _+>}
976+ !!
977+ ++ peek |/ {now/@da path} :: 87
978+ ^- (unit)
979+ !!
980+ ++ poke |/ {now/@da ovo/ovum} :: 42
981+ ^- {(list ovum) _+>}
982+ !!
983+ ++ wish |/ txt/@ta :: 20
984+ ^- *
985+ !!
986+ --
987+ ++ card {p/@tas q/*} :: typeless card
988+ ++ ovum {p/wire q/card} :: Arvo event
989+ ++ wire path :: event cause
990+```
991+
992+This is the Arvo ABI in a very real sense. Arvo is a core with
993+these six arms. To use these arms, we hardcode the axis of the
994+formula (`11`, `4`, `86`, etc) into the C code that calls Arvo,
995+because otherwise we'd need type metadata - which we can get, by
996+calling Arvo.
997+
998+It's important to understand the Arvo event/action structure, or
999+`++ovum`. An `ovum` is a `card`, which is any `[term noun]`
1000+cell, and a `++wire`, a `path` which indicates the location of
1001+the event. At the Unix level, the `wire` corresponds to a system
1002+module or context. For input events, this is the module that
1003+caused the event; for output actions, it's the module that
1004+performs the action.
1005+
1006+`++poke` sends Arvo an event `ovum`, producing a cell of action
1007+ova and a new Arvo core.
1008+
1009+`++peek` dereferences the Arvo namespace. It takes a date and a
1010+key, and produces `~` (`0`) or `[~ value]`.
1011+
1012+`++keep` asks Arvo the next time it wants to be woken up, for the
1013+given `wire`. (This input will probably be eliminated in favor
1014+of a single global timer.)
1015+
1016+`++wish` compiles a string of Hoon source. While just a
1017+convenience, it's a very convenient convenience.
1018+
1019+`++come` and `++load` are used by Arvo to reset itself (more
1020+precisely, to shift the Arvo state from an old kernel to a new
1021+one); there is no need to call them from C.
1022+
1023+Now that we understand the Arvo kernel interface, let's look at
1024+the `u3v` API. As usual, all the functions in `u3v` are
1025+commented, but unfortunately it's hard to describe this API as
1026+clean at present. The problem is that `u3v` remains design
1027+coupled to the old `vere` event handling code written for `u2`.
1028+But let's describe the functions you should be calling, assuming
1029+you're not writing the next event system. There are only two.
1030+
1031+`u3v_wish(str_c)` wraps the `++wish` functionality in a cache
1032+(which is read-only unless you're on the surface road).
1033+
1034+`u3v_do()` uses `wish` to provide a convenient interface for
1035+calling Hoon kernel functions by name. Even more conveniently,
1036+we tend to call `u3v_do()` with these convenient aliases:
1037+
1038+```
1039+ #define u3do(txt_c, arg) u3v_do(txt_c, arg)
1040+ #define u3dc(txt_c, a, b) u3v_do(txt_c, u3nc(a, b))
1041+ #define u3dt(txt_c, a, b, c) u3v_do(txt_c, u3nt(a, b, c))
1042+ #define u3dq(txt_c, a, b, c, d) u3v_do(txt_c, u3nt(a, b, c, d))
1043+```