@@ -1,606 +1,9 @@
11---
22navhome: /docs/
3−sort: 2
4−title: C runtime system
3+sort: 4
4+title: API overview by prefix
55---
66
7−# C runtime system
8−
9−The Urbit interpreter is built on a Nock runtime system written
10−in C, `u3`. This document is a relatively complete description
11−
12−You should keep reading if (a) you're planning to work on the
13−Urbit interpreter; (b) you're a language implementation geek; or
14−(c) you don't really understand anything until you've seen the
15−actual structs.
16−
17−## u3: noun processing in C
18−
19−`u3` is the C library that makes Urbit work. If it wasn't called
20−`u3`, it might be called `libnoun` - it's a library for making
21−and storing nouns.
22−
23−What's a noun? A noun is either a cell or an atom. A cell is an
24−ordered pair of any two nouns. An atom is an unsigned integer of
25−any size.
26−
27−To the C programmer, this is not a terribly complicated data
28−structure, so why do you need a library for it?
29−
30−One: nouns have a well-defined computation kernel, Nock, whose
31−spec fits on a page and gzips to 340 bytes. But the only
32−arithmetic operation in Nock is increment. So it's nontrivial
33−to compute both efficiently and correctly.
34−
35−Two: `u3` is designed to be a "solid-state interpreter," ie, a
36−single-level store which is transparently snapshotted. This
37−implies a specialized memory-management model, etc, etc.
38−
39−(Does `u3` depend on the higher levels of Urbit, Arvo and Hoon?
40−Yes and no. `u3` expects you to load something shaped like an
41−Arvo kernel, and use it as an event-processing function. But you
42−don't need to use this feature if you don't want, and your kernel
43−doesn't have to be Arvo proper - just Arvo-compatible. Think of
44−`u3` as the BIOS and Arvo as the boot kernel. And there are no
45−dependencies at all between Hoon the language and `u3`.)
46−
47−## c3: C in Urbit
48−
49−Under `u3` is the simple `c3` layer, which is just how we write C
50−in Urbit.
51−
52−When writing C in u3, please of course follow the conventions of
53−the code around you as regards indentation, etc. It's especially
54−important that every function have a header comment, even if it
55−says nothing interesting.
56−
57−But some of our idiosyncrasies go beyond convention. Yes, we've
58−done awful things to C. Here's what we did and why we did.
59−
60−### c3: integer types
61−
62−First, it's generally acknowledged that underspecified integer
63−types are C's worst disaster. C99 fixed this, but the `stdint`
64−types are wordy and annoying. We've replaced them with:
65−
66− /* Good integers.
67− */
68− typedef uint64_t c3_d; // double-word
69− typedef int64_t c3_ds; // signed double-word
70− typedef uint32_t c3_w; // word
71− typedef int32_t c3_ws; // signed word
72− typedef uint16_t c3_s; // short
73− typedef int16_t c3_ss; // signed short
74− typedef uint8_t c3_y; // byte
75− typedef int8_t c3_ys; // signed byte
76− typedef uint8_t c3_b; // bit
77−
78− typedef uint8_t c3_t; // boolean
79− typedef uint8_t c3_o; // loobean
80− typedef uint8_t c3_g; // 5-bit atom for a 32-bit log.
81− typedef uint32_t c3_l; // little; 31-bit unsigned integer
82− typedef uint32_t c3_m; // mote; also c3_l; LSB first a-z 4-char string.
83−
84− /* Bad integers.
85− */
86− typedef char c3_c; // does not match int8_t or uint8_t
87− typedef int c3_i; // int - really bad
88− typedef uintptr_t c3_p; // pointer-length uint - really really bad
89− typedef intptr_t c3_ps; // pointer-length int - really really bad
90−
91−Some of these need explanation. A loobean is a Nock boolean -
92−Nock, for mysterious reasons, uses 0 as true (always say "yes")
93−and 1 as false (always say "no").
94−
95−Nock and/or Hoon cannot tell the difference between a short atom
96−and a long one, but at the `u3` level every atom under `2^31` is
97−direct. The `c3_l` type is useful to annotate this. A `c3_m` is
98−a *mote* - a string of up to 4 characters in a `c3_l`, least
99−significant byte first. A `c3_g` should be a 5-bit atom. Of
100−course, C cannot enforce these constraints, only document them.
101−
102−Use the "bad" - ie, poorly specified - integer types only when
103−interfacing with external code that expects them.
104−
105−An enormous number of motes are defined in `i/c/motes.h`. There
106−is no reason to delete motes that aren't being used, or even to
107−modularize the definitions. Keep them alphabetical, though.
108−
109−### c3: variables and variable naming
110−
111−The C3 style uses Hoon style TLV variable names, with a quasi
112−Hungarian syntax. This is weird, but works really well, as long
113−as what you're doing isn't hideously complicated. (Then it works
114−badly, but we shouldn't need anything hideous in u3.)
115−
116−A TLV variable name is a random pronounceable three-letter
117−string, sometimes with some vague relationship to its meaning,
118−but usually not. Usually CVC (consonant-vowel-consonant) is a
119−good choice.
120−
121−You should use TLVs much the way math people use Greek letters.
122−The same concept should in general get the same name across
123−different contexts. When you're working in a given area, you'll
124−tend to remember the binding from TLV to concept by sheer power
125−of associative memory. When you come back to it, it's not that
126−hard to relearn. And of course, when in doubt, comment it.
127−
128−Variables take pseudo-Hungarian suffixes, matching in general the
129−suffix of the integer type:
130−
131− c3_w wor_w; // 32-bit word
132−
133−Unlike in standard Hungarian, there is no change for pointer
134−variables. C structure variables take a `_u` suffix.
135−
136−### c3: loobeans
137−
138−The code (from `defs.h`) tells the story:
139−
140− # define c3y 0
141− # define c3n 1
142−
143− # define _(x) (c3y == (x))
144− # define __(x) ((x) ? c3y : c3n)
145− # define c3a(x, y) __(_(x) && _(y))
146− # define c3o(x, y) __(_(x) || _(y))
147−
148−In short, use `_()` to turn a loobean into a boolean, `__` to go
149−the other way. Use `!` as usual, `c3y` for yes and `c3n` for no,
150−`c3a` for and and `c3o` for or.
151−
152−## u3: land of nouns
153−
154−The division between `c3` and `u3` is that you could theoretically
155−imagine using `c3` as just a generic C environment. Anything to do
156−with nouns is in `u3`.
157−
158−### u3: a map of the system
159−
160−There are two kinds of symbols in `u3`: regular and irregular.
161−Regular symbols follow this pattern:
162−
163− prefix purpose .h .c
164− -------------------------------------------------------
165− u3a_ allocation i/n/a.h n/a.c
166− u3e_ persistence i/n/e.h n/e.c
167− u3h_ hashtables i/n/h.h n/h.c
168− u3i_ noun construction i/n/i.h n/i.c
169− u3j_ jet control i/n/j.h n/j.c
170− u3m_ system management i/n/m.h n/m.c
171− u3n_ nock computation i/n/n.h n/n.c
172− u3r_ noun access (error returns) i/n/r.h n/r.c
173− u3t_ profiling i/n/t.h n/t.c
174− u3v_ arvo i/n/v.h n/v.c
175− u3x_ noun access (error crashes) i/n/x.h n/x.c
176− u3z_ memoization i/n/z.h n/z.c
177− u3k[a-g] jets (transfer, C args) i/j/k.h j/[a-g]/*.c
178− u3q[a-g] jets (retain, C args) i/j/q.h j/[a-g]/*.c
179− u3w[a-g] jets (retain, nock core) i/j/w.h j/[a-g]/*.c
180−
181−Irregular symbols always start with `u3` and obey no other rules.
182−They're defined in `i/n/u.h`. Finally, `i/all.h` includes all
183−these headers (fast compilers, yay) and is all you need to
184−program in `u3`.
185−
186−### u3: noun internals
187−
188−A noun is a `u3_noun` - currently defined as a 32-bit `c3_w`.
189−
190−If your `u3_noun` is less than `(1 << 31)`, it's a direct atom.
191−Every unsigned integer between `0` and `0x7fffffff` inclusive is
192−its own noun.
193−
194−If bit `31` is set in a `u3_noun`, bit `30` is always set - this
195−bit is reserved. Bit `29` is `1` if the noun is a cell, `0` if
196−it's an atom. Bits `28` through `0` are a word pointer into the
197−loom - see below. The structures are:
198−
199− typedef struct {
200− c3_w mug_w;
201− c3_w len_w;
202− c3_w buf_w[0]; // actually [len_w]
203− } u3a_atom;
204−
205− typedef struct {
206− c3_w mug_w;
207− u3_noun hed;
208− u3_noun tel;
209− } u3a_cell;
210−
211−The only thing that should be mysterious here is `mug_w`, which
212−is a 31-bit lazily computed nonzero short hash (FNV currently,
213−soon Murmur3). If `mug_w` is 0, the hash is not yet computed.
214−We also hijack this field for various hacks, such as saving the
215−new address of a noun when copying over.
216−
217−Also, the value `0xffffffff` is `u3_none`, which is never a valid
218−noun. Use the type `u3_weak` to express that a noun variable may
219−be `u3_none`.
220−
221−### u3: reference counts
222−
223−The only really essential thing you need to know about `u3` is
224−how to handle reference counts. Everything else, you can skip
225−and just get to work.
226−
227−u3 deals with reference-counted, immutable, acyclic nouns.
228−Unfortunately, we are not Apple and can't build reference
229−counting into your C compiler, so you need to count by hand.
230−
231−Every allocated noun (or any allocation object, because our
232−allocator is general-purpose) contains a counter which counts the
233−number of references to it - typically variables with type
234−`u3_noun`. When this counter goes to 0, the noun is freed.
235−
236−To tell `u3` that you've added a reference to a noun, call the
237−function `u3a_gain()` or its shorthand `u3k()`. (For your
238−convenience, this function returns its argument.) To tell `u3`
239−that you've destroyed a reference, call `u3a_lose()` or `u3z()`.
240−
241−(If you screw up by decrementing the counter too much, `u3` will
242−dump core in horrible ways. If you screw up by incrementing it
243−too much, `u3` will leak memory. To check for memory leaks,
244−set the `bug_o` flag in `u3e_boot()` - eg, run `vere` with `-g`.
245−Memory leaks are difficult to debug - the best way to handle
246−leaks is just to revert to a version that didn't have them, and
247−look over your code again.)
248−
249−(You can gain or lose a direct atom. It does nothing.)
250−
251−### u3: reference protocols
252−
253−*THIS IS THE MOST CRITICAL SECTION IN THE `u3` DOCUMENTATION.*
254−
255−The key question when calling a C function in a refcounted world
256−is what the function will do to the noun refcounts - and, if the
257−function returns a noun, what it does to the return.
258−
259−There are two semantic patterns, `transfer` and `retain`. In
260−`transfer` semantics, the caller "gives" a use count to the
261−callee, which "gives back" any return. For instance, if I have
262−
263− {
264− u3_noun foo = u3i_string("foobar");
265− u3_noun bar;
266−
267− bar = u3f_futz(foo);
268− [...]
269− u3z(bar);
270− }
271−
272−Suppose `u3f_futz()` has `transfer` semantics. At `[...]`, my
273−code holds one reference to `bar` and zero references to `foo` -
274−which has been freed, unless it's part of `bar`. My code now
275−owns `bar` and gets to work with it until it's done, at which
276−point a `u3z()` is required.
277−
278−On the other hand, if `u3f_futz()` has `retain` semantics, we
279−need to write
280−
281− {
282− u3_noun foo = u3i_string("foobar");
283− u3_noun bar;
284−
285− bar = u3f_futz(foo);
286− [...]
287− u3z(foo);
288− }
289−
290−because calling `u3f_futz()` does not release our ownership of
291−`foo`, which we have to free ourselves.
292−
293−But if we free `bar`, we are making a great mistake, because our
294−reference to it is not in any way registered in the memory
295−manager (which cannot track references in C variables, of
296−course). It is normal and healthy to have these uncounted
297−C references, but they must be treated with care.
298−
299−The bottom line is that it's essential for the caller to know
300−the refcount semantics of any function which takes or returns a
301−noun. (In some unusual circumstances, different arguments or
302−returns in one function may be handled differently.)
303−
304−Broadly speaking, as a design question, retain semantics are more
305−appropriate for functions which inspect or query nouns. For
306−instance, `u3h()` (which takes the head of a noun) retains, so
307−that we can traverse a noun tree without constantly incrementing
308−and decrementing.
309−
310−Transfer semantics are more appropriate for functions which make
311−nouns, which is obviously what most functions do.
312−
313−In general, though, in most places it's not worth thinking about
314−what your function does. There is a convention for it, which
315−depends on where it is, not what it does. Follow the convention.
316−
317−### u3: reference conventions
318−
319−The `u3` convention is that, unless otherwise specified, *all
320−functions have transfer semantics* - with the exception of the
321−prefixes: `u3r`, `u3x`, `u3z`, `u3q` and `u3w`. Also, within
322−jet directories `a` through `f` (but not `g`), internal functions
323−retain (for historical reasons).
324−
325−If functions outside this set have retain semantics, they need to
326−be commented, both in the `.h` and `.c` file, with `RETAIN` in
327−all caps. Yes, it's this important.
328−
329−### u3: system architecture
330−
331−If you just want to tinker with some existing code, it might be
332−enough to understand the above. If not, it's probably worth
333−taking the time to look at `u3` as a whole.
334−
335−`u3` is designed to work as a persistent event processor.
336−Logically, it computes a function of the form
337−
338− f(event, old state) -> (actions, new state)
339−
340−Obviously almost any computing model - including, but not limited
341−to, Urbit - can be defined in this form. To create the illusion
342−of a computer that never loses state and never fails, we:
343−
344−- log every event externally before it goes into u3
345−- keep a single reference to a permanent state noun.
346−- can abort any event without damaging the permanent state.
347−- snapshot the permanent state periodically, and/or prune logs.
348−
349−### u3: the road model
350−
351−`u3` uses a memory design which I'm sure someone has invented
352−somewhere before, because it's not very clever, but I've never
353−seen it anywhere in particular.
354−
355−Every allocation starts with a solid block of memory, which `u3`
356−calls the `loom`. How do we allocate on the loom? You're
357−probably familiar with the Unix heap-stack design, in which the
358−stack grows downward and the heap (malloc arena) grows upward:
359−
360− 0 brk ffff
361− | heap | stack |
362− |------------#################################+++++++++++++|
363− | | |
364− 0 sp ffff
365−
366−A road is a normal heap-stack system, except that the heap
367−and stack can point in *either direction*. Therefore, inside
368−a road, we can nest another road in the *opposite direction*.
369−
370−When the opposite road completes, its heap is left on top of
371−the opposite heap's stack. It's no more than the normal
372−behavior of a stack machine for all subcomputations to push
373−their results on the stack.
374−
375−The performance tradeoff of "leaping" - reversing directions in
376−the road - is that if the outer computation wants to preserve the
377−results of the inner one, not just use them for temporary
378−purposes, it has to *copy them*.
379−
380−This is a trivial cost in some cases, a prohibitive cost in
381−others. The upside, of course, is that all garbage accrued
382−in the inner computation is discarded at zero cost.
383−
384−The goal of the road system is the ability to *layer* memory
385−models. If you are allocating on a road, you have no idea
386−how deep within a nested road system you are - in other words,
387−you have no idea exactly how durable your result may be.
388−But free space is never fragmented within a road.
389−
390−Roads do not reduce the generality or performance of a memory
391−system, since even the most complex GC system can be nested
392−within a road at no particular loss of performance - a road
393−is just a block of memory.
394−
395−Each road (`u3a_road` to be exact) uses four pointers: `rut` is
396−the bottom of the arena, `hat` the top of the arena, `mat` the
397−bottom of the stack, `cap` the top of the stack. (Bear in mind
398−that the road "stack" is not actually used as the C function-call
399−stack, though it probably should be.)
400−
401−A "north" road has the stack high and the heap low:
402−
403− 0 rut hat ffff
404− | | | |
405− |~~~~~~~~~~~~-------##########################+++++++$~~~~~|
406− | | | |
407− 0 cap mat ffff
408−
409−A "south" road is the other way around:
410−
411− 0 mat cap ffff
412− | | | |
413− |~~~~~~~~~~~~$++++++##########################--------~~~~~|
414− | | | |
415− 0 hat rut ffff
416−
417−Legend: `-` is durable storage (heap); `+` is temporary storage
418−(stack); `~` is deep storage (immutable); `$` is the allocation
419−frame `#` is free memory.
420−
421−Pointer restrictions: pointers stored in `+` can point anywhere.
422−Pointers in `-` can only point to `-` or `~`; pointers in `~`
423−only point to `~`.
424−
425−To "leap" is to create a new inner road in the `###` free space.
426−but in the reverse direction, so that when the inner road
427−"falls" (terminates), its durable storage is left on the
428−temporary storage of the outer road.
429−
430−`u3` keeps a global variable, `u3_Road` or its alias `u3R`, which
431−points to the current road. (If we ever run threads in inner
432−roads - see below - this will become a thread-local variable.)
433−Relative to `u3R`, `+` memory is called `junior` memory; `-`
434−memory is `normal` memory; `~` is `senior` memory.
435−
436−### u3: explaining the road model
437−
438−But... why?
439−
440−We're now ready to understand why the road system works so
441−logically with the event and persistence model.
442−
443−The key is that *we don't update refcounts in senior memory.*
444−A pointer from an inner road to an outer road is not counted.
445−Also, the outmost, or `surface` road, is the only part of the
446−image that gets checkpointed.
447−
448−So the surface road contains the entire durable state of `u3`.
449−When we process an event, or perform any kind of complicated or
450−interesting calculation, *we process it in an inner road*. If
451−its results are saved, they need to be copied.
452−
453−Since processing in an inner road does not touch surface memory,
454−(a) we can leave the surface road in a read-only state and not
455−mark its pages dirty; (b) we can abort an inner calculation
456−without screwing up the surface; and (c) because inner results
457−are copied onto the surface, the surface doesn't get fragmented.
458−
459−All of (a), (b) and (c) are needed for checkpointing to be easy.
460−It might be tractable otherwise, but easy is even better.
461−
462−Moreover, while the surface is most definitely single-threaded,
463−we could easily run multiple threads in multiple inner roads
464−(as long as the threads don't have pointers into each others'
465−memory, which they obviously shouldn't).
466−
467−Moreover, in future, we'll experiment more with adding road
468−control hints to the programmer's toolbox. Reference counting is
469−expensive. We hypothesize that in many - if not most - cases,
470−the programmer can identify procedural structures whose garbage
471−should be discarded in one step by copying the results. Then,
472−within the procedure, we can switch the allocator into `sand`
473−mode, and stop tracking references at all.
474−
475−### u3: rules for C programming
476−
477−There are two levels at which we program in C: (1) above the
478−interpreter; (2) within the interpreter or jets. These have
479−separate rules which need to be respected.
480−
481−### u3: rules above the interpreter
482−
483−In its relations with Unix, Urbit follows a strict rule of "call
484−me, I won't call you." We do of course call Unix system calls,
485−but only for the purpose of actually computing.
486−
487−Above Urbit, you are in a normal C/Unix programming environment
488−and can call anything in or out of Urbit. Note that when using
489−`u3`, you're always on the surface road, which is not thread-safe
490−by default. Generally speaking, `u3` is designed to support
491−event-oriented, single-threaded programming.
492−
493−If you need threads which create nouns, you could use
494−`u3m_hate()` and `u3m_love()` to run these threads in subroads.
495−You'd need to make the global road pointer, `u3R`, a thread-local
496−variable instead. This seems perfectly practical, but we haven't
497−done it because we haven't needed to.
498−
499−### u3: rules within the interpreter
500−
501−Within the interpreter, your code can run either in the surface
502−road or in a deep road. You can test this by testing
503−
504− (&u3H->rod_u == u3R)
505−
506−ie: does the pier's home road equal the current road pointer?
507−
508−Normally in this context you assume you're obeying the rules of
509−running on an inner road, ie, "deep memory." Remember, however,
510−that the interpreter *can* run on surface memory - but anything
511−you can do deep, you can do on the surface. The converse is by
512−no means the case.
513−
514−In deep memory, think of yourself as if in a signal handler.
515−Your execution context is extremely fragile and may be terminated
516−without warning or cleanup at any time (for instance, by `ctrl-c`).
517−
518−For instance, you can't call `malloc` (or C++ `new`) in your C
519−code, because you don't have the right to modify data structures
520−at the global level, and will leave them in an inconsistent state
521−if your inner road gets terminated. (Instead, use our drop-in
522−replacements, `u3a_malloc()`, `u3a_free()`, `u3a_realloc()`.)
523−
524−A good example is the different meaning of `c3_assert()` inside
525−and outside the interpreter. At either layer, you can use
526−regular assert(), which will just kill your process. On the
527−surface, `c3_assert()` will just... kill your process.
528−
529−In deep execution, `c3_assert()` will issue an exception that
530−queues an error event, complete with trace stack, on the Arvo
531−event queue. Let's see how this happens.
532−
533−### u3: exceptions
534−
535−You produce an exception with
536−
537− /* u3m_bail(): bail out. Does not return.
538− **
539− ** Bail motes:
540− **
541− ** %exit :: semantic failure
542− ** %evil :: bad crypto
543− ** %intr :: interrupt
544− ** %fail :: execution failure
545− ** %foul :: assert failure
546− ** %need :: network block
547− ** %meme :: out of memory
548− ** %time :: timed out
549− ** %oops :: assertion failure
550− */
551− c3_i
552− u3m_bail(c3_m how_m);
553−
554−Broadly speaking, there are two classes of exception: internal
555−and external. An external exception begins in a Unix signal
556−handler. An internal exception begins with a call to longjmp()
557−on the main thread.
558−
559−There are also two kinds of exception: mild and severe. An
560−external exception is always severe. An internal exception is
561−normally mild, but some (like `c3__oops`, generated by
562−`c3_assert()`) are severe.
563−
564−Either way, exceptions come with a stack trace. The `u3` nock
565−interpreter is instrumented to retain stack trace hints and
566−produce them as a printable `(list tank)`.
567−
568−Mild exceptions are caught by the first virtualization layer and
569−returned to the caller, following the behavior of the Nock
570−virtualizer `++mock` (in `hoon.hoon`)
571−
572−Severe exceptions, or mild exceptions at the surface, terminate
573−the entire execution stack at any depth and send the cumulative
574−trace back to the `u3` caller.
575−
576−For instance, `vere` uses this trace to construct a `%crud`
577−event, which conveys our trace back toward the Arvo context where
578−it crashed. This lets any UI component anywhere, even on a
579−remote node, render the stacktrace as a consequence of the user's
580−action - even if its its direct cause was (for instance) a Unix
581−SIGINT or SIGALRM.
582−
583−### u3: C structures on the loom
584−
585−Normally, all data on the loom is nouns. Sometimes we break this
586−rule just a little, though - eg, in the `u3h` hashtables.
587−
588−To point to non-noun C structs on the loom, we use a `u3_post`,
589−which is just a loom word offset. A macro lets us declare this
590−as if it was a pointer:
591−
592− typedef c3_w u3_post;
593− #define u3p(type) u3_post
594−
595−Some may regard this as clever, others as pointless. Anyway, use
596−`u3to()` and `u3of()` to convert to and from pointers.
597−
598−When using C structs on the loom - generally a bad idea - make
599−sure anything which could be on the surface road is structurally
600−portable, eg, won't change size when the pointer size changes.
601−(Note also: we consider little-endian, rightly or wrongly, to
602−have won the endian wars.)
603−
6047## u3: API overview by prefix
6058
6069Let's run through the `u3` modules one by one. All public