@@ -0,0 +1,602 @@
4+ title/'Arvo Internals'
8+Generators are the most straightforward way to write programs for Urbit. They
9+are used for doing computations that do not require persistence: they take an
10+input and produce an output, then disappear. Generators might make sense for
11+for listing directory contents, or running unit tests, or fetching the
14+There are four kinds of generators: naked, `%say`, `%get`, and `%ask`.
18+A naked generator is simply a `gate`; that is, an is an anonymous function that
19+takes a `sample` (argument) and produces a noun. All you need to do is write a
20+`gate` and put it into file in the `/gen` directory. Let's take a look at a very
28+This generator takes one argument of any noun and produces it without any
29+changes. If we put this into a file named `echo.hoon` in the `/gen` directory,
30+we can run it from the [Dojo](/docs/using/shell):
37+This command just pass in 42 and get 42 back. But what about when we pass in
45+We just get a pile of numbers back in the form of a raw `noun`. The numbers that
46+compose that noun are called `atoms`, and how they are interpreted is a matter
47+of what `aura` is being applied to them. For the purposes of this lesson, an
48+`aura` tells the pretty-printer how to display an `atom`, but keep in mind that
49+an `aura` is type metadata and does other things, too.
51+In the example above, we didn't specify an `aura`, leaving the printer to fend
52+for itself. `"asdf"` is a `tape`, a type that is simply a `list` of `cords`. A
53+`cord` is itself an atom represented as a string of UTF-8 characters. When used
54+as part of a `tape`, the `cord` is only a single character. So each atom in the
55+`[97 115 100 102 0]` output corresponds to a component of the `tape`: 97 is
56+`a`, 115 is `s`, 100 is `d`, 102 is `f`, and `0` is the "null" that every list
59+We can tell the Dojo to "cast" -- apply a specific type to -- the output of our
60+generator see something more familiar:
67+Now let's create a generator with two samples instead of one. Save the code
68+below as `add.hoon` in the `/gen` directory.
75+Now, run the generator:
82+You may notice our generator takes a cell containing two `@ud`. This is
83+actually one of the limitations of naked generators. We can only pass one
84+argument. We can get around this, of course, by passing a cell, but it's less
85+than ideal. Also, a naked generator cannot be called without an argument.
86+This brings us to `%say` generators.
90+We use `%say` generators when we want to provide something else in Arvo, the
91+Urbit operating system, with metadata about the generator's output. This is
92+useful when a generator is needed to pipe data to another program, a frequent
95+To that end, `%say` generators use `mark`s to make it clear, to other Arvo
96+computations, what exactly their kind of data their output is. A `mark` is akin
97+to a MIME type on the Arvo level. A `mark` describes the data in
98+some way, indicating that it's an `%atom`, or that it's a standard such as
99+`%json`, or even that it's an application-specific data structure like
100+`%talk-command`. `mark`s are not specific to `%say` generators; any time data
101+moves between programs in Arvo, that data is marked.
103+So, more formally, a `%say` generator is a `cell`. The head of that cell is the
104+`%say` tag, and the tail is a `gate` that produces a `cask` -- a pair of the
105+output data and the `mark` describing that data.
107+Below is an example of a `%say` generator. Save it to `add.hoon` in the `/gen`
117+Now run the generator as below:
124+Notice that we used no argument, something that is possible with `%say`
125+generators but impossible with naked generators. We'll explain that in a moment.
126+For now, let's focus on the code that is necessary to make something a `%say`
133+Recall that the rune `:-` produces a cell, with the first following expression
134+as its head and the second following expression as its tail.
136+The expression above, then, creates a cell with `%say` as the head. The tail is
137+the `|= *` expression on the line that follows.
145+`|= *` constructs a gate that takes a noun. This gate will itself produce a
146+`cask`, which is cell formed by the prepending `:-`. The head of that `cask` is
147+`%noun` and the tail is rest of the program, `(add 40 2)`. The tail of the
148+`cask` will be our actual data produced by body of the program: in this case
149+just adding 40 and 2 together.
151+#### `%say` generators with arguments
153+We can modify the boilerplate code to allow arguments to be passed into a `%say`
154+generator, but in a way that gives us more power than we would have if we just
155+used a naked generator.
157+Naked generators are limited because they have no way of accessing data that
158+exists in Arvo, such as the date and time or pieces of fresh entropy. In `%say`
159+generators, however, we can access that kind of subject by identifying them
160+in the gate's sample, which we only specified as `*` in the previous few
161+examples. But we can do more with `%say` generators if we do more with that
162+sample. Any valid sample will follow this 3-tuple scheme:
164+`[[datetime, entropy, and beak] [list of unnamed arguments] [list of named arguments]]`
166+This entire structure is a noun, which is why `*` is a valid sample if we
167+wish to not use any of the information here in a generator. But let's look at
168+each of these three elements, piece by piece.
170+The *first part* of the above 3-tuple is a noun that is composed of three
173+`now` is the current time.
174+`eny` is 512 bits of entropy for seeding random number generators.
175+`beak` contains the current ship, desk, and case.
177+You can access each of those pieces of data by typing their names into the Dojo.
178+But in a generator, we need put faces (variable names) onto that data so we can
179+easily use it in the program. We can do that like below:
182+|= [[now=@da eny=@uvJ bec=beak] ~ ~]
185+Any of those pieces of data could be omitted by replacing part of the noun with
186+`*` rather than giving them faces. For example, `[now=@da * bec=beak]` if we
187+didn't want `eny`, or `[* * bec=beak]` if we only wanted `beak`.
189+The *second part* of the sample is a `list` of arguments that _must_ be passed
190+to the generator as it's run. Because it's a `list`, this element needs to be
191+terminated with a `~`. In the example above, we used `~`, the empty list, to
192+represent this second part. But that would mean we don't want to use any new
193+faces for our generator. So a sample where we want to declare a single required
194+argument would look like this.
200+In the above code, we use a `*` to ignore any of the information that would go
201+first part of the sample, kind of in the same way that we use a `~` to say
202+"nothing here" for the second or third parts of the sample.
204+But to use both parts together, our gate and sample would look like this:
207+|= [[now=@da eny=@uvJ bec=beak] [n=@ud ~] ~]
210+The *third part* of the sample is a list of _optional_ arguments. These
211+arguments may be passed to the program as its being run, but the program doesn't
212+require them. The syntax for the third part is just like the syntax for the
213+second part, besides its position:
219+Let's look at an example that uses all three parts. Save the code below in a
220+file called `dice.hoon` in your `/gen` directory.
224+|= [[now=@da eny=@uvJ bec=beak] [n=@ud ~] [bet=@ud ~]]
226+[(~(rad og eny) n) bet]
229+This is a very simple dice program with an optional betting functionality.
230+In the code, our sample specifies faces on all the all of the Arvo data, meaning
231+that we can can easily access them. We also require the argument `[n=@ud ~]`,
232+and allow the _optional_ argument `[bet=@ud ~]`.
234+But there's something new: `(~(rad og eny) n)`. This code pulls the `rad` arm
235+out of the `og` core with the subject of `eny`. Recall that `eny` is our entropy
236+value, so this is used to seed the generator. The `rad` arm will give us a
237+pseudorandom number between 0 and `n`, and the `bet` that it's in a cell with
238+will just return the argument that we might choose to give it.
240+We can run this generator like so:
259+We get a different value from the same generator between runs, something that
260+isn't possible with a naked generator. Another novelty is the ability to choose
261+to not use the second argument.
263+##### Arguments without a cell
265+Also unlike a naked generator, we don't need to put our arguments together into
266+a cell. Swap the code below into your `add.hoon` file.
270+|= [* [a=@ud b=@ud ~] ~]
275+In the above code we're again creating a `%say` generator, but the sample of our
276+gate is a little different than before. We are using `*` for the first part of
277+the sample, because we don't use any Arvo data in this program. We are using a
278+`~` for the third part, because we aren't allowing optional arguments -- the
279+list of optional arguments is _empty_.
281+Run a command like the one below in the Dojo. Notice that you can use two
282+arguments that aren't in the cells.
289+### `%ask` generators
291+We use the `%ask` generator when we want to create an interactive program
292+that prompts for inputs as it runs, rather than expecting arguments to be passed
293+in at the time of initiation.
295+Code-wise, you need two things to create a useful `%ask` generator.
297+The first is *`sole-result`*, which lives in `/sur/sole.hoon`. It's a
298+mold-builder that's required by any part of Arvo, such as the Dojo, that wants
299+to understand the output of the `%ask` generator. Because it's the only type of
300+`%ask` data that is intelligible to the system, the ultimate output of an `%ask`
301+generator must be `sole-result`. You can think of it as a function that takes
302+molds and digests them into a type for the operating system. The simplest way to
303+use `sole-result` is by calling the `produce` gate from `/lib/generators.hoon`.
305+The second is *`prompt`*, a gate that lives in `lib/generators.hoon`. `prompt`
306+allows you to get get information back from the user, which is essential to the
307+typical desired operation of an `%ask` generator.
309+Since this isn't easy to digest in the abstract, we'll use an example generator
310+to explain how this kind of generator works.
314+The code below is an `%ask` generator that checks if the user inputs "blue" when
315+prompted. Save it as `axe.hoon` in `/gen`.
323+^- (sole-result (cask tang))
324+%+ print leaf+"What is your favorite color?"
325+%+ prompt [%& %prompt "color: "]
329+ :~ leaf+"Oh. Thank you very much."
330+ leaf+"Right. Off you go then."
333+ leaf+"Into the Gorge of Eternal Peril with you!"
337+Run the generator from the Dojo:
341+What is your favorite color?
345+Something new happened. Instead of simply returning something, your Dojo's
346+prompt changed from `~your-urbit:dojo>` to `~your-urbit:dojo: color:`, and
347+expects additional input. Let's give it an input:
351+Into the Gorge of Eternal Peril with you!
355+Let's go over what exactly is happening in this code.
363+Here we bring in some of the types we are going to need from `/sur/sole` and
364+gates we will use from `/lib/generators`. We use some special runes for this.
366+`/-` is a Ford rune used to import types from `sur`.
368+`/+` is a Ford rune used to import libraries from `lib`.
370+`=,` is a rune that allows us to expose a namespace. We do this to avoid having
371+to write `sole-result:sole` instead of `sole-result` or `print:generators`
379+This code might be familiar. Just as with their `%say` cousins, `%ask`
380+generators need to produce a `cell`, the head of which specifies what kind of
381+generator we are running.
383+With `|= *`, we create a gate and ignore the standard arguments we are given,
384+because we're not using them.
387+^- (sole-result (cask tang))
390+`%ask` generators need to have the second half of the cell be a gate that
391+produces a `sole-result`, one that in this case contains a `cask` of `tang`.
392+We use the `^-` rune to constrain the generator's output to such a
395+A `cask` is a pair of a `mark` name and a noun. Recall that a `mark` can be
396+thought of as an Arvo-level MIME type for data.
398+A `tang` is a `list` of `tank` and a `tank` is a structure for printing data
399+that there are three types of: `leaf`, `palm`, and `rose`. A `leaf` is for
400+printing a single noun, a `rose` is for printing rows of data, and a `palm` is
401+printing backstep-indented lists.
404+%+ print leaf+"What is your favorite color?"
405+%+ prompt [%& %prompt "color: "]
406+|= t=tape :: we will come to this later
411+Because we imported `generators`, we can access its contained gates, three of
412+which we use in `axe.hoon`: `print`, `prompt`, and `produce`.
414+*`print`* is used for printing a `tank` to the console.
416+In our example, `%+` is the rune to call a gate, and our gate `print` takes one
417+argument which is a `tank` to print. The `+` here is syntactic sugar for
418+`[leaf "What is your favorite color?"]` that just makes it easier to write.
420+*`prompt`* is used to construct a prompt for the user to provide input.
421+It takes a single argument that is a tuple. Most `%ask` generators will want to
422+use the `prompt` gate.
424+The first element of the `prompt` sample is a flag that indicates what the
425+user typed should be echoed out to them or hidden. `%&` will produce echoed
426+output and `%|` will hide the output (for use in passwords or other secret
429+The second element of the `prompt` sample is a `%term` used The second argument
430+is intended to be information to use to create autocomplete options for the
431+prompt this is not yet implemented.
433+The third element of the `prompt` sample is the `tape` that we would like to
434+use to prompt the user. In the case of our example, we use `"color: "`.
436+*`produce`* is is used to construct the output of the generator. In our
437+example, we produce a `tang`.
444+Our gate here takes a `tape` that was produced by `prompt` if we needed another
445+type of data we could use `parse` to obtain it.
447+The rest of this generator should be intelligible to those with Hoon knowledge
450+One quirk that you should be aware of, though, is that `tang` prints in reverse
451+order from how it is created. The reason for this is that `tang` was originally
452+created to display stack trace information, which should be produced in
453+in reverse order. This leads to an annoyance: we either have to specify our
454+messages backwards or construct them in the order we want and then `flop` the
455+`list` This is a [known issue](https://github.com/urbit/arvo/issues/840) to be
458+### `%get` generators
460+`%get` generators are used for making HTTP requests through `eyre`. For this
461+section we'll assume knowledge of generators covered in previous sections, so
462+we can jump in with an example:
469+|= [* [url=tape ~] ~]
470+^- (sole-request:sole (cask json))
471+%+ curl (scan url auri:de-purl:html)
474+=/ my-json (de-json:html q.u.r.hit)
484+The above generator was written with the
485+[Studio Ghibli API](https://ghibliapi.herokuapp.com/). The generator below was
486+designed to use the this API to print the title of a Studio Ghibli film.
494+Here we import types from `/sur/sole.hoon` and gates from
495+`/lib/generators.hoon`. We also use `=,` to exposes the namespace of
496+`generators` that we just imported.
500+|= [* [url=tape ~] ~]
503+Like with `%say` and `%ask` generators, we must have include the "boilerplate":
504+code that produces a generator-specific `cell`. At the head of that cell is
505+`%get`, and the tail is a gate that takes a `tape` of the URL that we are trying
509+^- (sole-request:sole (cask json))
512+Here we make sure our gate is producing a `sole-request` which contains a `cask`
517+%+ curl (scan url auri:de-purl:html)
521+Here we can see the call to `curl`, the sample of which will parse our url and
522+make sure it's in the proper format.
524+The second part of the `curl` call is the `gate` that will get called with the
525+result provided by the actual HTTP request.
531+Here we verify that the `r` face of our response is not null before
532+continuing. If it is null, the generator will crash, as there is nothing left to
536+=/ my-json (de-json:html q.u.r.hit)
539+Here we verify that data was returned to us in the `json` Hoon type.
540+`q.u.r.hit` is the actual data we care about as a `cord`. `hit` is actually an
541+entire response object, including headers, which we don't care about for this
542+example. `de-json` out of `html` will parse the `cord` and produce a `json` type
543+for us that we can then use to inspect the data.
550+After parsing the JSON structure we want to make sure we got back a valid `json`
551+type. To do that, we test for null and crash that test passes. If not, we use
552+`=,` on the `dejs:format` core so that we can more easily pull arms out of it.
560+Now we come to our `produce` call. Recall that `produce` is used in all the
561+non-naked generators to output the final result. Here we want to produce a
562+`cask` of `json`, so we use the mark `%json` to indicate that.
564+Next we have what may be an unfamiliar rune: `%.`. This rune is simply `%-` with
565+the argument order reversed. `%title` is the argument passed to the next gate
566+that is called. `by` is the core that is the `map` engine. A Hoon `map` is a
567+key-value pair structure, sometimes referred to as a dictionary in other
568+languages. `got` is the arm in `by` that will produce a gate that we can call to
569+access members of a given `map`.
571+> The idiomatic way to write Hoon is to have the
572+ "heaviest" code paths lowest in the program, which is why `%.`
573+ is preferred in this case over `%-`. Use of this style is recommended, as it
574+ results in code that is much easier to read.
576+At the moment, however, we do not have a `map`, but instead have a `json` type.
583+Here we use `om`, which is an arm in `dejs:format` that will take a `json` and
584+return a `map`. `same` is a wet gate from the standard library that takes an
585+argument of any type and returns it as that type. We could use any other gate to
586+modify the data if that were useful for the particular data we are going to be
587+processing, but here we want the data exactly as it is.
589+Since `my-json` is a `unit` of `json`, the actual data we are looking for is in
590+`u.my-json`. We take this and give it to the gate produced by the call to `om`.
591+This gate is going to transform our data by taking in the `json` structure and
592+producing a `map`. Then we can look up keys in this map using `got` from
595+`%.` performs the same function as `%-`, the rune used to call a function,
596+except with the arguments reversed so that the function is last and
597+the arguments of that function come first. It's used here to keep with the
598+principle that heaviest, or most complex, expressions should go to the bottom.
600+The key we are looking up in our new `map` we've created from `my-json` is
601+`%title` which is an element in the original source to print out to produce as
602+the final result of the generator.