@@ -0,0 +1,602 @@
1+:- :~ navhome/'/docs/'
2+ next/'true'
3+ sort/'9'
4+ title/'Arvo Internals'
5+ ==
6+;>
7+
8+Generators are the most straightforward way to write programs for Urbit. They
9+are used for doing computations that do not require persistence: they take an
10+input and produce an output, then disappear. Generators might make sense for
11+for listing directory contents, or running unit tests, or fetching the
12+contents of a URL.
13+
14+There are four kinds of generators: naked, `%say`, `%get`, and `%ask`.
15+
16+### Naked Generators
17+
18+A naked generator is simply a `gate`; that is, an is an anonymous function that
19+takes a `sample` (argument) and produces a noun. All you need to do is write a
20+`gate` and put it into file in the `/gen` directory. Let's take a look at a very
21+simple one:
22+
23+```
24+|= a=*
25+a
26+```
27+
28+This generator takes one argument of any noun and produces it without any
29+changes. If we put this into a file named `echo.hoon` in the `/gen` directory,
30+we can run it from the [Dojo](/docs/using/shell):
31+
32+```
33+> +echo 42
34+42
35+```
36+
37+This command just pass in 42 and get 42 back. But what about when we pass in
38+`"asdf"`?
39+
40+```
41+> +echo "asdf"
42+[97 115 100 102 0]
43+```
44+
45+We just get a pile of numbers back in the form of a raw `noun`. The numbers that
46+compose that noun are called `atoms`, and how they are interpreted is a matter
47+of what `aura` is being applied to them. For the purposes of this lesson, an
48+`aura` tells the pretty-printer how to display an `atom`, but keep in mind that
49+an `aura` is type metadata and does other things, too.
50+
51+In the example above, we didn't specify an `aura`, leaving the printer to fend
52+for itself. `"asdf"` is a `tape`, a type that is simply a `list` of `cords`. A
53+`cord` is itself an atom represented as a string of UTF-8 characters. When used
54+as part of a `tape`, the `cord` is only a single character. So each atom in the
55+`[97 115 100 102 0]` output corresponds to a component of the `tape`: 97 is
56+`a`, 115 is `s`, 100 is `d`, 102 is `f`, and `0` is the "null" that every list
57+is terminated with.
58+
59+We can tell the Dojo to "cast" -- apply a specific type to -- the output of our
60+generator see something more familiar:
61+
62+```
63+> _tape +echo "asdf"
64+"asdf"
65+```
66+
67+Now let's create a generator with two samples instead of one. Save the code
68+below as `add.hoon` in the `/gen` directory.
69+
70+```
71+|= [a=@ud b=@ud]
72+(add a b)
73+```
74+
75+Now, run the generator:
76+
77+```
78+> +add [3 4]
79+7
80+```
81+
82+You may notice our generator takes a cell containing two `@ud`. This is
83+actually one of the limitations of naked generators. We can only pass one
84+argument. We can get around this, of course, by passing a cell, but it's less
85+than ideal. Also, a naked generator cannot be called without an argument.
86+This brings us to `%say` generators.
87+
88+### `%say` Generators
89+
90+We use `%say` generators when we want to provide something else in Arvo, the
91+Urbit operating system, with metadata about the generator's output. This is
92+useful when a generator is needed to pipe data to another program, a frequent
93+occurrence.
94+
95+To that end, `%say` generators use `mark`s to make it clear, to other Arvo
96+computations, what exactly their kind of data their output is. A `mark` is akin
97+to a MIME type on the Arvo level. A `mark` describes the data in
98+some way, indicating that it's an `%atom`, or that it's a standard such as
99+`%json`, or even that it's an application-specific data structure like
100+`%talk-command`. `mark`s are not specific to `%say` generators; any time data
101+moves between programs in Arvo, that data is marked.
102+
103+So, more formally, a `%say` generator is a `cell`. The head of that cell is the
104+`%say` tag, and the tail is a `gate` that produces a `cask` -- a pair of the
105+output data and the `mark` describing that data.
106+
107+Below is an example of a `%say` generator. Save it to `add.hoon` in the `/gen`
108+directory.
109+
110+```
111+:- %say
112+|= *
113+:- %noun
114+(add 40 2)
115+```
116+
117+Now run the generator as below:
118+
119+```
120+> +add
121+42
122+```
123+
124+Notice that we used no argument, something that is possible with `%say`
125+generators but impossible with naked generators. We'll explain that in a moment.
126+For now, let's focus on the code that is necessary to make something a `%say`
127+generator.
128+
129+```
130+:- %say
131+```
132+
133+Recall that the rune `:-` produces a cell, with the first following expression
134+as its head and the second following expression as its tail.
135+
136+The expression above, then, creates a cell with `%say` as the head. The tail is
137+the `|= *` expression on the line that follows.
138+
139+```
140+|= *
141+:- %noun
142+(add 40 2)
143+```
144+
145+`|= *` constructs a gate that takes a noun. This gate will itself produce a
146+`cask`, which is cell formed by the prepending `:-`. The head of that `cask` is
147+`%noun` and the tail is rest of the program, `(add 40 2)`. The tail of the
148+`cask` will be our actual data produced by body of the program: in this case
149+just adding 40 and 2 together.
150+
151+#### `%say` generators with arguments
152+
153+We can modify the boilerplate code to allow arguments to be passed into a `%say`
154+generator, but in a way that gives us more power than we would have if we just
155+used a naked generator.
156+
157+Naked generators are limited because they have no way of accessing data that
158+exists in Arvo, such as the date and time or pieces of fresh entropy. In `%say`
159+generators, however, we can access that kind of subject by identifying them
160+in the gate's sample, which we only specified as `*` in the previous few
161+examples. But we can do more with `%say` generators if we do more with that
162+sample. Any valid sample will follow this 3-tuple scheme:
163+
164+`[[datetime, entropy, and beak] [list of unnamed arguments] [list of named arguments]]`
165+
166+This entire structure is a noun, which is why `*` is a valid sample if we
167+wish to not use any of the information here in a generator. But let's look at
168+each of these three elements, piece by piece.
169+
170+The *first part* of the above 3-tuple is a noun that is composed of three
171+atoms:
172+
173+`now` is the current time.
174+`eny` is 512 bits of entropy for seeding random number generators.
175+`beak` contains the current ship, desk, and case.
176+
177+You can access each of those pieces of data by typing their names into the Dojo.
178+But in a generator, we need put faces (variable names) onto that data so we can
179+easily use it in the program. We can do that like below:
180+
181+```
182+|= [[now=@da eny=@uvJ bec=beak] ~ ~]
183+```
184+
185+Any of those pieces of data could be omitted by replacing part of the noun with
186+`*` rather than giving them faces. For example, `[now=@da * bec=beak]` if we
187+didn't want `eny`, or `[* * bec=beak]` if we only wanted `beak`.
188+
189+The *second part* of the sample is a `list` of arguments that _must_ be passed
190+to the generator as it's run. Because it's a `list`, this element needs to be
191+terminated with a `~`. In the example above, we used `~`, the empty list, to
192+represent this second part. But that would mean we don't want to use any new
193+faces for our generator. So a sample where we want to declare a single required
194+argument would look like this.
195+
196+```
197+|= [* [n=@ud ~] ~]
198+```
199+
200+In the above code, we use a `*` to ignore any of the information that would go
201+first part of the sample, kind of in the same way that we use a `~` to say
202+"nothing here" for the second or third parts of the sample.
203+
204+But to use both parts together, our gate and sample would look like this:
205+
206+```
207+|= [[now=@da eny=@uvJ bec=beak] [n=@ud ~] ~]
208+```
209+
210+The *third part* of the sample is a list of _optional_ arguments. These
211+arguments may be passed to the program as its being run, but the program doesn't
212+require them. The syntax for the third part is just like the syntax for the
213+second part, besides its position:
214+
215+```
216+|= [* ~ [bet=@ud ~]]
217+```
218+
219+Let's look at an example that uses all three parts. Save the code below in a
220+file called `dice.hoon` in your `/gen` directory.
221+
222+```
223+:- %say
224+|= [[now=@da eny=@uvJ bec=beak] [n=@ud ~] [bet=@ud ~]]
225+:- %noun
226+[(~(rad og eny) n) bet]
227+```
228+
229+This is a very simple dice program with an optional betting functionality.
230+In the code, our sample specifies faces on all the all of the Arvo data, meaning
231+that we can can easily access them. We also require the argument `[n=@ud ~]`,
232+and allow the _optional_ argument `[bet=@ud ~]`.
233+
234+But there's something new: `(~(rad og eny) n)`. This code pulls the `rad` arm
235+out of the `og` core with the subject of `eny`. Recall that `eny` is our entropy
236+value, so this is used to seed the generator. The `rad` arm will give us a
237+pseudorandom number between 0 and `n`, and the `bet` that it's in a cell with
238+will just return the argument that we might choose to give it.
239+
240+We can run this generator like so:
241+
242+```
243+> +dice 6, =bet 2
244+[4 2]
245+
246+> +dice 6
247+[5 0]
248+
249+> +dice 6
250+[2 0]
251+
252+> +dice 6, =bet 200
253+[0 200]
254+
255+> +dice
256+nest-fail
257+```
258+
259+We get a different value from the same generator between runs, something that
260+isn't possible with a naked generator. Another novelty is the ability to choose
261+to not use the second argument.
262+
263+##### Arguments without a cell
264+
265+Also unlike a naked generator, we don't need to put our arguments together into
266+a cell. Swap the code below into your `add.hoon` file.
267+
268+```
269+:- %say
270+|= [* [a=@ud b=@ud ~] ~]
271+:- %noun
272+(add a b)
273+```
274+
275+In the above code we're again creating a `%say` generator, but the sample of our
276+gate is a little different than before. We are using `*` for the first part of
277+the sample, because we don't use any Arvo data in this program. We are using a
278+`~` for the third part, because we aren't allowing optional arguments -- the
279+list of optional arguments is _empty_.
280+
281+Run a command like the one below in the Dojo. Notice that you can use two
282+arguments that aren't in the cells.
283+
284+```
285+> +add 40 2
286+42
287+```
288+
289+### `%ask` generators
290+
291+We use the `%ask` generator when we want to create an interactive program
292+that prompts for inputs as it runs, rather than expecting arguments to be passed
293+in at the time of initiation.
294+
295+Code-wise, you need two things to create a useful `%ask` generator.
296+
297+The first is *`sole-result`*, which lives in `/sur/sole.hoon`. It's a
298+mold-builder that's required by any part of Arvo, such as the Dojo, that wants
299+to understand the output of the `%ask` generator. Because it's the only type of
300+`%ask` data that is intelligible to the system, the ultimate output of an `%ask`
301+generator must be `sole-result`. You can think of it as a function that takes
302+molds and digests them into a type for the operating system. The simplest way to
303+use `sole-result` is by calling the `produce` gate from `/lib/generators.hoon`.
304+
305+The second is *`prompt`*, a gate that lives in `lib/generators.hoon`. `prompt`
306+allows you to get get information back from the user, which is essential to the
307+typical desired operation of an `%ask` generator.
308+
309+Since this isn't easy to digest in the abstract, we'll use an example generator
310+to explain how this kind of generator works.
311+
312+##### `%ask` example
313+
314+The code below is an `%ask` generator that checks if the user inputs "blue" when
315+prompted. Save it as `axe.hoon` in `/gen`.
316+
317+```
318+/- sole
319+/+ generators
320+=, [sole generators]
321+:- %ask
322+|= *
323+^- (sole-result (cask tang))
324+%+ print leaf+"What is your favorite color?"
325+%+ prompt [%& %prompt "color: "]
326+|= t=tape
327+%+ produce %tang
328+?: =(t "blue")
329+ :~ leaf+"Oh. Thank you very much."
330+ leaf+"Right. Off you go then."
331+ ==
332+:~ leaf+"Aaaaagh!"
333+ leaf+"Into the Gorge of Eternal Peril with you!"
334+==
335+```
336+
337+Run the generator from the Dojo:
338+
339+```
340+> +axe
341+What is your favorite color?
342+: color:
343+```
344+
345+Something new happened. Instead of simply returning something, your Dojo's
346+prompt changed from `~your-urbit:dojo>` to `~your-urbit:dojo: color:`, and
347+expects additional input. Let's give it an input:
348+
349+```
350+: color: red
351+Into the Gorge of Eternal Peril with you!
352+Aaaaagh!
353+```
354+
355+Let's go over what exactly is happening in this code.
356+
357+```
358+/- sole
359+/+ generators
360+=, [sole generators]
361+```
362+
363+Here we bring in some of the types we are going to need from `/sur/sole` and
364+gates we will use from `/lib/generators`. We use some special runes for this.
365+
366+`/-` is a Ford rune used to import types from `sur`.
367+
368+`/+` is a Ford rune used to import libraries from `lib`.
369+
370+`=,` is a rune that allows us to expose a namespace. We do this to avoid having
371+to write `sole-result:sole` instead of `sole-result` or `print:generators`
372+instead of `print`.
373+
374+```
375+:- %ask
376+|= *
377+```
378+
379+This code might be familiar. Just as with their `%say` cousins, `%ask`
380+generators need to produce a `cell`, the head of which specifies what kind of
381+generator we are running.
382+
383+With `|= *`, we create a gate and ignore the standard arguments we are given,
384+because we're not using them.
385+
386+```
387+^- (sole-result (cask tang))
388+```
389+
390+`%ask` generators need to have the second half of the cell be a gate that
391+produces a `sole-result`, one that in this case contains a `cask` of `tang`.
392+We use the `^-` rune to constrain the generator's output to such a
393+`sole-result`.
394+
395+A `cask` is a pair of a `mark` name and a noun. Recall that a `mark` can be
396+thought of as an Arvo-level MIME type for data.
397+
398+A `tang` is a `list` of `tank` and a `tank` is a structure for printing data
399+that there are three types of: `leaf`, `palm`, and `rose`. A `leaf` is for
400+printing a single noun, a `rose` is for printing rows of data, and a `palm` is
401+printing backstep-indented lists.
402+
403+```
404+%+ print leaf+"What is your favorite color?"
405+%+ prompt [%& %prompt "color: "]
406+|= t=tape :: we will come to this later
407+%+ produce %tang
408+```
409+
410+
411+Because we imported `generators`, we can access its contained gates, three of
412+which we use in `axe.hoon`: `print`, `prompt`, and `produce`.
413+
414+*`print`* is used for printing a `tank` to the console.
415+
416+In our example, `%+` is the rune to call a gate, and our gate `print` takes one
417+argument which is a `tank` to print. The `+` here is syntactic sugar for
418+`[leaf "What is your favorite color?"]` that just makes it easier to write.
419+
420+*`prompt`* is used to construct a prompt for the user to provide input.
421+It takes a single argument that is a tuple. Most `%ask` generators will want to
422+use the `prompt` gate.
423+
424+The first element of the `prompt` sample is a flag that indicates what the
425+user typed should be echoed out to them or hidden. `%&` will produce echoed
426+output and `%|` will hide the output (for use in passwords or other secret
427+text).
428+
429+The second element of the `prompt` sample is a `%term` used The second argument
430+is intended to be information to use to create autocomplete options for the
431+prompt this is not yet implemented.
432+
433+The third element of the `prompt` sample is the `tape` that we would like to
434+use to prompt the user. In the case of our example, we use `"color: "`.
435+
436+*`produce`* is is used to construct the output of the generator. In our
437+example, we produce a `tang`.
438+
439+
440+```
441+|= t=tape
442+```
443+
444+Our gate here takes a `tape` that was produced by `prompt` if we needed another
445+type of data we could use `parse` to obtain it.
446+
447+The rest of this generator should be intelligible to those with Hoon knowledge
448+at this point.
449+
450+One quirk that you should be aware of, though, is that `tang` prints in reverse
451+order from how it is created. The reason for this is that `tang` was originally
452+created to display stack trace information, which should be produced in
453+in reverse order. This leads to an annoyance: we either have to specify our
454+messages backwards or construct them in the order we want and then `flop` the
455+`list` This is a [known issue](https://github.com/urbit/arvo/issues/840) to be
456+resolved.
457+
458+### `%get` generators
459+
460+`%get` generators are used for making HTTP requests through `eyre`. For this
461+section we'll assume knowledge of generators covered in previous sections, so
462+we can jump in with an example:
463+
464+```
465+/- sole
466+/+ generators
467+=, generators
468+:- %get
469+|= [* [url=tape ~] ~]
470+^- (sole-request:sole (cask json))
471+%+ curl (scan url auri:de-purl:html)
472+|= hit/httr:eyre
473+?~ r.hit !!
474+=/ my-json (de-json:html q.u.r.hit)
475+?~ my-json !!
476+=, dejs:format
477+%+ produce %json
478+%. %title
479+%~ got by
480+%- (om same)
481+u.my-json
482+```
483+
484+The above generator was written with the
485+[Studio Ghibli API](https://ghibliapi.herokuapp.com/). The generator below was
486+designed to use the this API to print the title of a Studio Ghibli film.
487+
488+```
489+/- sole
490+/+ generators
491+=, generators
492+```
493+
494+Here we import types from `/sur/sole.hoon` and gates from
495+`/lib/generators.hoon`. We also use `=,` to exposes the namespace of
496+`generators` that we just imported.
497+
498+```
499+:- %get
500+|= [* [url=tape ~] ~]
501+```
502+
503+Like with `%say` and `%ask` generators, we must have include the "boilerplate":
504+code that produces a generator-specific `cell`. At the head of that cell is
505+`%get`, and the tail is a gate that takes a `tape` of the URL that we are trying
506+to access.
507+
508+```
509+^- (sole-request:sole (cask json))
510+```
511+
512+Here we make sure our gate is producing a `sole-request` which contains a `cask`
513+of the type `json`.
514+
515+
516+```
517+%+ curl (scan url auri:de-purl:html)
518+|= hit/httr:eyre
519+```
520+
521+Here we can see the call to `curl`, the sample of which will parse our url and
522+make sure it's in the proper format.
523+
524+The second part of the `curl` call is the `gate` that will get called with the
525+result provided by the actual HTTP request.
526+
527+```
528+?~ r.hit !!
529+```
530+
531+Here we verify that the `r` face of our response is not null before
532+continuing. If it is null, the generator will crash, as there is nothing left to
533+do.
534+
535+```
536+=/ my-json (de-json:html q.u.r.hit)
537+```
538+
539+Here we verify that data was returned to us in the `json` Hoon type.
540+`q.u.r.hit` is the actual data we care about as a `cord`. `hit` is actually an
541+entire response object, including headers, which we don't care about for this
542+example. `de-json` out of `html` will parse the `cord` and produce a `json` type
543+for us that we can then use to inspect the data.
544+
545+```
546+?~ my-json !!
547+=, dejs:format
548+```
549+
550+After parsing the JSON structure we want to make sure we got back a valid `json`
551+type. To do that, we test for null and crash that test passes. If not, we use
552+`=,` on the `dejs:format` core so that we can more easily pull arms out of it.
553+
554+```
555+%+ produce %json
556+%. %title
557+%~ got by
558+```
559+
560+Now we come to our `produce` call. Recall that `produce` is used in all the
561+non-naked generators to output the final result. Here we want to produce a
562+`cask` of `json`, so we use the mark `%json` to indicate that.
563+
564+Next we have what may be an unfamiliar rune: `%.`. This rune is simply `%-` with
565+the argument order reversed. `%title` is the argument passed to the next gate
566+that is called. `by` is the core that is the `map` engine. A Hoon `map` is a
567+key-value pair structure, sometimes referred to as a dictionary in other
568+languages. `got` is the arm in `by` that will produce a gate that we can call to
569+access members of a given `map`.
570+
571+> The idiomatic way to write Hoon is to have the
572+ "heaviest" code paths lowest in the program, which is why `%.`
573+ is preferred in this case over `%-`. Use of this style is recommended, as it
574+ results in code that is much easier to read.
575+
576+At the moment, however, we do not have a `map`, but instead have a `json` type.
577+
578+```
579+%- (om same)
580+u.my-json
581+```
582+
583+Here we use `om`, which is an arm in `dejs:format` that will take a `json` and
584+return a `map`. `same` is a wet gate from the standard library that takes an
585+argument of any type and returns it as that type. We could use any other gate to
586+modify the data if that were useful for the particular data we are going to be
587+processing, but here we want the data exactly as it is.
588+
589+Since `my-json` is a `unit` of `json`, the actual data we are looking for is in
590+`u.my-json`. We take this and give it to the gate produced by the call to `om`.
591+This gate is going to transform our data by taking in the `json` structure and
592+producing a `map`. Then we can look up keys in this map using `got` from
593+the `by` core.
594+
595+`%.` performs the same function as `%-`, the rune used to call a function,
596+except with the arguments reversed so that the function is last and
597+the arguments of that function come first. It's used here to keep with the
598+principle that heaviest, or most complex, expressions should go to the bottom.
599+
600+The key we are looking up in our new `map` we've created from `my-json` is
601+`%title` which is an element in the original source to print out to produce as
602+the final result of the generator.