git.iko.soy/urbit.org
on master help sign in
Merge pull request #1184 from tinnus-napbus/add-strings-guide
e01b326tinnus-napbus
content/docs/hoon/guides/_index.md+2
compare with master
unifiedsplit
@@ -11,3 +11,5 @@ insert_anchor_links = "right"
11## [CLI Apps](/docs/hoon/guides/cli-tutorial)
11## [CLI Apps](/docs/hoon/guides/cli-tutorial)
12
12
13## [Parsing](/docs/hoon/guides/parsing)
13## [Parsing](/docs/hoon/guides/parsing)
14+
15+## [Strings](/docs/hoon/guides/strings)
content/docs/hoon/guides/strings.mdnew file+531
@@ -0,0 +1,531 @@
1++++
2+title = "Strings"
3+weight = 4
4+template = "doc.html"
5++++
6+
7+This document discusses hoon's two main string types: `cord`s (as well as its
8+subsets `knot` and `term`) and `tape`s. The focus of this
9+document is on their basic properties, syntax and the most common text-related
10+functions you'll regularly encounter. In particular, it discusses conversions
11+and the encoding/decoding of atom auras in strings.
12+
13+Hoon has a system for writing more elaborate functional parsers, but that is not
14+touched on here. Instead, see the [Parsing](/docs/hoon/guides/parsing) guide.
15+Hoon also has a type for UTF-32 strings, but those are rarely used and not
16+discussed in this document.
17+
18+There are a good deal more text manipulation functions than are discussed here.
19+See the [Further Reading](#further-reading) section for details.
20+
21+## `tape`s vs. text atoms
22+
23+As mentioned, urbit mainly deals with two kinds of strings: `tape`s and
24+`cord`/`knot`/`term`s. The former is a list of individual UTF-8 characters.
25+The latter three encode UTF-8 strings in a single atom.
26+
27+Cords may contain any UTF-8 characters, while `knot`s and `term`s only allow a
28+smaller subset. Each of these are discussed below in the [Text
29+atoms](#text-atoms) section.
30+
31+Text atoms like `cord`s are more efficient to store and move around. They are
32+also more efficient to manipulate with simple bitwise operations. Their downside
33+is that UTF-8 characters vary in their byte-length. ASCII characters are all
34+8-bit, but others can occupy up to four bytes. Accounting for this variation in
35+character size can complicate otherwise simple functions. Tapes, on the other
36+hand, don't have this problem because each character is a separate item in the
37+list, regardless of it byte-length. This fact makes it much easier to process
38+tapes in non-trivial ways with simple list functions.
39+
40+In light of this, a general rule of thumb is to use cords for simple things like
41+storing chat messages or exchanging them over the network. If text requires
42+complex processing on the other hand, it is generally easier with tapes. Note
43+there _are_ cord manipulation functions in the standard library, so you needn't
44+always convert cords to tapes for processing, it just depends on the case.
45+
46+Next we'll look at these different types of strings in more detail.
47+
48+## Text atoms
49+
50+### `cord`
51+
52+A [`cord`](/docs/hoon/reference/stdlib/2q#cord) has an aura of `@t`. It denotes
53+UTF-8 text encoded in an atom, little-endian. That is, the first character in
54+the text is the least-significant byte. A cord may contain any UTF-8 characters,
55+there are no restrictions.
56+
57+The `hoon` syntax for a cord is some text wrapped in single-quotes like:
58+
59+```hoon
60+'This is a cord!'
61+```
62+
63+single-quotes and backslashes must be escaped with a backslash like:
64+
65+```hoon
66+'\'quotes\' \\backslashes\\'
67+```
68+
69+Characters can also be entered as hex, they just have to be escaped by a
70+backslash. For example, `'\21\21\21'` will render as `'!!!'`. This is useful for
71+entering special characters such as line breaks like `'foo\0abar'`.
72+
73+Cords divided over multiple lines are allowed. There are two ways to do this.
74+The first is to start and end with three single-quotes like:
75+
76+```hoon
77+'''
78+foo
79+bar
80+baz
81+'''
82+```
83+
84+The line endings will be encoded Unix-style as line feed characters like:
85+
86+```hoon
87+'foo\0abar\0abaz'
88+```
89+
90+The second is to begin with a single-quote like usual, then break the line by
91+ending it with a backslash and start the next line with a forward-slash like:
92+
93+```hoon
94+'foo\
95+/bar\
96+/baz'
97+```
98+
99+This will be parsed to:
100+
101+```hoon
102+'foobarbaz'
103+```
104+
105+### `knot`
106+
107+A [`knot`](/docs/hoon/reference/stdlib/2q#knot) has an aura of `@ta`, and is a
108+subset of a [`cord`](#cord). It allows lower-case letters, numbers, and four
109+special characters: Hyphen, tilde, underscore and period. Its restricted set of
110+characters is intended to be URL-safe.
111+
112+The `hoon` syntax for a knot is a string containing any of the aforementioned
113+characters prepended with `~.` like:
114+
115+```hoon
116+~.abc-123.def_456~ghi
117+```
118+
119+### `term`
120+
121+A [`term`](/docs/hoon/reference/stdlib/2q#term) has an aura of `@tas`, and is a
122+subset of a [`knot`](#knot). It only allows lower-case letters, numbers, and
123+hyphens. Additionally, the first character cannot be a hyphen or number. This is
124+a very restricted text atom, and is intended for naming data structures and the
125+like.
126+
127+The `hoon` syntax for a term is a string conforming to the prior description,
128+prepended with a `%` like:
129+
130+```hoon
131+%foo-123
132+```
133+
134+#### A note about `term` type inference
135+
136+There is actually an even more restricted text atom form with the same `%foo`
137+syntax as a term, where the type of the text is the text itself. For example, in
138+the dojo:
139+
140+```
141+> `%foo`%foo
142+%foo
143+```
144+
145+The hoon parser will, by default, infer the type of `%foo`-style syntax this
146+way. If we try with the dojo type printer:
147+
148+```
149+> ? %foo
150+ %foo
151+%foo
152+```
153+
154+This type-as-itself is used for many things, such as unions like:
155+
156+```hoon
157+?(%foo %bar %bas)
158+```
159+
160+In order to give `%foo` the more generic `@tas` aura, it must be explicitly
161+upcast like:
162+
163+```
164+> ? `@tas`%foo
165+ @tas
166+%foo
167+```
168+
169+This is something to be wary of. For example, if you wanted to form a `(set @tas)` you might think to do:
170+
171+```hoon
172+(silt (limo ~[%foo %bar %baz]))
173+```
174+
175+However, this will actually form a set of the union `?(%foo %bar %baz)` due to
176+the specificity of type inference:
177+
178+```
179+> ? (silt (limo ~[%foo %bar %baz]))
180+ ?(%~ [?(n=%bar n=%baz n=%foo) l=nlr(?(%bar %baz %foo)) r=nlr(?(%bar %baz %foo))])
181+[n=%baz l={%bar} r={%foo}]
182+```
183+
184+One further note about the type-as-itself form: Ocassionally you may wish to
185+form a union of strings which contain characters disallowed in `term`s. To get
186+around this, you can enclose the text after the `%` with single-quotes like
187+`%'HELLO!'`.
188+
189+### Aura type validity
190+
191+The hoon parser will balk at `cord`s, `knot`s and `term`s containing invalid
192+characters. However, because they're merely auras, any atom can be cast to them.
193+When cast (or clammed), they will **not** be validated in terms of whether the
194+characters are allowed in the specified aura.
195+
196+For example, you can do this:
197+
198+```
199+> `@tas`'!%* $@&'
200+%!%* $@&
201+```
202+
203+This means you cannot rely on mere aura-casting if you need the text to conform
204+to the specified aura's restrictions. Instead, there are a couple of function in
205+the standard library to check text aura validity:
206+[`+sane`](/docs/hoon/reference/stdlib/4b#sane) and
207+[`+sand`](/docs/hoon/reference/stdlib/4b#sane).
208+
209+The `+sane` function takes an argument of either `%ta` or `%tas` to validate
210+`@ta` and `@tas` respectively (you can technically give it `%t` for `@t` too but
211+there's no real point). It will return `%.y` if the given atom is valid for the
212+given aura, and `%.n` if it isn't. For example:
213+
214+```
215+> ((sane %tas) 'foo')
216+%.y
217+> ((sane %tas) 'foo!')
218+%.n
219+```
220+
221+The `+sand` function does the same thing, but rather than returning a `?` it
222+returns a `unit` of the given atom, or `~` if validation failed. For example:
223+
224+```
225+> `(unit @tas)`((sand %tas) 'foo')
226+[~ %foo]
227+> `(unit @tas)`((sand %tas) 'foo!')
228+~
229+```
230+
231+## `tape`
232+
233+A [`tape`](/docs/hoon/reference/stdlib/2q#tape) is the other
234+main string type in hoon. Rather than a single atom, it's instead a list of
235+individual `@tD` characters (the `D` specifies a bit-length of 8, see the
236+[Auras](/docs/hoon/reference/auras#bitwidth) documentation for
237+details). The head of the list is the first character in the string.
238+
239+The `hoon` syntax for a tape is some text wrapped in double-quotes like:
240+
241+```hoon
242+"This is a tape!"
243+```
244+
245+Double-quotes, backslashes and left-braces must be escaped by a backslash
246+character:
247+
248+```hoon
249+"\"double-quotes\" \\backslash\\ left-brace:\{"
250+```
251+
252+Like with `cord`s, characters can also be entered as hex escaped by a backslash
253+so `"\21\21\21"` renders as `"!!!"`.
254+
255+Tapes divided over multiple lines are allowed. Unlike [`cord`](#cord)s, there is
256+only one way to do this, which is by starting and ending with three
257+double-quotes like:
258+
259+```hoon
260+"""
261+foo
262+bar
263+baz
264+"""
265+```
266+
267+The line endings will be encoded Unix-style as line feed characters like:
268+
269+```hoon
270+"foo\0abar\0abaz"
271+```
272+
273+As mentioned earlier, tapes are lists of single characters:
274+
275+```
276+> `tape`~['f' 'o' 'o']
277+"foo"
278+```
279+
280+This means they can be manipulated with ordinary list functions:
281+
282+```
283+> `tape`(turn "foobar" succ)
284+"gppcbs"
285+```
286+
287+### Interpolation
288+
289+Tapes, unlike cords, allow string interpolation. Arbitrary `hoon` may be
290+embedded in the tape syntax and its product will be included in the resulting
291+tape. There are two ways to do it:
292+
293+#### Manual
294+
295+In the first case, the code to be evaluated is enclosed in braces. The type of
296+the product of the code must itself be a tape. For example, if the `@p` of our
297+ship is stored in `our`, simply doing `"{our}"` will fail because its type will
298+be `@p` rather than `tape`. Instead, we must explicitly use the
299+[`+scow`](/docs/hoon/reference/stdlib/4m#scow) function to
300+render `our` as a tape:
301+
302+```
303+> "{(scow %p our)}"
304+"~zod"
305+```
306+
307+Another example:
308+
309+```
310+> "[{(scow %p our)} {(scow %da now)}]"
311+"[~zod ~2021.10.3..08.59.10..2335]"
312+```
313+
314+#### Automatic
315+
316+Rather than having to manually render data as a `tape`, angle brackets _inside_
317+the braces tell the interpreter to automatically pretty-print the product of the
318+expression as a tape. This way we needn't use functions like `+scow` and can
319+just reference things like `our` directly:
320+
321+```
322+> "{<our>}"
323+~zod
324+```
325+
326+Another example:
327+
328+```
329+> "{<(add 1 2)>}"
330+"3"
331+```
332+
333+And another:
334+
335+```
336+> "{<our now>}"
337+"[~zod ~2021.10.3..09.01.14..1654]"
338+```
339+
340+## Conversions
341+
342+Tapes can easily be converted to cords and vice versa. There are two stdlib
343+functions for this purpose: [`+crip`](/docs/hoon/reference/stdlib/4b#crip) and
344+[`+trip`](/docs/hoon/reference/stdlib/4b#trip). The former converts a `tape` to
345+a `cord` and the latter does the opposite. For example:
346+
347+```
348+> (crip "foobar")
349+'foobar'
350+> (trip 'foobar')
351+"foobar"
352+```
353+
354+Knots and terms can also be converted to tapes with `+trip`:
355+
356+```
357+> (trip %foobar)
358+"foobar"
359+> (trip ~.foobar)
360+"foobar"
361+```
362+
363+Likewise, the output of `+crip` can be cast to a knot or term:
364+
365+```
366+> `@tas`(crip "foobar")
367+%foobar
368+> `@ta`(crip "foobar")
369+~.foobar
370+> `@tas`(need ((sand %tas) (crip "foobar")))
371+%foobar
372+```
373+
374+## Encoding in text
375+
376+It's common to encode atoms in cords or knots, particularly when constructing a
377+[scry](/docs/arvo/concepts/scry) [`path`](/docs/hoon/reference/stdlib/2q#path)
378+or just a `path` in general. There are two main functions for this purpose:
379+[`+scot`](/docs/hoon/reference/stdlib/4m#scot) and
380+[`+scow`](/docs/hoon/reference/stdlib/4m#scow). The former produces a `knot`,
381+and the latter produces a `tape`. Additionally, there are two more functions for
382+encoding `path`s in cords and tapes respectively:
383+[`+spat`](/docs/hoon/reference/stdlib/4m#spat) and
384+[`+spud`](/docs/hoon/reference/stdlib/4m#spud).
385+
386+### `+scot` and `+spat`
387+
388+`+scot` encodes atoms of various auras in a `knot` (or `cord`/`term` with
389+casting). It takes two arguments: the aura in a `@tas` and the atom to be
390+encoded. For example:
391+
392+```
393+> (scot %p ~zod)
394+~.~zod
395+
396+> (scot %da now)
397+~.~2021.10.4..07.35.54..6d41
398+
399+> (scot %ux 0xaa.bbbb)
400+~.0xaa.bbbb
401+```
402+
403+Note the aura of the atom needn't actually match the specified aura:
404+
405+```
406+> (scot %ud ~zod)
407+~.0
408+```
409+
410+Hoon can of course be evaluated in its arguments as well:
411+
412+```
413+> (scot %ud (add 1 1))
414+~.2
415+```
416+
417+You'll most commonly see this used in constructing a `path` like:
418+
419+```
420+> /(scot %p our)/garden/(scot %da now)/foo/(scot %ud 123.456)
421+[~.~zod %garden ~.~2021.10.4..07.43.14..a556 %foo ~.123.456 ~]
422+
423+> `path`/(scot %p our)/garden/(scot %da now)/foo/(scot %ud 123.456)
424+/~zod/garden/~2021.10.4..07.43.23..9a0f/foo/123.456
425+```
426+
427+`+spat` simply encodes a `path` in a cord like:
428+
429+```
430+> (spat /foo/bar/baz)
431+'/foo/bar/baz'
432+```
433+
434+### `+scow` and `+spud`
435+
436+`+scow` is the same as [`+scot`](#scot-and-spat) except it produces a tape
437+rather than a knot. For example:
438+
439+```
440+> (scow %p ~zod)
441+"~zod"
442+
443+> (scow %da now)
444+"~2021.10.4..07.45.25..b720"
445+
446+> (scow %ux 0xaa.bbbb)
447+"0xaa.bbbb"
448+```
449+
450+`+spud` simply encodes a `path` in a tape:
451+
452+```
453+> (spud /foo/bar/baz)
454+"/foo/bar/baz"
455+```
456+
457+## Decoding from text
458+
459+For decoding atoms of particular auras encoded in cords, there are three
460+functions: [`+slat`](/docs/hoon/reference/stdlib/4m#slat),
461+[`+slav`](/docs/hoon/reference/stdlib/4m#slav), and
462+[`+slaw`](/docs/hoon/reference/stdlib/4m#slaw). Additionally, there is
463+[`+stab`](/docs/hoon/reference/stdlib/4m#stab) for decoding a cord to a path.
464+
465+`+slav` parses the given cord with the aura specified as a `@tas`, crashing if
466+the parsing failed. For example:
467+
468+```
469+> `@da`(slav %da '~2021.10.4..11.26.54')
470+~2021.10.4..11.26.54
471+
472+> `@p`(slav %p '~zod')
473+~zod
474+
475+> (slav %p 'foo')
476+dojo: hoon expression failed
477+```
478+
479+`+slaw` is like `+slav` except it produces a `unit` which is null if parsing
480+failed, rather than crashing. For example:
481+
482+```
483+> `(unit @da)`(slaw %da '~2021.10.4..11.26.54')
484+[~ ~2021.10.4..11.26.54]
485+
486+> `(unit @p)`(slaw %p '~zod')
487+[~ ~zod]
488+
489+> (slaw %p 'foo')
490+~
491+```
492+
493+`+slat` is a curried version of `+slaw`, meaning it's given the aura and
494+produces a new gate which takes the actual cord. For example:
495+
496+```
497+> `(unit @da)`((slat %da) '~2021.10.4..11.26.54')
498+[~ ~2021.10.4..11.26.54]
499+
500+> `(unit @p)`((slat %p) '~zod')
501+[~ ~zod]
502+
503+> ((slat %p) 'foo')
504+~
505+```
506+
507+Finally, `+stab` parses a cord containing a path to a `path`. For example:
508+
509+```
510+> (stab '/foo/bar/baz')
511+/foo/bar/baz
512+```
513+
514+## Futher reading
515+
516+- [Parsing](/docs/hoon/guides/parsing) - A guide to writing fully-fledged
517+ functional parsers in hoon.
518+
519+- [Auras](/docs/hoon/reference/auras) - Details of auras in hoon.
520+
521+- [stdlib 2b: List logic](/docs/hoon/reference/stdlib/2b) - Standard library
522+ functions for manipulating lists, which are useful for dealing with tapes.
523+
524+- [stdlib 2q: Molds and Mold-builders](/docs/hoon/reference/stdlib/2q) - Several
525+ text types are defined in this section of the standard library.
526+
527+- [stdlib 4b: Text processing](/docs/hoon/reference/stdlib/4b) - Standard
528+ library functions for manipulating and converting tapes and strings.
529+
530+- [stdlib 4m: Formatting functions](/docs/hoon/reference/stdlib/4m) - Standard
531+ library functions for encoding and decoding atom auras in strings.