@@ -0,0 +1,333 @@
1++++
2+title = "Writing Marks"
3+weight = 2
4+template = "doc.html"
5++++
6+
7+Here we'll walk through a practical example of writing a `mark` file.
8+
9+We'll create a `mark` for CSV (comma separated values) files, a simple format for storing tabular data in a text file. Note that there's already a `csv.hoon` `mark` and library, but we'll create new ones for demonstrative purposes. It shouldn't be an issue to overwrite the existing ones on a fakezod.
10+
11+CSV files separate fields with commas and rows with line breaks. They look something like:
12+
13+```
14+foo,bar,baz
15+blah,blah,blah
16+1,2,3
17+```
18+
19+There is a little complexity surrounding special characters in fields and line endings, but otherwise the only other rule is that all rows must have the same number of fields. You can refer to [RFC4180 on the IETF website](https://datatracker.ietf.org/doc/html/rfc4180) for more details.
20+
21+We'll represent such a structure in Hoon as a `(list (list @t))` like:
22+
23+```hoon
24+[['foo' 'bar' 'baz' ~] ['blah' 'blah' 'blah' ~] ['1' '2' '3' ~] ~]
25+```
26+
27+We could perhaps create the type with a `$|` rune to include row-length validation in the mold itself, but a `(list (list @t))` is simpler for demonstrative purposes.
28+
29+## A simple mark
30+
31+Let's begin with the simplest `mark` file:
32+
33+```hoon
34+|_ csv=(list (list @t))
35+++ grab
36+ |%
37+ ++ noun (list (list @t))
38+ --
39+++ grow
40+ |%
41+ ++ noun csv
42+ --
43+++ grad %noun
44+--
45+```
46+
47+The door takes a `(list (list @t))` as its sample, and we've given it a face of `csv` so we can easily reference it. Note its face could be anything, it needn't be the name of our `mark`. When we're doing something with data that has a `%csv` `mark` like converting it to another `mark` or creating a diff, this is where our data will reside.
48+
49+Next we have the `+grab` arm of our door, which contains a core with arms for converting _to_ our `mark` _from_ other `mark`s. We've given it one arm for the `%noun` `mark` - the most generic `mark` which will take any `noun`. Our `+noun` arm will simply clam whatever it's given with the `(list (list @t))` `mold`.
50+
51+Next is the `+grow` arm which does the inverse of `+grab`, converting _from_ our `mark` _to_ another `mark`. We've also given it a `+noun` arm, this time it will simply return the door's sample named `csv`, which is of course already a `noun`.
52+
53+Note that the `+noun` arm is _mandatory_ in `+grab`. Clay cannot build a `mark` core without it. Conversion arms for any other `mark`s apart from `%noun` are optional.
54+
55+Finally we have the `+grad` arm. This arm specifies functions for revision control like creating diffs, patching files and so on. In our case, rather than writing all those functions, we've just delegated those tasks to the `%noun` `mark`. We can do this because we've specified conversion routines to and from the `%noun` `mark` in our `+grow` and `+grab` arms. When we modify a file with a `%csv` `mark`, Clay will convert our data to a `%noun` `mark`, execute the necessary `+grad` functions from the `%noun` `mark` file, and then convert it back to a `%csv` `mark` again.
56+
57+So now we have a valid `%csv` `mark` file. If we save this as `csv.hoon` in the `/mar` directory we could store `%csv` data in Clay. This may be sufficient for some applications, but what if we want to import a CSV file from Unix or elsewhere? In the next section, we'll look at conversions to and from a `%mime` `mark` to address this.
58+
59+## `%mime` conversions
60+
61+The `$mime` type represents raw data from Unix or elsewhere. For example, if a text file from Unix containing the word `foo` were converted to a `$mime` type in Urbit, it would look something like:
62+
63+```hoon
64+[/text/plain q=[p=3 q=7.303.014]]
65+```
66+
67+`/text/plain` is its [MIME type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types) and `p.q` is the byte-length of `q.q`, which is the data itself as an `atom`.
68+
69+The `%mime` `mark` is used by Clay to store and convert `$mime` data. It's an important `mark` for moving files from Unix to Urbit and vice versa. When you add a file to a `desk` you have mounted to Unix and `|commit` the change, Clay will first receive the file as a `%mime` `mark`, then convert from a `%mime` to whatever `mark` matches the file extension. For example, `foo.txt` will be converted from `%mime` to `%txt`. Additionally, data fetched by Iris over HTTP will come in as a `$mime-data:http`, which is an unvalidated form of `$mime` that you may wish to convert to a `%mime` `mark` and then to another `mark`. Likewise with Eyre, some of the lower-level interfaces receive HTTP requests with `$mime-data:http` in them.
70+
71+So with the nature of the `%mime` `mark` hopefully now clear, the reason we want conversion methods to and from `%mime` in our `%csv` `mark` is so we can import CSV files from Unix and vice versa.
72+
73+Since a CSV file on Unix will just be a long string with ASCII or UTF-8 encoding, we can treat `q.q` in the `$mime` as a `cord`, and thus write a parser to convert it to a `(list (list @t))`. For this purpose, here's a library: `csv.hoon`, which you can view in full on the [Examples](/docs/arvo/clay/marks/examples#libcsvhoon) page.
74+
75+The library contains four functions:
76+
77+- `+de-csv` - Parse a CSV `cord` to a `(list (list @t))`.
78+- `+en-csv` - Encode a `(list (list @t))` as a CSV `cord`.
79+- `+validate` - Check all rows of `(list (list @t))` are the same length.
80+- `+csv-join` - Ignore this for now, we'll use it later on.
81+
82+The decoding and encoding arms use parsing functions from the Hoon standard library. It's not important to be familiar with parsing in Hoon for our purposes here, but you can have a look at the [Parsing Guide](/docs/hoon/guides/parsing) in the Hoon documentation if you're interested. The important thing to note is that `+de-csv` takes a valid CSV-format `@t` and returns a `(list (list @t))`, and `+en-csv` does the reverse - it takes a `(list (list @t))` and returns a CSV-format `@t`.
83+
84+Let's try the library in the dojo. After we've added it to `/lib` and run `|commit`, we can build the file:
85+
86+```
87+> =csv -build-file %/lib/csv/hoon
88+```
89+
90+...try decode a CSV-format `@t`:
91+
92+```
93+> (de-csv:csv 'foo,bar,baz\0ablah,blah,blah\0a1,2,3')
94+~[<|foo bar baz|> <|blah blah blah|> <|1 2 3|>]
95+```
96+
97+...and try encode a `(list (list @t))` as a CSV-format `@t`:
98+
99+```
100+> (en-csv:csv [['foo' 'bar' 'baz' ~] ['blah' 'blah' 'blah' ~] ['1' '2' '3' ~] ~])
101+'foo,bar,baz\0ablah,blah,blah\0a1,2,3\0a'
102+```
103+
104+With that working, we can add an import for our library to our `%csv` `mark` defintion and add a `+mime` arm to both our `+grab` and `+grow` arms:
105+
106+```hoon
107+/+ *csv
108+|_ csv=(list (list @t))
109+++ grab
110+ |%
111+ ++ mime |=((pair mite octs) (de-csv q.q))
112+ ++ noun
113+ |= n=*
114+ ^- (list (list @t))
115+ =/ result ((list (list @t)) n)
116+ ?> (validate result)
117+ result
118+ --
119+++ grow
120+ |%
121+ ++ mime
122+ ?> (validate csv)
123+ [/text/csv (as-octs:mimes:html (en-csv csv))]
124+ ++ noun
125+ ?> (validate csv)
126+ csv
127+ --
128+++ grad %noun
129+--
130+```
131+
132+In `+grab` we've added a `+mime` arm to convert _from_ a `%mime` `mark` _to_ our `%csv` `mark`. It's a simple gate that takes a `$mime` (specified as `(pair mite octs)` to avoid conflict with the arm name), runs the data through the `+de-csv` function and returns a `(list (list @t))` of the CSV data.
133+
134+We've also added a `+mime` arm to `+grow` for converting _from_ our `%csv` `mark` _to_ a `%mime` `mark`. We encode our `(list (list @t))` `csv` sample with our `+en-csv` function and then run that through `as-octs:mimes:html` to get a `$octs` (so it has the byte-length). We also add the `/text/csv` MIME type so it's a valid `$mime`.
135+
136+Additionally, we've used the `+validate` function in a few places to make sure our CSV data has consistent row lengths.
137+
138+If we save the above mark file as `csv.hoon` in `/mar` and `|commit %home`, we should now be able to import CSV files into Urbit. Let's give it a go. In the root of our `%home` `desk`, let's add a file named `foo.csv` with the following contents:
139+
140+```
141+foo,bar,baz
142+blah,blah,blah
143+1,2,3
144+```
145+
146+If we now `|commit %home`, we should see it's been successfully added:
147+
148+```
149+> |commit %home
150+>=
151++ /~zod/home/4/foo/csv
152+```
153+
154+And if we try reading the file with the `-read` thread:
155+
156+```
157+> -read [%x our %home da+now /foo/csv]
158+~[<|foo bar baz|> <|blah blah blah|> <|1 2 3|>]
159+```
160+
161+We can see our `%csv` `mark` has successfully converted our `foo.csv` file to a `(list (list @t))` when it was imported.
162+
163+Let's try the other direction now. We can create a new `bar.csv` files in the root of `%home` from the dojo like so:
164+
165+```
166+> */bar/csv ~[['abc' 'def' ~] ['ghi' 'jkl' ~]]
167++ /~zod/home/5/bar/csv
168+```
169+
170+And if we check it in the terminal on the Unix side we can see it's been correctly encoded:
171+
172+```
173+> cat zod/home/bar.csv
174+abc,def
175+ghi,jkl
176+```
177+
178+So now our `%csv` `mark` lets us move data in and out of Urbit. In the next section, we'll look at the `+grad` arm in more detail.
179+
180+## `+grad`
181+
182+So far we've just delegated `+grad` functions to the `%noun` `mark`, but now we'll look at writing our own.
183+
184+For demonstrative purposes, we can just poach the algorithms used in the `+grad` arm of the `%txt` `mark` and modify them to take our `(list (list @t))` type instead of a `wain`. It's not the most efficient algorithm for a CSV file but it'll do the job.
185+
186+Our diff format will be a `(urge:clay (list @t))`, and we'll use some `differ` functions from `zuse.hoon` like `+loss`, `+lusk` and `+lurk` to produce diffs and apply patches.
187+
188+The [csv.hoon library](/docs/arvo/clay/marks/examples#libcsvhoon) we imported also contains a `+csv-join` function which we'll use in the `+join` arm, just to save space here.
189+
190+Here's the new `%csv` `mark` defintion:
191+
192+```hoon
193+/+ *csv
194+|_ csv=(list (list @t))
195+++ grab
196+ |%
197+ ++ mime |=((pair mite octs) (de-csv q.q))
198+ ++ noun
199+ |= n=*
200+ ^- (list (list @t))
201+ =/ result ((list (list @t)) n)
202+ ?> (validate result)
203+ result
204+ --
205+++ grow
206+ |%
207+ ++ mime
208+ ?> (validate csv)
209+ [/text/csv (as-octs:mimes:html (en-csv csv))]
210+ ++ noun
211+ ?> (validate csv)
212+ csv
213+ --
214+++ grad
215+ |%
216+ ++ form %csv-diff
217+ ++ diff
218+ |= bob=(list (list @t))
219+ ^- (urge:clay (list @t))
220+ ?> (validate csv)
221+ ?> (validate bob)
222+ (lusk:differ csv bob (loss:differ csv bob))
223+ ++ pact
224+ |= dif=(urge:clay (list @t))
225+ ^- (list (list @t))
226+ =/ result (lurk:differ csv dif)
227+ ?> (validate result)
228+ result
229+ ++ join
230+ |= $: ali=(urge:clay (list @t))
231+ bob=(urge:clay (list @t))
232+ ==
233+ ^- (unit (urge:clay (list @t)))
234+ (csv-join ali bob)
235+ ++ mash
236+ |= $: [ship desk (urge:clay (list @t))]
237+ [ship desk (urge:clay (list @t))]
238+ ==
239+ ^- (urge:clay (list @t))
240+ ~|(%csv-mash !!)
241+ --
242+--
243+```
244+
245+In our modified `+grad` arm, we've replaced the `%noun` delegation with a core containing five arms: `+form`, `+diff`, `+pact`, `+join`, and `+mash`. These arms are all required for a valid `+grad` if it's not delegated to another `mark`. We'll now look at each in detail.
246+
247+### `+form`
248+
249+```hoon
250+++ form %csv-diff
251+```
252+
253+`+form` simply specifies the `mark` of the diff file that may be produced by other `+grad` functions. If your diff is the same type as your `mark`, it could just specify itself like `%csv`. In our case our diff is a `(urge:clay (list @t))` rather than a `(list (list @t))`, so we need a separate mark file for the diff itself.
254+
255+Here's another `mark` file which can be saved as `csv-diff.hoon` in `/mar`:
256+
257+```hoon
258+|_ dif=(urge:clay (list @t))
259+++ grab
260+ |%
261+ ++ noun (urge:clay (list @t))
262+ --
263+++ grow
264+ |%
265+ ++ noun dif
266+ --
267+++ grad %noun
268+--
269+```
270+
271+It's very bare-bones, we just need it for our `%csv` `mark` to work. In our `%csv` `mark`, we've specified it as `%csv-diff` in `+form`.
272+
273+### `+diff`
274+
275+```hoon
276+++ diff
277+ |= bob=(list (list @t))
278+ ^- (urge:clay (list @t))
279+ ?> (validate csv)
280+ ?> (validate bob)
281+ (lusk:differ csv bob (loss:differ csv bob))
282+```
283+
284+This arm produces the diff of two `%csv` files. The first `%csv` file will be given as the sample of the parent door, which if you'll recall we gave a face of `csv`. The second `%csv` file will be given as the sample of the gate in `+diff`, which we've named `bob` here. We then just produce the diff of these two files and return it as the type of the mark specified in `+form`, which in our case is `(urge:clay (list @t))` for a `%csv-diff`. Clay will use `+diff` when a file is revised, so it doesn't have to store a whole new copy of the file each time it's modified.
285+
286+### `+pact`
287+
288+```hoon
289+++ pact
290+ |= dif=(urge:clay (list @t))
291+ ^- (list (list @t))
292+ =/ result (lurk:differ csv dif)
293+ ?> (validate result)
294+ result
295+```
296+
297+`+pact` patches a `%csv` file with the given diff. Its gate takes a diff and applies it to the `%csv` given as the sample of the parent door (which we gave a face of `csv`). If the patch succeeds, it will return a new `%csv` file - a valid `(list (list @t))`. When we read a file that's been modified in Clay, Clay will apply all the diffs it has with `+pact` and return the resulting file.
298+
299+### `+join`
300+
301+```hoon
302+++ join
303+ |= $: ali=(urge:clay (list @t))
304+ bob=(urge:clay (list @t))
305+ ==
306+ ^- (unit (urge:clay (list @t)))
307+ (csv-join ali bob)
308+```
309+
310+The `+join` arm merges two different diffs. It takes them both as the sample of its gate (which we've named `ali` and `bob`), and returns a new diff wrapped in a `unit` like `(unit (urge:clay (list @t)))`. The `unit` will be `~` if the merge failed due to a conflict. This is used by Clay in some cases when `desk`s are merged. If diff merges are not possible for your use case, you could just have it always return `~`.
311+
312+### `+mash`
313+
314+```hoon
315+++ mash
316+ |= $: [ship desk (urge:clay (list @t))]
317+ [ship desk (urge:clay (list @t))]
318+ ==
319+ ^- (urge:clay (list @t))
320+ ~|(%csv-mash !!)
321+```
322+
323+This is like `+join` except it forces a diff merge even if there's a conflict. Rather than returning a `unit`, it just returns the diff - a `(urge:clay (list @t))` in our case. Also unlike `+join`, it takes the `ship` and `desk` each diff came from as well as the diff itself.
324+
325+The `+mash` arm is not used by Clay in its file revision operations, so it's safe to just make it a dummy arm that crashes as we've done here. If you were to use it, it would likely just be used manually in an agent, thread or generator.
326+
327+An example of its use would be the `%txt` `mark`, which includes a proper `+mash` function that produces a diff with any conflicts annotated, though how you have `+mash` handle conficts would depend on your use case. If there were no conflicts between the two diffs, it should produce the same diff as the `+join` arm.
328+
329+## Conclusion
330+
331+So there you have it, a fully functional `mark` for CSV files. A `mark` file can be as complex or as simple as you'd like, they're very flexible depending on your use case. Additional conversion methods can always be added as they're needed. For example, with just a few lines of code we could add arms for converting CSV files to `%json` or `%txt` and vice versa.
332+
333+In the next document, we'll look at building and using `mark` cores and `mark` conversion gates in our own code.