c6ccc64 · urbit.org · git.iko.soy

git.iko.soy/urbit.org

on master help sign in

add marks section to clay docs (#769)

c6ccc64tinnus-napbus5 years ago

content/docs/arvo/clay/architecture.md+12−88

compare with master

unifiedsplit

@@ -23,7 +23,7 @@ state is persistent.

23In a more traditional OS, everything in RAM can be erased at any

24time by power failure, and is always erased on reboot. Thus, a

25primary purpose of a filesystem is to ensure files persist across

26−power failures and reboots. In Arvo, both power failures and

26+power failures and reboots. In Arvo, both power failures and

27reboots are special cases of suspending computation, which is

28done safely since our event log is already persistent. Therefore,

29Clay is not needed in Arvo for persistence. Why, then, do we have a

@@ -37,7 +37,7 @@ tree. It is easy and intuitive to read from and write to a

37filesystem tree.

39Second, Clay has a distributed revision-control system baked into

40−it. Traditional filesystems are not revision controlled, so

40+it. Traditional filesystems are not revision controlled, so

41userspace software -- such as git -- is written on top of them to

42do so. Clay natively provides the same functionality as modern

43DVCSes, and more.

@@ -101,25 +101,25 @@ the topmost commit is always numbered, and every numbered commit

101is an ancestor of every later numbered commit.

102

103There are three ways to refer to particular commits in the

104−revision history. Firstly, one can use the revision number.

104+revision history. Firstly, one can use the revision number.

105Secondly, one can use any absolute time between the one numbered

106commit and the next (inclusive of the first, exclusive of the

107second). Thirdly, every `desk` has a `map` of labels to revision

108numbers. These labels may be used to refer to specific commits.

109

110Additionally, Clay is a global filesystem, so data on other urbits

111−is easily accessible the same way as data on our local urbit. In

111+is easily accessible the same way as data on our local urbit. In

112general, the path to a particular revision of a `desk` is

113−`/~urbit-name/desk-name/revision`. Thus, to get `/try/readme/md`

113+`/~urbit-name/desk-name/revision`. Thus, to get `/try/readme/md`

114from revision 5 of the home `desk` on `~sampel-sipnym`, we refer to

115−`/~sampel-sipnym/home/5/try/readme/md`. Clay's namespace is thus

115+`/~sampel-sipnym/home/5/try/readme/md`. Clay's namespace is thus

116global and referentially transparent.

117

118### A Typed Filesystem

119

120Since Clay is a general filesystem for storing data of arbitrary

121types, in order to revision control correctly it needs to be

122−aware of types all the way through. Traditional revision control

122+aware of types all the way through. Traditional revision control

123does an excellent job of handling source code, so for source code

124we act very similar to traditional revision control. The

125challenge is to handle other data similarly well.

@@ -145,7 +145,7 @@ branches and merged at will.

145Suppose Alice is tasked with touching up a picture, improving the

146color balance, adjusting the contrast, and so forth, while Bob

147has the job of cropping the picture to fit where it's needed and

148−adding textual overlay. Without type-aware revision control,

148+adding textual overlay. Without type-aware revision control,

149these changes must be made serially, requiring Alice and Bob to

150explicitly coordinate their efforts. With type-aware revision

151control, these operations may be performed in parallel, and then

@@ -168,87 +168,11 @@ different sentences may be flagged as a conflict. In general,

168prose should be diffed by sentence, not by line.

169

170As far as we are aware, Clay is the first generalized,

171−type-aware revision control system. We'll go into the workings

171+type-aware revision control system. We'll go into the workings

172of this system in some detail.

173

174### Marks

175

176−Central to a typed filesystem is the idea of types. In Clay, we

176+Central to a typed filesystem is the idea of file types. In Clay, we

177−call these `mark`s. A `mark` is a file that defines a type,

177+call these `mark`s. See the [Marks](/docs/arvo/clay/marks/marks)

178−conversion routines to and from the `mark`, and diff, patch, and

178+section for more details.

179−merge routines.

180−

181−For example, a `%txt` mark may be a list of lines of text, and it

182−may include conversions to `%mime` to allow it to be serialized

183−and sent to a browser or to the Unix filesystem. It will also

184−include Hunt-McIlroy diff, patch, and merge algorithms.

185−

186−A `%json` `mark` would be defined as a JSON object in the code, and

187−it would have a parser to convert from `%txt` and a printer to

188−convert back to `%txt`. The diff, patch, and merge algorithms are

189−fairly straightforward for JSON, though they're very different

190−from the text ones.

191−

192−More formally, a `mark` is a core with three [arms](/docs/glossary/arm/): `+grab`,

193−`+grow`, and `+grad`. In `+grab` is a series of functions to

194−convert from other `mark`s to the given `mark`. In `+grow` is a

195−series of functions to convert from the given `mark` to other

196−`mark`s. In `+grad` is `+diff`, `+pact`, `+join`, and `+mash`, and

197−`+form`.

198−

199−The types are as follows, in an informal pseudocode:

200−

201−```

202− ++ grab:

203− ++ mime: <mime> -> <mark-type>

204− ++ txt: <txt> -> <mark-type>

205− ...

206− ++ grow:

207− ++ mime: <mark-type> -> <mime>

208− ++ txt: <mark-type> -> <txt>

209− ...

210− ++ grad

211− ++ form: <mark-type>

212− ++ diff: (<mark-type>, <mark-type>) -> <diff-type>

213− ++ pact: (<mark-type>, <diff-type>) -> <mark-type>

214− ++ join: (<diff-type>, <diff-type>) -> <diff-type> or NULL

215− ++ mash: (<diff-type>, <diff-type>) -> <diff-type>

216−```

217−

218−These types are basically what you would expect. Not every `mark`

219−has each of these functions defined -- all of them are optional

220−in the general case.

221−

222−In general, for a particular `mark`, the `+grab` and `+grow` entries

223−(if they exist) should be inverses of each other.

224−

225−In `+grad`, `+diff` takes two instances of a `mark` and produces a diff of them

226−whose `mark` is given by `+form`. `+pact` takes an instance of a `mark` and

227−patches it with the given diff. `+join` takes two diffs and attempts to merge

228−them into a single diff. If there are conflicts, it produces null. `+mash` takes

229−two diffs and forces a merge, annotating any conflicts.

230−

231−In general, if `+diff` called with A and B produces diff D, then

232−`+pact` called with A and D should produce B. Also, if `+join`

233−of two diffs does not produce null, then `+mash` of the same

234−diffs should produce the same result.

235−

236−Alternately, instead of `+diff`, `+pact`, `+join`, and

237−`+mash`, a `mark` can provide the same functionality by defining

238−`+grad` to be the name of another `mark` to which we wish to

239−delegate the revision control responsibilities. Then, before

240−running any of those functions, Clay will convert to the other

241−`mark`, and convert back afterward. For example, the `%hoon` `mark`

242−is revision-controlled in the same way as `%txt`, so its `+grad`

243−is simply `++ grad %txt`. Of course, `+txt` must be defined in

244−`+grow` and `+grab` as well.

245−

246−Every file in Clay has a `mark`, and that `mark` must have a

247−fully-functioning `+grad`. `Mark`s are used for more than just

248−Clay, and other `mark`s don't need a `+grad`, but if a piece of

249−data is to be saved to Clay, we must know how to revision-control

250−it.

251−

252−Additionally, if a file is to be synced out to Unix, then it must

253−have conversion routines to and from the `%mime` `mark`.

254−

content/docs/arvo/clay/clay.md+4−1

@@ -5,7 +5,6 @@ template = "doc.html"

5aliases = ["/docs/learn/arvo/clay/"]

6+++

8−

9Our filesystem.

8Our filesystem.

11`%clay` is version-controlled, referentially-transparent, and global.

10`%clay` is version-controlled, referentially-transparent, and global.

@@ -56,3 +55,7 @@ Details of the various `task`s you can use to interact with Clay.

56[Examples](/docs/arvo/clay/examples)

55[Examples](/docs/arvo/clay/examples)

58Example usage of the various Clay `task`s.

57Example usage of the various Clay `task`s.

58+

59+[Marks](/docs/arvo/clay/marks/marks)

60+

61+Details of `mark`s as well as a guide to writing them and their usage.

content/docs/arvo/clay/marks/_index.mdnew file+23

@@ -0,0 +1,23 @@

1++++

2+title = "Marks"

3+weight = 10

4+sort_by = "weight"

5+template = "sections/docs/chapters.html"

6+insert_anchor_links = "right"

7++++

9+[Overview](/docs/arvo/clay/marks/marks)

10+

11+Overview of `mark`s.

12+

13+[Writing Marks](/docs/arvo/clay/marks/writing-marks)

14+

15+A practical walk-through of writing a `mark` file.

16+

17+[Using Marks](/docs/arvo/clay/marks/using-marks)

18+

19+Details of using `mark` conversion gates and `mark` cores in your own code.

20+

21+[Examples](/docs/arvo/clay/marks/examples)

22+

23+The example code used in the Writing Marks guide.

content/docs/arvo/clay/marks/examples.mdnew file+248

@@ -0,0 +1,248 @@

1++++

2+title = "Examples"

3+weight = 4

4+template = "doc.html"

5++++

7+These are the files used in the [Writing Marks](/docs/arvo/clay/marks/writing-marks) guide.

9+## `/lib/csv/hoon`

10+

11+```hoon

12+|%

13+++ validate :: All rows same length?

14+ |= csv=(list (list @t))

15+ ^- ?

16+ =/ result

17+ %+ roll csv

18+ |= [row=(list @t) =flag ref=(unit @ud)]

19+ ?. flag

20+ [flag ref]

21+ ?~ ref

22+ [flag `(lent row)]

23+ ?. =(u.ref (lent row))

24+ [%.n ref]

25+ [flag ref]

26+ flag.result

27+++ en-csv :: csv -> atom (to mime)

28+ |= csv=(list (list @t))

29+ |^ ^- @t

30+ (cat 3 (crip (join '\0a' (turn csv en-row))) '\0a')

31+ ++ en-row :: encode row

32+ |= row=(list @t)

33+ ^- @t

34+ (crip (join ',' (turn row en-field)))

35+ ++ en-field :: encode field

36+ |= field=@t

37+ ^- @t

38+ %+ rash

39+ field

40+ %+ cook

41+ |= =tape

42+ ^- @t

43+ ?~ tape ''

44+ ?: =('"' i.tape)

45+ (cat 3 (crip tape) '"')

46+ (crip tape)

47+ ;~ pose

48+ (full (star ;~(less (mask "\",\0a\0d") next)))

49+ ;~ plug

50+ (easy '"')

51+ (star ;~(pose (cold '""' doq) next))

52+ ==

53+ ==

54+ --

55+++ de-csv :: atom -> csv (from mime)

56+ |= dat=@t

57+ |^ ^- (list (list @t))

58+ =/ parsed (parse dat)

59+ ?. (validate parsed)

60+ ~|(%mixed-col-count !!)

61+ parsed

62+ ++ parse :: parse cord

63+ |= dat=@t

64+ |^ ^- (list (list @t))

65+ %+ rash

66+ dat

67+ %+ most

68+ ;~(pose (jest '\0d\0a') (mask "\0a\0d"))

69+ (most com parse-field)

70+ ++ parse-field :: parse field from cord

71+ |^

72+ %+ cook

73+ |=(a=tape (crip a))

74+ ;~(pose unenclosed enclosed)

75+ ++ unenclosed :: unquoted field

76+ ;~ sfix

77+ (star ;~(less (mask "\",\0a\0d") next))

78+ ;~ pose

79+ ;~ plug

80+ ;~(pose (jest '\0d\0a') (mask "\0a\0d"))

81+ ;~(less next (easy ~))

82+ ==

83+ ;~(less doq (easy ~))

84+ ==

85+ ==

86+ ++ enclosed :: quoted field

87+ |^

88+ ;~ sfix

89+ (ifix [prefix suffix] content)

90+ ;~ pose

91+ ;~(simu (mask ",\0a\0d") (easy ~))

92+ ;~ plug

93+ ;~(pose (jest '\0d\0a') (mask "\0a\0d"))

94+ ;~(less next (easy ~))

95+ ==

96+ ;~(less next (easy ~))

97+ ==

98+ ==

99+ ++ content :: quoted field contents

100+ %- star

101+ ;~ pose

102+ (cold '"' (jest '""'))

103+ %+ cold '\0a'

104+ ;~(pose (jest '\0d\0a') (just '\0d'))

105+ ;~(less doq next)

106+ ==

107+ ++ prefix ;~(pose ;~(plug (star ace) doq) doq) :: quoted field prefix

108+ ++ suffix ;~(pose ;~(plug doq (star ace)) doq) :: quoted field suffix

109+ --

110+ --

111+ --

112+ --

113+++ csv-join

114+ |= [ali=(urge:clay (list @t)) bob=(urge:clay (list @t))]

115+ ^- (unit (urge:clay (list @t)))

116+ |^

117+ =. ali (clean ali)

118+ =. bob (clean bob)

119+ |- ^- (unit (urge:clay (list @t)))

120+ ?~ ali `bob

121+ ?~ bob `ali

122+ ?- -.i.ali

123+ %&

124+ ?- -.i.bob

125+ %&

126+ ?: =(p.i.ali p.i.bob)

127+ %+ bind $(ali t.ali, bob t.bob)

128+ |=(cud=(urge:clay (list @t)) [i.ali cud])

129+ ?: (gth p.i.ali p.i.bob)

130+ %+ bind $(p.i.ali (sub p.i.ali p.i.bob), bob t.bob)

131+ |=(cud=(urge:clay (list @t)) [i.bob cud])

132+ %+ bind $(ali t.ali, p.i.bob (sub p.i.bob p.i.ali))

133+ |=(cud=(urge:clay (list @t)) [i.ali cud])

134+ ::

135+ %|

136+ ?: =(p.i.ali (lent p.i.bob))

137+ %+ bind $(ali t.ali, bob t.bob)

138+ |=(cud=(urge:clay (list @t)) [i.bob cud])

139+ ?: (gth p.i.ali (lent p.i.bob))

140+ %+ bind $(p.i.ali (sub p.i.ali (lent p.i.bob)), bob t.bob)

141+ |=(cud=(urge:clay (list @t)) [i.bob cud])

142+ ~

143+ ==

144+ ::

145+ %|

146+ ?- -.i.bob

147+ %|

148+ ?. =(i.ali i.bob)

149+ ~

150+ %+ bind $(ali t.ali, bob t.bob)

151+ |=(cud=(urge:clay (list @t)) [i.ali cud])

152+ ::

153+ %&

154+ ?: =(p.i.bob (lent p.i.ali))

155+ %+ bind $(ali t.ali, bob t.bob)

156+ |=(cud=(urge:clay (list @t)) [i.ali cud])

157+ ?: (gth p.i.bob (lent p.i.ali))

158+ %+ bind $(ali t.ali, p.i.bob (sub p.i.bob (lent p.i.ali)))

159+ |=(cud=(urge:clay (list @t)) [i.ali cud])

160+ ~

161+ ==

162+ ==

163+ ++ clean :: clean

164+ |= wig=(urge:clay (list @t))

165+ ^- (urge:clay (list @t))

166+ ?~ wig ~

167+ ?~ t.wig wig

168+ ?: ?=(%& -.i.wig)

169+ ?: ?=(%& -.i.t.wig)

170+ $(wig [[%& (add p.i.wig p.i.t.wig)] t.t.wig])

171+ [i.wig $(wig t.wig)]

172+ ?: ?=(%| -.i.t.wig)

173+ $(wig [[%| (welp p.i.wig p.i.t.wig) (welp q.i.wig q.i.t.wig)] t.t.wig])

174+ [i.wig $(wig t.wig)]

175+ --

176+--

177+```

178+

179+## `/mar/csv/hoon`

180+

181+```hoon

182+/+ *csv

183+|_ csv=(list (list @t))

184+++ grab

185+ |%

186+ ++ mime |=((pair mite octs) (de-csv q.q))

187+ ++ noun

188+ |= n=*

189+ ^- (list (list @t))

190+ =/ result ((list (list @t)) n)

191+ ?> (validate result)

192+ result

193+ --

194+++ grow

195+ |%

196+ ++ mime

197+ ?> (validate csv)

198+ [/text/csv (as-octs:mimes:html (en-csv csv))]

199+ ++ noun

200+ ?> (validate csv)

201+ csv

202+ --

203+++ grad

204+ |%

205+ ++ form %csv-diff

206+ ++ diff

207+ |= bob=(list (list @t))

208+ ^- (urge:clay (list @t))

209+ ?> (validate csv)

210+ ?> (validate bob)

211+ (lusk:differ csv bob (loss:differ csv bob))

212+ ++ pact

213+ |= dif=(urge:clay (list @t))

214+ ^- (list (list @t))

215+ =/ result (lurk:differ csv dif)

216+ ?> (validate result)

217+ result

218+ ++ join

219+ |= $: ali=(urge:clay (list @t))

220+ bob=(urge:clay (list @t))

221+ ==

222+ ^- (unit (urge:clay (list @t)))

223+ (csv-join ali bob)

224+ ++ mash

225+ |= $: [ship desk (urge:clay (list @t))]

226+ [ship desk (urge:clay (list @t))]

227+ ==

228+ ^- (urge:clay (list @t))

229+ ~|(%csv-mash !!)

230+ --

231+--

232+```

233+

234+## `/mar/csv-diff/hoon`

235+

236+```hoon

237+|_ dif=(urge:clay (list @t))

238+++ grab

239+ |%

240+ ++ noun (urge:clay (list @t))

241+ --

242+++ grow

243+ |%

244+ ++ noun dif

245+ --

246+++ grad %noun

247+--

248+```

content/docs/arvo/clay/marks/marks.mdnew file+59

@@ -0,0 +1,59 @@

1++++

2+title = "Overview"

3+weight = 1

4+template = "doc.html"

5++++

7+Clay is a typed filesystem, and we call these file types `mark`s. When talking about Hoon and Arvo we'll often talk of types like `@ud`, `(list @t)`, etc. A `mark` will specify such a type for its files, but it does more than just that - it also defines conversion routines to and from other `mark`s, as well as diff, patch, and merge routines.

9+For example, a `%txt` `mark` defines the type of a `%txt` file as a `wain` (a `(list @t)`). It defines a conversion function to a `%mime` `mark` to allow it to be serialized and sent to a browser or to the Unix filesystem. It also includes Hunt-McIlroy diff, patch, and merge algorithms. Conversion functions will be different for different `mark`s, as will things like diff algorithms. An image file like a `%png`, for example, just replaces the old blob of data with the new one rather than implementing a complex binary diff and patch algorithm, so how these functions are implemented depends on the file type and use case.

10+

11+`mark` files are stored in the `/mar` directory. The `path` of the `%txt` `mark`, for example, is `/mar/txt/hoon`.

12+

13+A `mark` is a door (a core with a sample) with three arms: `+grab`, `+grow`, and `+grad`. The door's sample defines its type. In `+grab` is a series of functions to convert from other `mark`s to the given `mark`. In `+grow` is a series of functions to convert from the given `mark` to other `mark`s. In `+grad` is `+diff`, `+pact`, `+join`, `+mash`, and `+form`.

14+

15+Here's its basic structure in an informal pseudocode:

16+

17+```

18+|_ <mark-type>

19+++ grab:

20+ ++ noun: <noun> -> <mark-type>

21+ ++ mime: <mime> -> <mark-type>

22+ ++ txt: <txt> -> <mark-type>

23+ ...

24+++ grow:

25+ ++ noun: <mark-type> -> <noun>

26+ ++ mime: <mark-type> -> <mime>

27+ ++ txt: <mark-type> -> <txt>

28+ ...

29+++ grad

30+ ++ form: <diff-mark>

31+ ++ diff: (<mark-type>, <mark-type>) -> <diff-type>

32+ ++ pact: (<mark-type>, <diff-type>) -> <mark-type>

33+ ++ join: (<diff-type>, <diff-type>) -> <diff-type> or NULL

34+ ++ mash: (<diff-type>, <diff-type>) -> <diff-type>

35+```

36+

37+These types are basically what you would expect. In `+grab`, only a `+noun` arm is mandatory, the rest are optional. The `+grow` arm itself is optional, as are any arms within it. In `+grad`, all arms are mandatory unless revision control is delegated to another `mark`, which we'll discuss later.

38+

39+In general, for a particular `mark`, the `+grab` and `+grow` entries should be inverses of each other. They needn't be symmetrical though - you may want to be able to convert from your `mark` to `%json` but not from `%json` to your `mark`, for example.

40+

41+In `+grad`, `+diff` takes two instances of a `mark` and produces a diff of them whose `mark` is given by `+form`. `+pact` takes an instance of a `mark` and patches it with the given diff. In general, if `+diff` called with A and B produces diff D, then `+pact` called with A and D should produce B.

42+

43+`+join` takes two diffs and attempts to merge them into a single diff. If there are conflicts, it produces null. `+mash` takes two diffs and forces a merge, even if there are conflicts. How your `+mash` function force-merges conflicting diffs is up to you. The `%txt` `mark` annotates any conflicts, for example, but there may be other ways. If `+join` of two diffs does not produce null, then `+mash` of the same diffs should produce the same result. Note that `+mash` is not used by Clay in its ordinary filesystem operations, so you may wish to leave it as a dummy arm that crashes if called.

44+

45+Alternately, instead of `+form`, `+diff`, `+pact`, `+join`, and `+mash`, a `mark` can provide the same functionality by defining `+grad` to be the name of another `mark` to which we wish to delegate the revision control responsibilities. Then, before running any of those functions, Clay will convert to the other `mark`, and convert back afterward. For example, the `%hoon` `mark` is revision-controlled in the same way as `%txt`, so its `+grad` is simply `++ grad %txt`. Of course, `+txt` must be defined in `+grow` and `+grab` as well.

46+

47+We mentioned that `+noun` is the only mandatory arm in `+grab`, but there are a couple of others that, while not mandatory, are of particular interest.

48+

49+The first is a `+mime` arm for converting to and from the `%mime` `mark`. When you `|commit` a file to a `desk` mounted to Unix, Clay will receive the data as a `%mime` `mark`, and then convert it to the `mark` matching the file extension. It will perform the same operation in reverse when mounting a `desk` to Unix. For this reason, any `mark` you wish to be able to access from the Unix filesystem should have `%mime` conversion routines. In certain cases (such as the scry interface), Eyre will also need to convert your `mark` to a `%mime` in order to encode it in an HTTP response, so you may require a `+mime` arm for that reason as well.

50+

51+The second case of interest is the `+json` arm for converting to and from a `%json` `mark`. If, for example, you want to write a Gall agent to which you can subscribe through Eyre's channel system, it must produce data with a `mark` containing `%json` conversion routines. If it doesn't, Eyre will not be able to deliver the data to the subscribed HTTP client in the SSE stream.

52+

53+## Sections

54+

55+[Writing Marks](/docs/arvo/clay/marks/writing-marks) - A practical walk-through of writing a `mark` file.

56+

57+[Using Marks](/docs/arvo/clay/marks/using-marks) - Details of using `mark` conversion gates and `mark` cores in your own code.

58+

59+[Examples](/docs/arvo/clay/marks/examples) - The example code used in the Writing Marks guide.

content/docs/arvo/clay/marks/using-marks.mdnew file+240

@@ -0,0 +1,240 @@

1++++

2+title = "Using Marks"

3+weight = 3

4+template = "doc.html"

5++++

7+In the last document, [Writing Marks](/docs/arvo/clay/marks/writing-marks), we walked through writing `mark` files and touched on how Clay handles them. They needn't just be left to background vane processes though, you can also use them yourself in your code.

9+There are two kinds of cores that Clay can build for you: A `mark` conversion gate and a `mark` core. Each has two kinds: Statically typed and dynamically typed. Clay has a `care` for producing each of these:

10+

11+- `%b` - Build a dynamically typed `mark` core.

12+- `%c` - Build a dynamically typed `mark` conversion gate.

13+- `%e` - Build a statically typed `mark` core.

14+- `%f` - Build a statically typed `mark` conversion gate.

15+

16+You can either use these by `%pass`ing Clay a [%warp task](/docs/arvo/clay/tasks#warp) with the appropriate `care`, or else with a [Clay scry](/docs/arvo/clay/scry). In the examples here we've used the latter.

17+

18+## mark conversion gates

19+

20+`mark` conversion gates simply convert from one `mark` to another.

21+

22+### Static

23+

24+A static `mark` conversion gate looks like `$-(a b)`, where `a` is the type of the `mark` you're converting _from_, and `b` is type of the `mark` you're converting _to_. For example, a `mark` conversion gate from `%txt` to `%mime` would look like `$-(wain mime)`. You'd simply feed it a `wain` and get a `$mime` in return.

25+

26+#### Example

27+

28+We get our `%txt` to `%mime` `mark` conversion gate with a `%f` scry like so:

29+

30+```

31+> =txt-to-mime .^($-(wain mime) %cf /===/txt/mime)

32+```

33+

34+Note we had to specify the type of the gate as `$-(wain mime)` in the scry - if the type returned by the scry doesn't match that specification it'll fail. This is where a statically typed `mark` conversion gate differs from the dynamically typed gate, which we'll discuss later.

35+

36+Now that we have our conversion gate, we can just call it with a valid `wain` and we'll get our `$mime` in return:

37+

38+```

39+> (txt-to-mime ~['foo'])

40+[p=/text/plain q=[p=3 q=7.303.014]]

41+```

42+

43+### Dynamic

44+

45+A dynamically typed `mark` conversion gate is called a `$tube:clay`, and looks like:

46+

47+```hoon

48++$ tube $-(vase vase)

49+```

50+

51+As you can see from the type definition, it takes and returns a `vase` rather than needing the types explicitly defined. Rather than failing on a type mismatch when the scry is performed, it'll instead fail when it's actually run and fed a `vase` of the wrong type. Apart from handling `vase`s, it otherwise behaves the same as a statically typed `mark` conversion gate.

52+

53+#### Example

54+

55+We get our `%txt` to `%mime` `$tube` with a `%c` scry like so:

56+

57+```

58+> =txt-mime-tube .^(tube:clay %cc /===/txt/mime)

59+```

60+

61+And then we can again feed it the `wain` that a `%txt` `mark` wants, only this time it's wrapped in a `vase`:

62+

63+```

64+> !< mime (txt-mime-tube !>(~['foo']))

65+[p=/text/plain q=[p=3 q=7.303.014]]

66+```

67+

68+We then get our `$mime` back, but also in a `vase`.

69+

70+## mark cores

71+

72+While a `mark` conversion gate is built from functions defined in `+grab` and `+grow`, a `mark` core gives you everything in `+grad` so you can create diffs, merge diffs, patch files, etc. An extra arm `+vale` is also included that lets you convert a `noun` to the type the `%mark` takes by running the `+noun` arm of `+grab` in the original `mark` file.

73+

74+### Static

75+

76+A statically typed `mark` core is a `(nave:clay a b)` where `a` is the type of the `mark` and `b` is the type for diffs (which is the type of the `mark` specified in `+form:grad`). For example, a static `mark` core for a `%txt` `mark` looks like `(nave:clay wain (urge:clay cord))`.

77+

78+`+nave:clay` looks like this in full:

79+

80+```hoon

81+++ nave

82+ |$ [typ dif]

83+ $_

84+ ^?

85+ |%

86+ ++ diff |~([old=typ new=typ] *dif)

87+ ++ form *mark

88+ ++ join |~([a=dif b=dif] *(unit (unit dif)))

89+ ++ mash

90+ |~ [a=[ship desk dif] b=[ship desk dif]]

91+ *(unit dif)

92+ ++ pact |~([typ dif] *typ)

93+ ++ vale |~(noun *typ)

94+ --

95+```

96+

97+In brief, the arms of the core do the following:

98+

99+- `+form` - `mark` for diffs.

100+- `+vale` - Clam `noun` to the `mark`'s type.

101+- `+diff` - Create diff of two files.

102+- `+pact` - Patch a file with a diff.

103+- `+join` - Merge two diffs, returning `~` if there's a conflict.

104+- `+mash` - Force merge of two diffs.

105+

106+#### Examples

107+

108+First we get the `%txt` `mark` core with a `%e` scry:

109+

110+```

111+> =txt-nave .^((nave:clay wain (urge:clay cord)) %ce /===/txt)

112+```

113+

114+Note we specified the `mark` type `wain` and diff type `(urge:clay cord)` in the `nave` returned by the scry.

115+

116+We can see the `mark` for diffs with `+form`:

117+

118+```

119+> form.txt-nave

120+%txt-diff

121+```

122+

123+To clam a noun to the type of the `mark` (a `wain` in the case of a `%txt` `mark`), we can call `+vale` with a `noun`:

124+

125+```

126+> (vale:txt-nave ~['foo' 'bar'])

127+<|foo bar|>

128+```

129+

130+We can create a diff of two files with `+diff`:

131+

132+```

133+> (diff:txt-nave ~['foo' 'bar' 'baz'] ~['foo' 'zoo' 'baz'])

134+~[[%.y p=1] [%.n p=<|bar|> q=<|zoo|>] [%.y p=1]]

135+```

136+

137+Let's create some more diffs for experimentation:

138+

139+```

140+> =diff-a (diff:txt-nave ~['foo' 'bar' 'baz'] ~['foo' 'zoo' 'baz'])

141+> =diff-b (diff:txt-nave ~['foo' 'bar' 'baz'] ~['zap' 'bar' 'baz'])

142+> =diff-c (diff:txt-nave ~['foo' 'bar' 'baz'] ~['foo' 'bla' 'baz'])

143+```

144+

145+If we try merging diffs `a` and `b` with `+join`, we get a new merged diff back in a `unit`:

146+

147+```

148+> (join:txt-nave diff-a diff-b)

149+[~ [~ ~[[%.n p=<|foo|> q=<|zap|>] [%.n p=<|bar|> q=<|zoo|>] [%.y p=1]]]]

150+```

151+

152+If we try merging diffs `a` and `c` however, we get `~` because of a conflict:

153+

154+```

155+> (join:txt-nave diff-a diff-c)

156+[~ ~]

157+```

158+

159+If we run `+mash` on `a` and `b` we get the same diff as with `+join` (sans the `unit`):

160+

161+```

162+> (mash:txt-nave [our %home diff-a] [our %blah diff-b])

163+[~ ~[[%.n p=<|foo|> q=<|zap|>] [%.n p=<|bar|> q=<|zoo|>] [%.y p=1]]]

164+```

165+

166+If we `+mash` `a` and `c`, however, we get a diff with the conflict annotated rather than just `~`:

167+

168+```

169+> (mash:txt-nave [our %home diff-a] [our %blah diff-c])

170+[ ~

171+ ~[

172+ [%.y p=1]

173+ [ %.n

174+ p=<|bar|>

175+ q

176+ <|>>>>>>>>>>>> ~zod/home zoo ++++++++++++ bar ------------ bla <<<<<<<<<<<< ~zod/blah|>

177+ ]

178+ [%.y p=1]

179+ ]

180+]

181+```

182+

183+Note that the `%txt` `mark` annotates conflicts, but there are no specific rules around how the `+mash` arm should force-merge conflicting diffs apart from that it must return a valid diff, however the `mark` specified in `+form` defines that.

184+

185+Finally, we can patch our `wain` with diff `a` and get a new, modified `wain`:

186+

187+```

188+> (pact:txt-nave ~['foo' 'bar' 'baz'] diff-a)

189+<|foo zoo baz|>

190+```

191+

192+### Dynamic

193+

194+A dynamically typed `mark` core is a `$dais:clay`, which looks like:

195+

196+```hoon

197++$ dais

198+ $_ ^|

199+ |_ sam=vase

200+ ++ diff |~(new=_sam *vase)

201+ ++ form *mark

202+ ++ join |~([a=vase b=vase] *(unit (unit vase)))

203+ ++ mash

204+ |~ [a=[ship desk diff=vase] b=[ship desk diff=vase]]

205+ *(unit vase)

206+ ++ pact |~(diff=vase sam)

207+ ++ vale |~(noun sam)

208+ --

209+```

210+

211+It has the same arms as the statically typed `mark` core, the difference is that it takes and returns `vase`s rather than the data directly. Additionally, it's a door rather than an ordinary core, and takes the type of the `mark` wrapped in a `vase` as its argument.

212+

213+#### Examples

214+

215+We can get a `%txt` `mark` `dais` with a `%b` scry:

216+

217+```

218+> =txt-dais .^(dais:clay %cb /===/txt)

219+```

220+

221+Its `+form` arm functions the same as a static core:

222+

223+```

224+> form.txt-dais

225+%txt-diff

226+```

227+

228+So does `+vale` except it returns a `vase`:

229+

230+```

231+> !< wain (vale:txt-dais ~['foo' 'bar'])

232+<|foo bar|>

233+```

234+

235+All the other arms also deal in `vase`s. Because it's a door, the `+pact` and `+diff` arms take the original file in a `vase` as the door sample rather than being given directly to the arm. Calling `+diff` arm, for example, works like so:

236+

237+```

238+> !< (urge:clay cord) (~(diff txt-dais !>(~['foo' 'bar'])) !>(~['foo' 'baz']))

239+~[[%.y p=1] [%.n p=<|bar|> q=<|baz|>]]

240+```

content/docs/arvo/clay/marks/writing-marks.mdnew file+333

@@ -0,0 +1,333 @@

1++++

2+title = "Writing Marks"

3+weight = 2

4+template = "doc.html"

5++++

7+Here we'll walk through a practical example of writing a `mark` file.

9+We'll create a `mark` for CSV (comma separated values) files, a simple format for storing tabular data in a text file. Note that there's already a `csv.hoon` `mark` and library, but we'll create new ones for demonstrative purposes. It shouldn't be an issue to overwrite the existing ones on a fakezod.

10+

11+CSV files separate fields with commas and rows with line breaks. They look something like:

12+

13+```

14+foo,bar,baz

15+blah,blah,blah

16+1,2,3

17+```

18+

19+There is a little complexity surrounding special characters in fields and line endings, but otherwise the only other rule is that all rows must have the same number of fields. You can refer to [RFC4180 on the IETF website](https://datatracker.ietf.org/doc/html/rfc4180) for more details.

20+

21+We'll represent such a structure in Hoon as a `(list (list @t))` like:

22+

23+```hoon

24+[['foo' 'bar' 'baz' ~] ['blah' 'blah' 'blah' ~] ['1' '2' '3' ~] ~]

25+```

26+

27+We could perhaps create the type with a `$|` rune to include row-length validation in the mold itself, but a `(list (list @t))` is simpler for demonstrative purposes.

28+

29+## A simple mark

30+

31+Let's begin with the simplest `mark` file:

32+

33+```hoon

34+|_ csv=(list (list @t))

35+++ grab

36+ |%

37+ ++ noun (list (list @t))

38+ --

39+++ grow

40+ |%

41+ ++ noun csv

42+ --

43+++ grad %noun

44+--

45+```

46+

47+The door takes a `(list (list @t))` as its sample, and we've given it a face of `csv` so we can easily reference it. Note its face could be anything, it needn't be the name of our `mark`. When we're doing something with data that has a `%csv` `mark` like converting it to another `mark` or creating a diff, this is where our data will reside.

48+

49+Next we have the `+grab` arm of our door, which contains a core with arms for converting _to_ our `mark` _from_ other `mark`s. We've given it one arm for the `%noun` `mark` - the most generic `mark` which will take any `noun`. Our `+noun` arm will simply clam whatever it's given with the `(list (list @t))` `mold`.

50+

51+Next is the `+grow` arm which does the inverse of `+grab`, converting _from_ our `mark` _to_ another `mark`. We've also given it a `+noun` arm, this time it will simply return the door's sample named `csv`, which is of course already a `noun`.

52+

53+Note that the `+noun` arm is _mandatory_ in `+grab`. Clay cannot build a `mark` core without it. Conversion arms for any other `mark`s apart from `%noun` are optional.

54+

55+Finally we have the `+grad` arm. This arm specifies functions for revision control like creating diffs, patching files and so on. In our case, rather than writing all those functions, we've just delegated those tasks to the `%noun` `mark`. We can do this because we've specified conversion routines to and from the `%noun` `mark` in our `+grow` and `+grab` arms. When we modify a file with a `%csv` `mark`, Clay will convert our data to a `%noun` `mark`, execute the necessary `+grad` functions from the `%noun` `mark` file, and then convert it back to a `%csv` `mark` again.

56+

57+So now we have a valid `%csv` `mark` file. If we save this as `csv.hoon` in the `/mar` directory we could store `%csv` data in Clay. This may be sufficient for some applications, but what if we want to import a CSV file from Unix or elsewhere? In the next section, we'll look at conversions to and from a `%mime` `mark` to address this.

58+

59+## `%mime` conversions

60+

61+The `$mime` type represents raw data from Unix or elsewhere. For example, if a text file from Unix containing the word `foo` were converted to a `$mime` type in Urbit, it would look something like:

62+

63+```hoon

64+[/text/plain q=[p=3 q=7.303.014]]

65+```

66+

67+`/text/plain` is its [MIME type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types) and `p.q` is the byte-length of `q.q`, which is the data itself as an `atom`.

68+

69+The `%mime` `mark` is used by Clay to store and convert `$mime` data. It's an important `mark` for moving files from Unix to Urbit and vice versa. When you add a file to a `desk` you have mounted to Unix and `|commit` the change, Clay will first receive the file as a `%mime` `mark`, then convert from a `%mime` to whatever `mark` matches the file extension. For example, `foo.txt` will be converted from `%mime` to `%txt`. Additionally, data fetched by Iris over HTTP will come in as a `$mime-data:http`, which is an unvalidated form of `$mime` that you may wish to convert to a `%mime` `mark` and then to another `mark`. Likewise with Eyre, some of the lower-level interfaces receive HTTP requests with `$mime-data:http` in them.

70+

71+So with the nature of the `%mime` `mark` hopefully now clear, the reason we want conversion methods to and from `%mime` in our `%csv` `mark` is so we can import CSV files from Unix and vice versa.

72+

73+Since a CSV file on Unix will just be a long string with ASCII or UTF-8 encoding, we can treat `q.q` in the `$mime` as a `cord`, and thus write a parser to convert it to a `(list (list @t))`. For this purpose, here's a library: `csv.hoon`, which you can view in full on the [Examples](/docs/arvo/clay/marks/examples#libcsvhoon) page.

74+

75+The library contains four functions:

76+

77+- `+de-csv` - Parse a CSV `cord` to a `(list (list @t))`.

78+- `+en-csv` - Encode a `(list (list @t))` as a CSV `cord`.

79+- `+validate` - Check all rows of `(list (list @t))` are the same length.

80+- `+csv-join` - Ignore this for now, we'll use it later on.

81+

82+The decoding and encoding arms use parsing functions from the Hoon standard library. It's not important to be familiar with parsing in Hoon for our purposes here, but you can have a look at the [Parsing Guide](/docs/hoon/guides/parsing) in the Hoon documentation if you're interested. The important thing to note is that `+de-csv` takes a valid CSV-format `@t` and returns a `(list (list @t))`, and `+en-csv` does the reverse - it takes a `(list (list @t))` and returns a CSV-format `@t`.

83+

84+Let's try the library in the dojo. After we've added it to `/lib` and run `|commit`, we can build the file:

85+

86+```

87+> =csv -build-file %/lib/csv/hoon

88+```

89+

90+...try decode a CSV-format `@t`:

91+

92+```

93+> (de-csv:csv 'foo,bar,baz\0ablah,blah,blah\0a1,2,3')

94+~[<|foo bar baz|> <|blah blah blah|> <|1 2 3|>]

95+```

96+

97+...and try encode a `(list (list @t))` as a CSV-format `@t`:

98+

99+```

100+> (en-csv:csv [['foo' 'bar' 'baz' ~] ['blah' 'blah' 'blah' ~] ['1' '2' '3' ~] ~])

101+'foo,bar,baz\0ablah,blah,blah\0a1,2,3\0a'

102+```

103+

104+With that working, we can add an import for our library to our `%csv` `mark` defintion and add a `+mime` arm to both our `+grab` and `+grow` arms:

105+

106+```hoon

107+/+ *csv

108+|_ csv=(list (list @t))

109+++ grab

110+ |%

111+ ++ mime |=((pair mite octs) (de-csv q.q))

112+ ++ noun

113+ |= n=*

114+ ^- (list (list @t))

115+ =/ result ((list (list @t)) n)

116+ ?> (validate result)

117+ result

118+ --

119+++ grow

120+ |%

121+ ++ mime

122+ ?> (validate csv)

123+ [/text/csv (as-octs:mimes:html (en-csv csv))]

124+ ++ noun

125+ ?> (validate csv)

126+ csv

127+ --

128+++ grad %noun

129+--

130+```

131+

132+In `+grab` we've added a `+mime` arm to convert _from_ a `%mime` `mark` _to_ our `%csv` `mark`. It's a simple gate that takes a `$mime` (specified as `(pair mite octs)` to avoid conflict with the arm name), runs the data through the `+de-csv` function and returns a `(list (list @t))` of the CSV data.

133+

134+We've also added a `+mime` arm to `+grow` for converting _from_ our `%csv` `mark` _to_ a `%mime` `mark`. We encode our `(list (list @t))` `csv` sample with our `+en-csv` function and then run that through `as-octs:mimes:html` to get a `$octs` (so it has the byte-length). We also add the `/text/csv` MIME type so it's a valid `$mime`.

135+

136+Additionally, we've used the `+validate` function in a few places to make sure our CSV data has consistent row lengths.

137+

138+If we save the above mark file as `csv.hoon` in `/mar` and `|commit %home`, we should now be able to import CSV files into Urbit. Let's give it a go. In the root of our `%home` `desk`, let's add a file named `foo.csv` with the following contents:

139+

140+```

141+foo,bar,baz

142+blah,blah,blah

143+1,2,3

144+```

145+

146+If we now `|commit %home`, we should see it's been successfully added:

147+

148+```

149+> |commit %home

150+>=

151++ /~zod/home/4/foo/csv

152+```

153+

154+And if we try reading the file with the `-read` thread:

155+

156+```

157+> -read [%x our %home da+now /foo/csv]

158+~[<|foo bar baz|> <|blah blah blah|> <|1 2 3|>]

159+```

160+

161+We can see our `%csv` `mark` has successfully converted our `foo.csv` file to a `(list (list @t))` when it was imported.

162+

163+Let's try the other direction now. We can create a new `bar.csv` files in the root of `%home` from the dojo like so:

164+

165+```

166+> */bar/csv ~[['abc' 'def' ~] ['ghi' 'jkl' ~]]

167++ /~zod/home/5/bar/csv

168+```

169+

170+And if we check it in the terminal on the Unix side we can see it's been correctly encoded:

171+

172+```

173+> cat zod/home/bar.csv

174+abc,def

175+ghi,jkl

176+```

177+

178+So now our `%csv` `mark` lets us move data in and out of Urbit. In the next section, we'll look at the `+grad` arm in more detail.

179+

180+## `+grad`

181+

182+So far we've just delegated `+grad` functions to the `%noun` `mark`, but now we'll look at writing our own.

183+

184+For demonstrative purposes, we can just poach the algorithms used in the `+grad` arm of the `%txt` `mark` and modify them to take our `(list (list @t))` type instead of a `wain`. It's not the most efficient algorithm for a CSV file but it'll do the job.

185+

186+Our diff format will be a `(urge:clay (list @t))`, and we'll use some `differ` functions from `zuse.hoon` like `+loss`, `+lusk` and `+lurk` to produce diffs and apply patches.

187+

188+The [csv.hoon library](/docs/arvo/clay/marks/examples#libcsvhoon) we imported also contains a `+csv-join` function which we'll use in the `+join` arm, just to save space here.

189+

190+Here's the new `%csv` `mark` defintion:

191+

192+```hoon

193+/+ *csv

194+|_ csv=(list (list @t))

195+++ grab

196+ |%

197+ ++ mime |=((pair mite octs) (de-csv q.q))

198+ ++ noun

199+ |= n=*

200+ ^- (list (list @t))

201+ =/ result ((list (list @t)) n)

202+ ?> (validate result)

203+ result

204+ --

205+++ grow

206+ |%

207+ ++ mime

208+ ?> (validate csv)

209+ [/text/csv (as-octs:mimes:html (en-csv csv))]

210+ ++ noun

211+ ?> (validate csv)

212+ csv

213+ --

214+++ grad

215+ |%

216+ ++ form %csv-diff

217+ ++ diff

218+ |= bob=(list (list @t))

219+ ^- (urge:clay (list @t))

220+ ?> (validate csv)

221+ ?> (validate bob)

222+ (lusk:differ csv bob (loss:differ csv bob))

223+ ++ pact

224+ |= dif=(urge:clay (list @t))

225+ ^- (list (list @t))

226+ =/ result (lurk:differ csv dif)

227+ ?> (validate result)

228+ result

229+ ++ join

230+ |= $: ali=(urge:clay (list @t))

231+ bob=(urge:clay (list @t))

232+ ==

233+ ^- (unit (urge:clay (list @t)))

234+ (csv-join ali bob)

235+ ++ mash

236+ |= $: [ship desk (urge:clay (list @t))]

237+ [ship desk (urge:clay (list @t))]

238+ ==

239+ ^- (urge:clay (list @t))

240+ ~|(%csv-mash !!)

241+ --

242+--

243+```

244+

245+In our modified `+grad` arm, we've replaced the `%noun` delegation with a core containing five arms: `+form`, `+diff`, `+pact`, `+join`, and `+mash`. These arms are all required for a valid `+grad` if it's not delegated to another `mark`. We'll now look at each in detail.

246+

247+### `+form`

248+

249+```hoon

250+++ form %csv-diff

251+```

252+

253+`+form` simply specifies the `mark` of the diff file that may be produced by other `+grad` functions. If your diff is the same type as your `mark`, it could just specify itself like `%csv`. In our case our diff is a `(urge:clay (list @t))` rather than a `(list (list @t))`, so we need a separate mark file for the diff itself.

254+

255+Here's another `mark` file which can be saved as `csv-diff.hoon` in `/mar`:

256+

257+```hoon

258+|_ dif=(urge:clay (list @t))

259+++ grab

260+ |%

261+ ++ noun (urge:clay (list @t))

262+ --

263+++ grow

264+ |%

265+ ++ noun dif

266+ --

267+++ grad %noun

268+--

269+```

270+

271+It's very bare-bones, we just need it for our `%csv` `mark` to work. In our `%csv` `mark`, we've specified it as `%csv-diff` in `+form`.

272+

273+### `+diff`

274+

275+```hoon

276+++ diff

277+ |= bob=(list (list @t))

278+ ^- (urge:clay (list @t))

279+ ?> (validate csv)

280+ ?> (validate bob)

281+ (lusk:differ csv bob (loss:differ csv bob))

282+```

283+

284+This arm produces the diff of two `%csv` files. The first `%csv` file will be given as the sample of the parent door, which if you'll recall we gave a face of `csv`. The second `%csv` file will be given as the sample of the gate in `+diff`, which we've named `bob` here. We then just produce the diff of these two files and return it as the type of the mark specified in `+form`, which in our case is `(urge:clay (list @t))` for a `%csv-diff`. Clay will use `+diff` when a file is revised, so it doesn't have to store a whole new copy of the file each time it's modified.

285+

286+### `+pact`

287+

288+```hoon

289+++ pact

290+ |= dif=(urge:clay (list @t))

291+ ^- (list (list @t))

292+ =/ result (lurk:differ csv dif)

293+ ?> (validate result)

294+ result

295+```

296+

297+`+pact` patches a `%csv` file with the given diff. Its gate takes a diff and applies it to the `%csv` given as the sample of the parent door (which we gave a face of `csv`). If the patch succeeds, it will return a new `%csv` file - a valid `(list (list @t))`. When we read a file that's been modified in Clay, Clay will apply all the diffs it has with `+pact` and return the resulting file.

298+

299+### `+join`

300+

301+```hoon

302+++ join

303+ |= $: ali=(urge:clay (list @t))

304+ bob=(urge:clay (list @t))

305+ ==

306+ ^- (unit (urge:clay (list @t)))

307+ (csv-join ali bob)

308+```

309+

310+The `+join` arm merges two different diffs. It takes them both as the sample of its gate (which we've named `ali` and `bob`), and returns a new diff wrapped in a `unit` like `(unit (urge:clay (list @t)))`. The `unit` will be `~` if the merge failed due to a conflict. This is used by Clay in some cases when `desk`s are merged. If diff merges are not possible for your use case, you could just have it always return `~`.

311+

312+### `+mash`

313+

314+```hoon

315+++ mash

316+ |= $: [ship desk (urge:clay (list @t))]

317+ [ship desk (urge:clay (list @t))]

318+ ==

319+ ^- (urge:clay (list @t))

320+ ~|(%csv-mash !!)

321+```

322+

323+This is like `+join` except it forces a diff merge even if there's a conflict. Rather than returning a `unit`, it just returns the diff - a `(urge:clay (list @t))` in our case. Also unlike `+join`, it takes the `ship` and `desk` each diff came from as well as the diff itself.

324+

325+The `+mash` arm is not used by Clay in its file revision operations, so it's safe to just make it a dummy arm that crashes as we've done here. If you were to use it, it would likely just be used manually in an agent, thread or generator.

326+

327+An example of its use would be the `%txt` `mark`, which includes a proper `+mash` function that produces a diff with any conflicts annotated, though how you have `+mash` handle conficts would depend on your use case. If there were no conflicts between the two diffs, it should produce the same diff as the `+join` arm.

328+

329+## Conclusion

330+

331+So there you have it, a fully functional `mark` for CSV files. A `mark` file can be as complex or as simple as you'd like, they're very flexible depending on your use case. Additional conversion methods can always be added as they're needed. For example, with just a few lines of code we could add arms for converting CSV files to `%json` or `%txt` and vice versa.

332+

333+In the next document, we'll look at building and using `mark` cores and `mark` conversion gates in our own code.