@@ -0,0 +1,505 @@
1+---
2+next: true
3+sort: 12
4+title: API Connectors
5+---
6+
7+Most people have lots of data stored in online services, many of
8+which have APIs. API connectors allow the user to access this
9+data from within their Urbit.
10+
11+A security driver allows the user to make authenticated requests
12+to the service's API, but the user still needs to know the API
13+and send and receive JSON. An API connector puts a layer of
14+porcelain over this to allow the user easier control over their
15+data.
16+
17+Each connector should perform two basic functions. First, it
18+should expose a tree of the data in the service that's accessible
19+through `.^` and FUSE. Second, it should expose event streams as
20+actions occur in the service. Let's take a look at how the `%gh`
21+app accomplishes both for Github.
22+
23+After starting `%gh` (`|start %gh`), let's look at the root of
24+the tree that `%gh` exposes:
25+
26+ ~sampel-sipnym:dojo> .^(arch %gy /=gh=)
27+ [fil=~ dir={[p=~.issues q=~]}]
28+
29+`%gh` is currently a skeleton -- it contains examples of all the
30+functionality necessary for a connector, but many endpoints
31+aren't implemented. As we can see here, only issues are
32+implemented. We can explore this tree:
33+
34+ ~sampel-sipnym:dojo> .^(arch %gy /=gh=/issues)
35+ [fil=~ dir={[p=~.by-repo q=~] [p=~.mine q=~]}]
36+
37+And eventually:
38+
39+ /- gh
40+ .^(issue:gh %gx /=gh=/issues/by-repo/philipcmonktest/testing/7)
41+ <issue...>
42+ login.user:.^(issue:gh %gx /=gh=/issues/by-repo/philipcmonktest/testing/7)
43+ 'philipcmonktest'
44+
45+The other thing an API connector needs to do is expose event
46+streams. One straightforward way to use this capacity is with
47+the `:pipe` app:
48+
49+ :pipe|connect %gh /listen/philipcmonktest/testing/issues/'issue_comment' %public
50+
51+Now creating an issue results in the following message in your
52+`%public` channel:
53+
54+ ~sampel-sipnym[philipcmonktest@github]: opened issue #11: i found a bug!
55+
56+You'll also get notifications of issue status changes and
57+comments.
58+
59+As is often the case, the best way to write an API connector is
60+to copy an existing one and modify it. We'll dissect the Github
61+connector here. This code is in `/=home=/app/gh/hoon`, and it's
62+also reproduced at the bottom of this page (in case the code
63+drifts out of sync with this doc).
64+
65+We'll go over the two parts to an API connector (one-time reading vs listening
66+for events) separately.
67+
68+# Reading
69+
70+A connector exposes a tree of data. Every read request has a
71+`care`, which is generally either `%x` or `%y. `%x` is a request
72+for a particular piece of data, while `%y` is a request for a
73+directory listing. Roughly, `%x` means Unix `cat` and `%y` means
74+Unix `ls`.
75+
76+> Sometimes you wish to expose a tree where a part of the path
77+> can't be enumerated. For example, a `%y` of `/issues/by-repo`
78+> "should" produce a list of all Github users, but we don't do
79+> that because it's too long. Instead, we just produce our own
80+> username (from the `web.plan` file). You can still access
81+> repos from other uesrs, you just don't see them in the
82+> directory listing.
83+
84+The usual flow for implementing this tree of data makes heavy use
85+of the `connector` library. This library is well documented in
86+the source, so check out `/=home=/lib/connector/hoon`.
87+
88+Most of the Github-specific logic is in `++places`, which is a
89+list of all the places we can request. A place consists of:
90+
91+ ++ place
92+ $: guard/mold
93+ read-x/$-(path move)
94+ read-y/$-(path move)
95+ sigh-x/$-(jon/json (unit sub-result))
96+ sigh-y/$-(jon/json (unit arch))
97+ ==
98+
99+- `guard`, the type of the paths we should match. For example,
100+ to match `/issues/<user>/<repo>` use `{$issues @t @t $~}`.
101+
102+- `read-x`, called when someone tries to read the place with
103+ care `%x`. Should produce a single move, usually either a
104+ `%diff` response if we can immediately answer or a `%hiss`
105+ http request if we need to make a request to the api. See the
106+ `++read-*` functions in `++helpers` for some common handlers.
107+
108+- `read-y`, same as `read-x` except with care `%y`.
109+
110+- `sigh-x`, called when an http response comes back on this
111+ place. You're given the json of the result, and you should
112+ produce either a result or null. Null represents an error.
113+ If you didn't create an http request in `read-x`, then this
114+ should never be called. Use `++sigh-strange` from `++helpers`
115+ to unconditionally signal an error.
116+
117+- `sigh-y`, same as `sigh-x` except with care `%y`. Note that a
118+ `%y` request must produce an arch, unlike a `%x` request,
119+ which may produce data of any mark.
120+
121+The `connector` library needs to be initialized with definitions
122+of `move` and `sub-result` (all the types of data that can be
123+returned by a place). `read-x` and `read-y` produce moves and
124+`sigh-x` produces a sub-result.
125+
126+Filling out the list of places is a lot of grunt work, but most
127+places are fairly straightforward, and the `++read-*` helper
128+functions are useful. Check out the library source for more
129+information on those.
130+
131+Besides the list of places, we just need to handle the flow of
132+control. There are four functions we need to define:
133+
134+- `++peek` should usually just produce `~`. If there are cases
135+ where we can respond directly to a `.^` request without
136+ blocking on anything, we could do it here, but it's generally
137+ not worth the hassle.
138+
139+- `++peer-scry` is where the actual handling for a read request
140+ goes. In general, we just need to call `++read` from the
141+ `connector` library with the bone, list of places, care, and
142+ path. This will match the path to the appropriate place and
143+ run either `++read-x` or `++read-y`, depending on the care.
144+
145+- If `++read-x` or `++read-y` made a successful API request, then
146+ it'll come back on `++sigh-httr`. Here, we just need to parse
147+ out the wire and call `++sigh` from the `connector` library
148+ with the list of places, care, path, and HTTP result. This
149+ will match the path to the appropriate place and run either
150+ `++sigh-x` or `++sigh-y`, depending on the care.
151+
152+- If `++read-x` or `++read-y` made an API request that failed,
153+ then we'll get a stack trace in `++sigh-tang`. Here, we just
154+ print it out and move on.
155+
156+That's really all there is to the reading portion of API
157+connectors.
158+
159+# Listening
160+
161+Listening for events is fairly service-specific. In some
162+services, we poll for changes. For an example of that, check out
163+the Twitter connector. In Github, we power our event streams
164+with webhooks.
165+
166+For Github, when someone subscribes to
167+`/listen/<user>/<repo>/<events...>`, we want to produce
168+well-typed results when they occur.
169+
170+We only want to create one webhook per event, so in our state we
171+have `hook`, which is a map of event names to the set of bones
172+which are subscribed to that event. We must always keep this
173+up-to-date.
174+
175+Flow of control starts in `++peer-listen`, where we just call
176+`++listen`. In `++listen`, for each event in the list of events,
177+we check to see if we have that hook set up (by checking whether
178+it exists in `hook`). If so, we call `++update-hook` to add th
179+current bone to the set of listeners. Otherwise, we call
180+`++create-hook`, which sends a request to Github to set up the
181+new webhook. We also create an entry in `hook` with the current
182+bone.
183+
184+When we created the webhook, we told Github to send the event to
185+`/~/to/gh/gh-<event>.json?anon&wire=/`. This turns into a poke.
186+Let's parse out this url. The first `gh` is the app name to
187+poke. The next portion, `gh-<event>` is the mark to convert the
188+JSON to. For example, when the "issues" event fires, it'll get
189+converted to mark `gh-issues`. This mark has a conversion
190+routine from `json` to the `issues` type, which is defined in the
191+`gh` structure (`/=home=/sur/gh/hoon`). Once this conversion
192+happens, the `gh` app will get poked with the correctly marked
193+data in `++poke`.
194+
195+In `++poke`, we check `hook` to find the subscribers to the event
196+that was fired, and we update them.
197+
198+That's really all there is to it. Webhook flow isn't well
199+standardized, so even if your chosen service is powered by
200+webhooks, your listening code might look rather different. The
201+point is to expose the correct interface.
202+
203+```
204+:: This is a connector for the Github API v3.
205+::
206+:: You can interact with this in a few different ways:
207+::
208+:: - .^({type} %gx /=gh={/endpoint}) to read data or
209+:: .^(arch %gy /=gh={/endpoint}) to explore the possible
210+:: endpoints.
211+::
212+:: - subscribe to /listen/{owner}/{repo}/{events...} for
213+:: webhook-powered event notifications. For event list, see
214+:: https://developer.github.com/webhooks/.
215+::
216+:: This is written with the standard structure for api
217+:: connectors, as described in lib/connector.hoon.
218+::
219+/? 314
220+/- gh, plan-acct
221+/+ gh-parse, connector
222+::
223+!:
224+=> |%
225+ ++ move (pair bone card)
226+ ++ card
227+ $% {$diff sub-result}
228+ {$them wire (unit hiss)}
229+ {$hiss wire {$~ $~} $httr {$hiss hiss}}
230+ ==
231+ ::
232+ :: Types of results we produce to subscribers.
233+ ::
234+ ++ sub-result
235+ $% {$arch arch}
236+ {$gh-issue issue:gh}
237+ {$gh-list-issues (list issue:gh)}
238+ {$gh-issues issues:gh}
239+ {$gh-issue-comment issue-comment:gh}
240+ {$json json}
241+ {$null $~}
242+ ==
243+ ::
244+ :: Types of webhooks we expect.
245+ ::
246+ ++ hook-response
247+ $% {$gh-issues issues:gh}
248+ {$gh-issue-comment issue-comment:gh}
249+ ==
250+ --
251+=+ connector=(connector move sub-result) :: Set up connector library
252+::
253+|_ $: hid/bowl
254+ hook/(map @t {id/@t listeners/(set bone)}) :: map events to listeners
255+ ==
256+:: ++ prep _`. :: Clear state when code changes
257+::
258+:: List of endpoints
259+::
260+++ places
261+ |= wir/wire
262+ ^- (list place:connector)
263+ =+ (helpers:connector ost.hid wir "https://api.github.com")
264+ => |% :: gh-specific helpers
265+ ++ sigh-list-issues-x
266+ |= jon/json
267+ %+ bind ((ar:jo issue:gh-parse) jon)
268+ |= issues/(list issue:gh)
269+ gh-list-issues+issues
270+ ::
271+ ++ sigh-list-issues-y
272+ |= jon/json
273+ %+ bind ((ar:jo issue:gh-parse) jon)
274+ |= issues/(list issue:gh)
275+ :- `(shax (jam issues))
276+ (malt (turn issues |=(issue:gh [(rsh 3 2 (scot %ui number)) ~])))
277+ --
278+ :~ ^- place :: /
279+ :* guard=$~
280+ read-x=read-null
281+ read-y=(read-static %issues ~)
282+ sigh-x=sigh-strange
283+ sigh-y=sigh-strange
284+ ==
285+ ^- place :: /issues
286+ :* guard={$issues $~}
287+ read-x=read-null
288+ read-y=(read-static %mine %by-repo ~)
289+ sigh-x=sigh-strange
290+ sigh-y=sigh-strange
291+ ==
292+ ^- place :: /issues/mine
293+ :* guard={$issues $mine $~}
294+ read-x=(read-get /issues)
295+ read-y=(read-get /issues)
296+ sigh-x=sigh-list-issues-x
297+ sigh-y=sigh-list-issues-y
298+ ==
299+ ^- place :: /issues/by-repo
300+ :* guard={$issues $by-repo $~}
301+ read-x=read-null
302+ ^= read-y
303+ |= pax/path
304+ =+ /(scot %p our.hid)/home/(scot %da now.hid)/web/plan
305+ =+ .^({* acc/(map knot plan-acct)} %cx -)
306+ ::
307+ ((read-static usr:(~(got by acc) %github) ~) pax)
308+ sigh-x=sigh-strange
309+ sigh-y=sigh-strange
310+ ==
311+ ^- place :: /issues/by-repo/<user>
312+ :* guard={$issues $by-repo @t $~}
313+ read-x=read-null
314+ read-y=|=(pax/path (get /users/[-.+>.pax]/repos))
315+ sigh-x=sigh-strange
316+ ^= sigh-y
317+ |= jon/json
318+ %+ bind ((ar:jo repository:gh-parse) jon)
319+ |= repos/(list repository:gh)
320+ [~ (malt (turn repos |=(repository:gh [name ~])))]
321+ ==
322+ ^- place :: /issues/by-repo/<user>/<repo>
323+ :* guard={$issues $by-repo @t @t $~}
324+ read-x=|=(pax/path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues))
325+ read-y=|=(pax/path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues))
326+ sigh-x=sigh-list-issues-x
327+ sigh-y=sigh-list-issues-y
328+ ==
329+ ^- place :: /issues/by-repo/<user>/<repo>
330+ :* guard={$issues $by-repo @t @t @t $~}
331+ ^= read-x
332+ |=(pax/path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues/[-.+>+>.pax]))
333+ ::
334+ read-y=(read-static ~)
335+ ^= sigh-x
336+ |= jon/json
337+ %+ bind (issue:gh-parse jon)
338+ |= issue/issue:gh
339+ gh-issue+issue
340+ ::
341+ sigh-y=sigh-strange
342+ ==
343+ ==
344+::
345+:: When a peek on a path blocks, ford turns it into a peer on
346+:: /scry/{care}/{path}. You can also just peer to this
347+:: directly.
348+::
349+:: We hand control to ++scry.
350+::
351+++ peer-scry
352+ |= pax/path
353+ ^- {(list move) _+>.$}
354+ ?> ?=({care *} pax)
355+ :_ +>.$ :_ ~
356+ (read:connector ost.hid (places %read pax) i.pax t.pax)
357+::
358+:: HTTP response. We make sure the response is good, then
359+:: produce the result (as JSON) to whoever sent the request.
360+::
361+++ sigh-httr
362+ |= {way/wire res/httr}
363+ ^- {(list move) _+>.$}
364+ ?. ?=({$read care @ *} way)
365+ ~& res=res
366+ [~ +>.$]
367+ =* style i.way
368+ =* ren i.t.way
369+ =* pax t.t.way
370+ :_ +>.$ :_ ~
371+ :+ ost.hid %diff
372+ (sigh:connector (places ren style pax) ren pax res)
373+::
374+:: HTTP error. We just print it out, though maybe we should
375+:: also produce a result so that the request doesn't hang?
376+::
377+++ sigh-tang
378+ |= {way/wire tan/tang}
379+ ^- {(list move) _+>.$}
380+ %- (slog >%gh-sigh-tang< tan)
381+ [[ost.hid %diff null+~]~ +>.$]
382+::
383+:: We can't actually give the response to pretty much anything
384+:: without blocking, so we just block unconditionally.
385+::
386+++ peek
387+ |= {ren/@tas tyl/path}
388+ ^- (unit (unit (pair mark *)))
389+ ~ ::``noun/[ren tyl]
390+::
391+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
392+:: Webhook-powered event streams (/listen) ::
393+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
394+::
395+:: To listen to a webhook-powered stream of events, subscribe
396+:: to /listen/<user>/<repo>/<events...>
397+::
398+:: We hand control to ++listen.
399+::
400+++ peer-listen
401+ |= pax/path
402+ ^- {(list move) _+>.$}
403+ ?. ?=({@ @ *} pax)
404+ ~& [%bad-listen-path pax]
405+ [~ +>.$]
406+ (listen pax)
407+::
408+:: This core handles event subscription requests by starting or
409+:: updating the webhook flow for each event.
410+::
411+++ listen
412+ |= pax/path
413+ =| mow/(list move)
414+ =< abet:listen
415+ |%
416+ ++ abet :: Resolve core.
417+ ^- {(list move) _+>.$}
418+ [(flop mow) +>.$]
419+ ::
420+ ++ send-hiss :: Send a hiss
421+ |= hiz/hiss
422+ ^+ +>
423+ =+ wir=`wire`[%x %listen pax]
424+ +>.$(mow [[ost.hid %hiss wir `~ %httr [%hiss hiz]] mow])
425+ ::
426+ :: Create or update a webhook to listen for a set of events.
427+ ::
428+ ++ listen
429+ ^+ .
430+ =+ pax=pax :: TMI-proofing
431+ ?> ?=({@ @ *} pax)
432+ =+ events=t.t.pax
433+ |- ^+ +>+.$
434+ ?~ events
435+ +>+.$
436+ ?: (~(has by hook) i.events)
437+ $(+>+ (update-hook i.events), events t.events)
438+ $(+>+ (create-hook i.events), events t.events)
439+ ::
440+ :: Set up a webhook.
441+ ::
442+ ++ create-hook
443+ |= event/@t
444+ ^+ +>
445+ ?> ?=({@ @ *} pax)
446+ =+ clean-event=`tape`(turn (trip event) |=(a/@tD ?:(=('_' a) '-' a)))
447+ =. hook
448+ %+ ~(put by hook) (crip clean-event)
449+ =+ %+ fall
450+ (~(get by hook) (crip clean-event))
451+ *{id/@t listeners/(set bone)}
452+ [id (~(put in listeners) ost.hid)]
453+ %- send-hiss
454+ :* %+ scan
455+ =+ [(trip i.pax) (trip i.t.pax)]
456+ "https://api.github.com/repos/{-<}/{->}/hooks"
457+ auri:epur
458+ %post ~ ~
459+ %- taco %- crip %- pojo %- jobe :~
460+ name+s+%web
461+ active+b+&
462+ events+a+~[s+event] ::(turn `(list ,@t)`t.t.pax |=(a=@t s/a))
463+ :- %config
464+ %- jobe :~
465+ =+ =+ clean-event
466+ "http://107.170.195.5:8443/~/to/gh/gh-{-}.json?anon&wire=/"
467+ [%url s+(crip -)]
468+ [%'content_type' s+%json]
469+ ==
470+ ==
471+ ==
472+ ::
473+ :: Add current bone to the list of subscribers for this event.
474+ ::
475+ ++ update-hook
476+ |= event/@t
477+ ^+ +>
478+ =+ hok=(~(got by hook) event)
479+ %_ +>.$
480+ hook
481+ %+ ~(put by hook) event
482+ hok(listeners (~(put in listeners.hok) ost.hid))
483+ ==
484+ --
485+::
486+:: Pokes that aren't caught in more specific arms are handled
487+:: here. These should be only from webhooks firing, so if we
488+:: get any mark that we shouldn't get from a webhook, we reject
489+:: it. Otherwise, we spam out the event to everyone who's
490+:: listening for that event.
491+::
492+++ poke
493+ |= response/hook-response
494+ ^- {(list move) _+>.$}
495+ =+ hook-data=(~(get by hook) (rsh 3 3 -.response))
496+ ?~ hook-data
497+ ~& [%strange-hook hook response]
498+ [~ +>.$]
499+ :: ~& response=response
500+ :_ +>.$
501+ %+ turn (~(tap in listeners.u.hook-data))
502+ |= ost/bone
503+ [ost %diff response]
504+--
505+```