@@ -0,0 +1,505 @@
7+Most people have lots of data stored in online services, many of
8+which have APIs. API connectors allow the user to access this
9+data from within their Urbit.
11+A security driver allows the user to make authenticated requests
12+to the service's API, but the user still needs to know the API
13+and send and receive JSON. An API connector puts a layer of
14+porcelain over this to allow the user easier control over their
17+Each connector should perform two basic functions. First, it
18+should expose a tree of the data in the service that's accessible
19+through `.^` and FUSE. Second, it should expose event streams as
20+actions occur in the service. Let's take a look at how the `%gh`
21+app accomplishes both for Github.
23+After starting `%gh` (`|start %gh`), let's look at the root of
24+the tree that `%gh` exposes:
26+ ~sampel-sipnym:dojo> .^(arch %gy /=gh=)
27+ [fil=~ dir={[p=~.issues q=~]}]
29+`%gh` is currently a skeleton -- it contains examples of all the
30+functionality necessary for a connector, but many endpoints
31+aren't implemented. As we can see here, only issues are
32+implemented. We can explore this tree:
34+ ~sampel-sipnym:dojo> .^(arch %gy /=gh=/issues)
35+ [fil=~ dir={[p=~.by-repo q=~] [p=~.mine q=~]}]
40+ .^(issue:gh %gx /=gh=/issues/by-repo/philipcmonktest/testing/7)
42+ login.user:.^(issue:gh %gx /=gh=/issues/by-repo/philipcmonktest/testing/7)
45+The other thing an API connector needs to do is expose event
46+streams. One straightforward way to use this capacity is with
49+ :pipe|connect %gh /listen/philipcmonktest/testing/issues/'issue_comment' %public
51+Now creating an issue results in the following message in your
54+ ~sampel-sipnym[philipcmonktest@github]: opened issue #11: i found a bug!
56+You'll also get notifications of issue status changes and
59+As is often the case, the best way to write an API connector is
60+to copy an existing one and modify it. We'll dissect the Github
61+connector here. This code is in `/=home=/app/gh/hoon`, and it's
62+also reproduced at the bottom of this page (in case the code
63+drifts out of sync with this doc).
65+We'll go over the two parts to an API connector (one-time reading vs listening
66+for events) separately.
70+A connector exposes a tree of data. Every read request has a
71+`care`, which is generally either `%x` or `%y. `%x` is a request
72+for a particular piece of data, while `%y` is a request for a
73+directory listing. Roughly, `%x` means Unix `cat` and `%y` means
76+> Sometimes you wish to expose a tree where a part of the path
77+> can't be enumerated. For example, a `%y` of `/issues/by-repo`
78+> "should" produce a list of all Github users, but we don't do
79+> that because it's too long. Instead, we just produce our own
80+> username (from the `web.plan` file). You can still access
81+> repos from other uesrs, you just don't see them in the
84+The usual flow for implementing this tree of data makes heavy use
85+of the `connector` library. This library is well documented in
86+the source, so check out `/=home=/lib/connector/hoon`.
88+Most of the Github-specific logic is in `++places`, which is a
89+list of all the places we can request. A place consists of:
95+ sigh-x/$-(jon/json (unit sub-result))
96+ sigh-y/$-(jon/json (unit arch))
99+- `guard`, the type of the paths we should match. For example,
100+ to match `/issues/<user>/<repo>` use `{$issues @t @t $~}`.
102+- `read-x`, called when someone tries to read the place with
103+ care `%x`. Should produce a single move, usually either a
104+ `%diff` response if we can immediately answer or a `%hiss`
105+ http request if we need to make a request to the api. See the
106+ `++read-*` functions in `++helpers` for some common handlers.
108+- `read-y`, same as `read-x` except with care `%y`.
110+- `sigh-x`, called when an http response comes back on this
111+ place. You're given the json of the result, and you should
112+ produce either a result or null. Null represents an error.
113+ If you didn't create an http request in `read-x`, then this
114+ should never be called. Use `++sigh-strange` from `++helpers`
115+ to unconditionally signal an error.
117+- `sigh-y`, same as `sigh-x` except with care `%y`. Note that a
118+ `%y` request must produce an arch, unlike a `%x` request,
119+ which may produce data of any mark.
121+The `connector` library needs to be initialized with definitions
122+of `move` and `sub-result` (all the types of data that can be
123+returned by a place). `read-x` and `read-y` produce moves and
124+`sigh-x` produces a sub-result.
126+Filling out the list of places is a lot of grunt work, but most
127+places are fairly straightforward, and the `++read-*` helper
128+functions are useful. Check out the library source for more
129+information on those.
131+Besides the list of places, we just need to handle the flow of
132+control. There are four functions we need to define:
134+- `++peek` should usually just produce `~`. If there are cases
135+ where we can respond directly to a `.^` request without
136+ blocking on anything, we could do it here, but it's generally
137+ not worth the hassle.
139+- `++peer-scry` is where the actual handling for a read request
140+ goes. In general, we just need to call `++read` from the
141+ `connector` library with the bone, list of places, care, and
142+ path. This will match the path to the appropriate place and
143+ run either `++read-x` or `++read-y`, depending on the care.
145+- If `++read-x` or `++read-y` made a successful API request, then
146+ it'll come back on `++sigh-httr`. Here, we just need to parse
147+ out the wire and call `++sigh` from the `connector` library
148+ with the list of places, care, path, and HTTP result. This
149+ will match the path to the appropriate place and run either
150+ `++sigh-x` or `++sigh-y`, depending on the care.
152+- If `++read-x` or `++read-y` made an API request that failed,
153+ then we'll get a stack trace in `++sigh-tang`. Here, we just
154+ print it out and move on.
156+That's really all there is to the reading portion of API
161+Listening for events is fairly service-specific. In some
162+services, we poll for changes. For an example of that, check out
163+the Twitter connector. In Github, we power our event streams
166+For Github, when someone subscribes to
167+`/listen/<user>/<repo>/<events...>`, we want to produce
168+well-typed results when they occur.
170+We only want to create one webhook per event, so in our state we
171+have `hook`, which is a map of event names to the set of bones
172+which are subscribed to that event. We must always keep this
175+Flow of control starts in `++peer-listen`, where we just call
176+`++listen`. In `++listen`, for each event in the list of events,
177+we check to see if we have that hook set up (by checking whether
178+it exists in `hook`). If so, we call `++update-hook` to add th
179+current bone to the set of listeners. Otherwise, we call
180+`++create-hook`, which sends a request to Github to set up the
181+new webhook. We also create an entry in `hook` with the current
184+When we created the webhook, we told Github to send the event to
185+`/~/to/gh/gh-<event>.json?anon&wire=/`. This turns into a poke.
186+Let's parse out this url. The first `gh` is the app name to
187+poke. The next portion, `gh-<event>` is the mark to convert the
188+JSON to. For example, when the "issues" event fires, it'll get
189+converted to mark `gh-issues`. This mark has a conversion
190+routine from `json` to the `issues` type, which is defined in the
191+`gh` structure (`/=home=/sur/gh/hoon`). Once this conversion
192+happens, the `gh` app will get poked with the correctly marked
195+In `++poke`, we check `hook` to find the subscribers to the event
196+that was fired, and we update them.
198+That's really all there is to it. Webhook flow isn't well
199+standardized, so even if your chosen service is powered by
200+webhooks, your listening code might look rather different. The
201+point is to expose the correct interface.
204+:: This is a connector for the Github API v3.
206+:: You can interact with this in a few different ways:
208+:: - .^({type} %gx /=gh={/endpoint}) to read data or
209+:: .^(arch %gy /=gh={/endpoint}) to explore the possible
212+:: - subscribe to /listen/{owner}/{repo}/{events...} for
213+:: webhook-powered event notifications. For event list, see
214+:: https://developer.github.com/webhooks/.
216+:: This is written with the standard structure for api
217+:: connectors, as described in lib/connector.hoon.
221+/+ gh-parse, connector
225+ ++ move (pair bone card)
227+ $% {$diff sub-result}
228+ {$them wire (unit hiss)}
229+ {$hiss wire {$~ $~} $httr {$hiss hiss}}
232+ :: Types of results we produce to subscribers.
236+ {$gh-issue issue:gh}
237+ {$gh-list-issues (list issue:gh)}
238+ {$gh-issues issues:gh}
239+ {$gh-issue-comment issue-comment:gh}
244+ :: Types of webhooks we expect.
247+ $% {$gh-issues issues:gh}
248+ {$gh-issue-comment issue-comment:gh}
251+=+ connector=(connector move sub-result) :: Set up connector library
254+ hook/(map @t {id/@t listeners/(set bone)}) :: map events to listeners
256+:: ++ prep _`. :: Clear state when code changes
262+ ^- (list place:connector)
263+ =+ (helpers:connector ost.hid wir "https://api.github.com")
264+ => |% :: gh-specific helpers
265+ ++ sigh-list-issues-x
267+ %+ bind ((ar:jo issue:gh-parse) jon)
268+ |= issues/(list issue:gh)
269+ gh-list-issues+issues
271+ ++ sigh-list-issues-y
273+ %+ bind ((ar:jo issue:gh-parse) jon)
274+ |= issues/(list issue:gh)
275+ :- `(shax (jam issues))
276+ (malt (turn issues |=(issue:gh [(rsh 3 2 (scot %ui number)) ~])))
281+ read-y=(read-static %issues ~)
286+ :* guard={$issues $~}
288+ read-y=(read-static %mine %by-repo ~)
292+ ^- place :: /issues/mine
293+ :* guard={$issues $mine $~}
294+ read-x=(read-get /issues)
295+ read-y=(read-get /issues)
296+ sigh-x=sigh-list-issues-x
297+ sigh-y=sigh-list-issues-y
299+ ^- place :: /issues/by-repo
300+ :* guard={$issues $by-repo $~}
304+ =+ /(scot %p our.hid)/home/(scot %da now.hid)/web/plan
305+ =+ .^({* acc/(map knot plan-acct)} %cx -)
307+ ((read-static usr:(~(got by acc) %github) ~) pax)
311+ ^- place :: /issues/by-repo/<user>
312+ :* guard={$issues $by-repo @t $~}
314+ read-y=|=(pax/path (get /users/[-.+>.pax]/repos))
318+ %+ bind ((ar:jo repository:gh-parse) jon)
319+ |= repos/(list repository:gh)
320+ [~ (malt (turn repos |=(repository:gh [name ~])))]
322+ ^- place :: /issues/by-repo/<user>/<repo>
323+ :* guard={$issues $by-repo @t @t $~}
324+ read-x=|=(pax/path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues))
325+ read-y=|=(pax/path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues))
326+ sigh-x=sigh-list-issues-x
327+ sigh-y=sigh-list-issues-y
329+ ^- place :: /issues/by-repo/<user>/<repo>
330+ :* guard={$issues $by-repo @t @t @t $~}
332+ |=(pax/path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues/[-.+>+>.pax]))
334+ read-y=(read-static ~)
337+ %+ bind (issue:gh-parse jon)
345+:: When a peek on a path blocks, ford turns it into a peer on
346+:: /scry/{care}/{path}. You can also just peer to this
349+:: We hand control to ++scry.
353+ ^- {(list move) _+>.$}
356+ (read:connector ost.hid (places %read pax) i.pax t.pax)
358+:: HTTP response. We make sure the response is good, then
359+:: produce the result (as JSON) to whoever sent the request.
362+ |= {way/wire res/httr}
363+ ^- {(list move) _+>.$}
364+ ?. ?=({$read care @ *} way)
372+ (sigh:connector (places ren style pax) ren pax res)
374+:: HTTP error. We just print it out, though maybe we should
375+:: also produce a result so that the request doesn't hang?
378+ |= {way/wire tan/tang}
379+ ^- {(list move) _+>.$}
380+ %- (slog >%gh-sigh-tang< tan)
381+ [[ost.hid %diff null+~]~ +>.$]
383+:: We can't actually give the response to pretty much anything
384+:: without blocking, so we just block unconditionally.
387+ |= {ren/@tas tyl/path}
388+ ^- (unit (unit (pair mark *)))
389+ ~ ::``noun/[ren tyl]
391+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
392+:: Webhook-powered event streams (/listen) ::
393+::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
395+:: To listen to a webhook-powered stream of events, subscribe
396+:: to /listen/<user>/<repo>/<events...>
398+:: We hand control to ++listen.
402+ ^- {(list move) _+>.$}
404+ ~& [%bad-listen-path pax]
408+:: This core handles event subscription requests by starting or
409+:: updating the webhook flow for each event.
416+ ++ abet :: Resolve core.
417+ ^- {(list move) _+>.$}
420+ ++ send-hiss :: Send a hiss
423+ =+ wir=`wire`[%x %listen pax]
424+ +>.$(mow [[ost.hid %hiss wir `~ %httr [%hiss hiz]] mow])
426+ :: Create or update a webhook to listen for a set of events.
430+ =+ pax=pax :: TMI-proofing
436+ ?: (~(has by hook) i.events)
437+ $(+>+ (update-hook i.events), events t.events)
438+ $(+>+ (create-hook i.events), events t.events)
440+ :: Set up a webhook.
446+ =+ clean-event=`tape`(turn (trip event) |=(a/@tD ?:(=('_' a) '-' a)))
448+ %+ ~(put by hook) (crip clean-event)
450+ (~(get by hook) (crip clean-event))
451+ *{id/@t listeners/(set bone)}
452+ [id (~(put in listeners) ost.hid)]
455+ =+ [(trip i.pax) (trip i.t.pax)]
456+ "https://api.github.com/repos/{-<}/{->}/hooks"
459+ %- taco %- crip %- pojo %- jobe :~
462+ events+a+~[s+event] ::(turn `(list ,@t)`t.t.pax |=(a=@t s/a))
466+ "http://107.170.195.5:8443/~/to/gh/gh-{-}.json?anon&wire=/"
468+ [%'content_type' s+%json]
473+ :: Add current bone to the list of subscribers for this event.
478+ =+ hok=(~(got by hook) event)
481+ %+ ~(put by hook) event
482+ hok(listeners (~(put in listeners.hok) ost.hid))
486+:: Pokes that aren't caught in more specific arms are handled
487+:: here. These should be only from webhooks firing, so if we
488+:: get any mark that we shouldn't get from a webhook, we reject
489+:: it. Otherwise, we spam out the event to everyone who's
490+:: listening for that event.
493+ |= response/hook-response
494+ ^- {(list move) _+>.$}
495+ =+ hook-data=(~(get by hook) (rsh 3 3 -.response))
497+ ~& [%strange-hook hook response]
499+ :: ~& response=response
501+ %+ turn (~(tap in listeners.u.hook-data))
503+ [ost %diff response]