9Most people have lots of data stored in online services, many of
10which have APIs. API connectors allow the user to access this
11data from within their Urbit.
13A security driver allows the user to make authenticated requests
14to the service's API, but the user still needs to know the API
15and send and receive JSON. An API connector puts a layer of
16porcelain over this to allow the user easier control over their
19Each connector should perform two basic functions. First, it
20should expose a tree of the data in the service that's accessible
21through `.^` and FUSE. Second, it should expose event streams as
22actions occur in the service. Let's take a look at how the `%gh`
23app accomplishes both for Github.
25After starting `%gh` (`|start %gh`), let's look at the root of
26the tree that `%gh` exposes:
28 ~your-urbit:dojo> .^(arch %gy /=gh=)
29 [fil=~ dir={[p=~.issues q=~]}]
31`%gh` is currently a skeleton -- it contains examples of all the
32functionality necessary for a connector, but many endpoints
33aren't implemented. As we can see here, only issues are
34implemented. We can explore this tree:
36 ~your-urbit:dojo> .^(arch %gy /=gh=/issues)
37 [fil=~ dir={[p=~.by-repo q=~] [p=~.mine q=~]}]
42 .^(issue:gh %gx /=gh=/issues/by-repo/philipcmonktest/testing/7)
44 login.user:.^(issue:gh %gx /=gh=/issues/by-repo/philipcmonktest/testing/7)
47The other thing an API connector needs to do is expose event
48streams. One straightforward way to use this capacity is with
51 :pipe|connect %gh /listen/philipcmonktest/testing/issues/'issue_comment' %public
53Now creating an issue results in the following message in your
54`%public` `:talk` channel:
56 ~your-urbit[philipcmonktest@github]: opened issue #11: i found a bug!
58You'll also get notifications of issue status changes and
61As is often the case, the best way to write an API connector is
62to copy an existing one and modify it. We'll dissect the Github
63connector here. This code is in `/=home=/app/gh/hoon`, and it's
64also reproduced at the bottom of this page (in case the code
65drifts out of sync with this doc).
67We'll go over the two parts to an API connector (one-time reading vs listening
68for events) separately.
72A connector exposes a tree of data. Every read request has a
73`care`, which is generally either `%x` or `%y. `%x` is a request
74for a particular piece of data, while `%y` is a request for a
75directory listing. Roughly, `%x` means Unix `cat` and `%y` means
78> Sometimes you wish to expose a tree where a part of the path
79> can't be enumerated. For example, a `%y` of `/issues/by-repo`
80> "should" produce a list of all Github users, but we don't do
81> that because it's too long. Instead, we just produce our own
82> username (from the `web.plan` file). You can still access
83> repos from other users, you just don't see them in the
86The usual flow for implementing this tree of data makes heavy use
87of the `connector` library. This library is well documented in
88the source, so check out `/=home=/lib/connector/hoon`.
90Most of the Github-specific logic is in `++places`, which is a
91list of all the places we can request. A place consists of:
97 sigh-x/$-(jon/json (unit sub-result))
98 sigh-y/$-(jon/json (unit arch))
101- `guard`, the type of the paths we should match. For example,
102 to match `/issues/<user>/<repo>` use `{$issues @t @t $~}`.
104- `read-x`, called when someone tries to read the place with
105 `care` `%x`. Should produce a single move, usually either a
106 `%diff` response if we can immediately answer or a `%hiss`
107 http request if we need to make a request to the api. See the
108 `++read-*` functions in `++helpers` for some common handlers.
110- `read-y`, same as `read-x` except with care `%y`.
112- `sigh-x`, called when an http response comes back on this
113 place. You're given the json of the result, and you should
114 produce either a result or null. Null represents an error.
115 If you didn't create an http request in `read-x`, then this
116 should never be called. Use `++sigh-strange` from `++helpers`
117 to unconditionally signal an error.
119- `sigh-y`, same as `sigh-x` except with care `%y`. Note that a
120 `%y` request must produce an arch, unlike a `%x` request,
121 which may produce data of any mark.
123The `connector` library needs to be initialized with definitions
124of `move` and `sub-result` (all the types of data that can be
125returned by a place). `read-x` and `read-y` produce moves and
126`sigh-x` produces a sub-result.
128Filling out the list of places is a lot of grunt work, but most
129places are fairly straightforward, and the `++read-*` helper
130functions are useful. Check out the library source for more
133Besides the list of places, we just need to handle the flow of
134control. There are four functions we need to define:
136- `++peek` should usually just produce `~`. If there are cases
137 where we can respond directly to a `.^` request without
138 blocking on anything, we could do it here, but it's generally
139 not worth the hassle.
141- `++peer-scry` is where the actual handling for a read request
142 goes. In general, we just need to call `++read` from the
143 `connector` library with the bone, list of places, care, and
144 path. This will match the path to the appropriate place and
145 run either `++read-x` or `++read-y`, depending on the care.
147- If `++read-x` or `++read-y` made a successful API request, then
148 it'll come back on `++sigh-httr`. Here, we just need to parse
149 out the wire and call `++sigh` from the `connector` library
150 with the list of places, care, path, and HTTP result. This
151 will match the path to the appropriate place and run either
152 `++sigh-x` or `++sigh-y`, depending on the care.
154- If `++read-x` or `++read-y` made an API request that failed,
155 then we'll get a stack trace in `++sigh-tang`. Here, we just
156 print it out and move on.
158That's really all there is to the reading portion of API
163Listening for events is fairly service-specific. In some
164services, we poll for changes. For an example of that, check out
165the Twitter connector. In Github, we power our event streams
168For Github, when someone subscribes to
169`/listen/<user>/<repo>/<events...>`, we want to produce
170well-typed results when they occur.
172We only want to create one webhook per event, so in our state we
173have `hook`, which is a map of event names to the set of bones
174which are subscribed to that event. We must always keep this
177Flow of control starts in `++peer-listen`, where we just call
178`++listen`. In `++listen`, for each event in the list of events,
179we check to see if we have that hook set up (by checking whether
180it exists in `hook`). If so, we call `++update-hook` to add the
181current bone to the set of listeners. Otherwise, we call
182`++create-hook`, which sends a request to Github to set up the
183new webhook. We also create an entry in `hook` with the current
186When we created the webhook, we told Github to send the event to
187`/~/to/gh/gh-<event>.json?anon&wire=/`. This turns into a poke.
188Let's parse out this url. The first `gh` is the app name to
189poke. The next portion, `gh-<event>` is the mark to convert the
190JSON to. For example, when the "issues" event fires, it'll get
191converted to mark `gh-issues`. This mark has a conversion
192routine from `json` to the `issues` type, which is defined in the
193`gh` structure (`/=home=/sur/gh/hoon`). Once this conversion
194happens, the `gh` app will get poked with the correctly marked
197In `++poke`, we check `hook` to find the subscribers to the event
198that was fired, and we update them.
200That's really all there is to it. Webhook flow isn't well
201standardized, so even if your chosen service is powered by
202webhooks, your listening code might look rather different. The
203point is to expose the correct interface.
206:: This is a connector for the Github API v3.
208:: You can interact with this in a few different ways:
210:: - .^({type} %gx /=gh={/endpoint}) to read data or
211:: .^(arch %gy /=gh={/endpoint}) to explore the possible
214:: - subscribe to /listen/{owner}/{repo}/{events...} for
215:: webhook-powered event notifications. For event list, see
216:: https://developer.github.com/webhooks/.
218:: This is written with the standard structure for api
219:: connectors, as described in lib/connector.hoon.
223/+ gh-parse, connector
227 ++ move (pair bone card)
229 $% {$diff sub-result}
230 {$them wire (unit hiss)}
231 {$hiss wire {$~ $~} $httr {$hiss hiss}}
234 :: Types of results we produce to subscribers.
239 {$gh-list-issues (list issue:gh)}
240 {$gh-issues issues:gh}
241 {$gh-issue-comment issue-comment:gh}
246 :: Types of webhooks we expect.
249 $% {$gh-issues issues:gh}
250 {$gh-issue-comment issue-comment:gh}
253=+ connector=(connector move sub-result) :: Set up connector library
256 hook/(map @t {id/@t listeners/(set bone)}) :: map events to listeners
258:: ++ prep _`. :: Clear state when code changes
264 ^- (list place:connector)
265 =+ (helpers:connector ost.hid wir "https://api.github.com")
266 => |% :: gh-specific helpers
267 ++ sigh-list-issues-x
269 %+ bind ((ar:jo issue:gh-parse) jon)
270 |= issues/(list issue:gh)
271 gh-list-issues+issues
273 ++ sigh-list-issues-y
275 %+ bind ((ar:jo issue:gh-parse) jon)
276 |= issues/(list issue:gh)
277 :- `(shax (jam issues))
278 (malt (turn issues |=(issue:gh [(rsh 3 2 (scot %ui number)) ~])))
283 read-y=(read-static %issues ~)
288 :* guard={$issues $~}
290 read-y=(read-static %mine %by-repo ~)
294 ^- place :: /issues/mine
295 :* guard={$issues $mine $~}
296 read-x=(read-get /issues)
297 read-y=(read-get /issues)
298 sigh-x=sigh-list-issues-x
299 sigh-y=sigh-list-issues-y
301 ^- place :: /issues/by-repo
302 :* guard={$issues $by-repo $~}
306 =+ /(scot %p our.hid)/home/(scot %da now.hid)/web/plan
307 =+ .^({* acc/(map knot plan-acct)} %cx -)
309 ((read-static usr:(~(got by acc) %github) ~) pax)
313 ^- place :: /issues/by-repo/<user>
314 :* guard={$issues $by-repo @t $~}
316 read-y=|=(pax/path (get /users/[-.+>.pax]/repos))
320 %+ bind ((ar:jo repository:gh-parse) jon)
321 |= repos/(list repository:gh)
322 [~ (malt (turn repos |=(repository:gh [name ~])))]
324 ^- place :: /issues/by-repo/<user>/<repo>
325 :* guard={$issues $by-repo @t @t $~}
326 read-x=|=(pax/path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues))
327 read-y=|=(pax/path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues))
328 sigh-x=sigh-list-issues-x
329 sigh-y=sigh-list-issues-y
331 ^- place :: /issues/by-repo/<user>/<repo>
332 :* guard={$issues $by-repo @t @t @t $~}
334 |=(pax/path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues/[-.+>+>.pax]))
336 read-y=(read-static ~)
339 %+ bind (issue:gh-parse jon)
347:: When a peek on a path blocks, ford turns it into a peer on
348:: /scry/{care}/{path}. You can also just peer to this
351:: We hand control to ++scry.
355 ^- {(list move) _+>.$}
358 (read:connector ost.hid (places %read pax) i.pax t.pax)
360:: HTTP response. We make sure the response is good, then
361:: produce the result (as JSON) to whoever sent the request.
364 |= {way/wire res/httr}
365 ^- {(list move) _+>.$}
366 ?. ?=({$read care @ *} way)
374 (sigh:connector (places ren style pax) ren pax res)
376:: HTTP error. We just print it out, though maybe we should
377:: also produce a result so that the request doesn't hang?
380 |= {way/wire tan/tang}
381 ^- {(list move) _+>.$}
382 %- (slog >%gh-sigh-tang< tan)
383 [[ost.hid %diff null+~]~ +>.$]
385:: We can't actually give the response to pretty much anything
386:: without blocking, so we just block unconditionally.
389 |= {ren/@tas tyl/path}
390 ^- (unit (unit (pair mark *)))
393::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
394:: Webhook-powered event streams (/listen) ::
395::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
397:: To listen to a webhook-powered stream of events, subscribe
398:: to /listen/<user>/<repo>/<events...>
400:: We hand control to ++listen.
404 ^- {(list move) _+>.$}
406 ~& [%bad-listen-path pax]
410:: This core handles event subscription requests by starting or
411:: updating the webhook flow for each event.
418 ++ abet :: Resolve core.
419 ^- {(list move) _+>.$}
422 ++ send-hiss :: Send a hiss
425 =+ wir=`wire`[%x %listen pax]
426 +>.$(mow [[ost.hid %hiss wir `~ %httr [%hiss hiz]] mow])
428 :: Create or update a webhook to listen for a set of events.
432 =+ pax=pax :: TMI-proofing
438 ?: (~(has by hook) i.events)
439 $(+>+ (update-hook i.events), events t.events)
440 $(+>+ (create-hook i.events), events t.events)
448 =+ clean-event=`tape`(turn (trip event) |=(a/@tD ?:(=('_' a) '-' a)))
450 %+ ~(put by hook) (crip clean-event)
452 (~(get by hook) (crip clean-event))
453 *{id/@t listeners/(set bone)}
454 [id (~(put in listeners) ost.hid)]
457 =+ [(trip i.pax) (trip i.t.pax)]
458 "https://api.github.com/repos/{-<}/{->}/hooks"
461 %- taco %- crip %- pojo %- jobe :~
464 events+a+~[s+event] ::(turn `(list ,@t)`t.t.pax |=(a=@t s/a))
468 "http://107.170.195.5:8443/~/to/gh/gh-{-}.json?anon&wire=/"
470 [%'content_type' s+%json]
475 :: Add current bone to the list of subscribers for this event.
480 =+ hok=(~(got by hook) event)
483 %+ ~(put by hook) event
484 hok(listeners (~(put in listeners.hok) ost.hid))
488:: Pokes that aren't caught in more specific arms are handled
489:: here. These should be only from webhooks firing, so if we
490:: get any mark that we shouldn't get from a webhook, we reject
491:: it. Otherwise, we spam out the event to everyone who's
492:: listening for that event.
495 |= response/hook-response
496 ^- {(list move) _+>.$}
497 =+ hook-data=(~(get by hook) (rsh 3 3 -.response))
499 ~& [%strange-hook hook response]
501 :: ~& response=response
503 %+ turn (~(tap in listeners.u.hook-data))
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list