10Most people have lots of data stored in online services, many of
11which have APIs. API connectors allow the user to access this
12data from within their Urbit.
14A security driver allows the user to make authenticated requests
15to the service's API, but the user still needs to know the API
16and send and receive JSON. An API connector puts a layer of
17porcelain over this to allow the user easier control over their
20Each connector should perform two basic functions. First, it
21should expose a tree of the data in the service that's accessible
22through `.^` and FUSE. Second, it should expose event streams as
23actions occur in the service. Let's take a look at how the `%gh`
24app accomplishes both for Github.
26After starting `%gh` (`|start %gh`), let's look at the root of
27the tree that `%gh` exposes:
30 ~your-urbit:dojo> .^(arch %gy /=gh=)
31 [fil=~ dir={[p=~.issues q=~]}]
34`%gh` is currently a skeleton -- it contains examples of all the
35functionality necessary for a connector, but many endpoints
36aren't implemented. As we can see here, only issues are
37implemented. We can explore this tree:
40 ~your-urbit:dojo> .^(arch %gy /=gh=/issues)
41 [fil=~ dir={[p=~.by-repo q=~] [p=~.mine q=~]}]
48 .^(issue:gh %gx /=gh=/issues/by-repo/philipcmonktest/testing/7)
50 login.user:.^(issue:gh %gx /=gh=/issues/by-repo/philipcmonktest/testing/7)
54The other thing an API connector needs to do is expose event
55streams. One straightforward way to use this capacity is with
59 :pipe|connect %gh /listen/philipcmonktest/testing/issues/'issue_comment' %public
62Now creating an issue results in the following message in your
63`%public` `:talk` channel:
66 ~your-urbit[philipcmonktest@github]: opened issue #11: i found a bug!
69You'll also get notifications of issue status changes and
72As is often the case, the best way to write an API connector is
73to copy an existing one and modify it. We'll dissect the Github
74connector here. This code is in `/=home=/app/gh/hoon`, and it's
75also reproduced at the bottom of this page (in case the code
76drifts out of sync with this doc).
78We'll go over the two parts to an API connector (one-time reading vs listening
79for events) separately.
83A connector exposes a tree of data. Every read request has a
84`care`, which is generally either `%x` or `%y. `%x` is a request
85for a particular piece of data, while `%y` is a request for a
86directory listing. Roughly, `%x` means Unix `cat` and `%y` means
89> Sometimes you wish to expose a tree where a part of the path
90> can't be enumerated. For example, a `%y` of `/issues/by-repo`
91> "should" produce a list of all Github users, but we don't do
92> that because it's too long. Instead, we just produce our own
93> username (from the `web.plan` file). You can still access
94> repos from other users, you just don't see them in the
97The usual flow for implementing this tree of data makes heavy use
98of the `connector` library. This library is well documented in
99the source, so check out `/=home=/lib/connector/hoon`.
101The `connector` library needs to be initialized with definitions
102of `move` and `sub-result` (all the types of data that can be
103returned by a place). This can just be a line at the top of the
107 =+ connector=(connector move sub-result) :: Set up connector library
110Most of the Github-specific logic is in `++places`, which is a
111list of all the places we can request. A place consists of:
118 sigh-x/$-(jon/json (unit sub-result))
119 sigh-y/$-(jon/json (unit arch))
123- `guard`, the type of the paths we should match. For example, to match `/issues/<user>/<repo>` use `{$issues @t @t $~}`.
125- `read-x`, called when someone tries to read the place with `care` `%x`. Should produce a single move, usually either a `%diff` response if we can immediately answer or a `%hiss` http request if we need to make a request to the api. See the `++read-*` functions in `++helpers` for some common handlers.
127- `read-y`, same as `read-x` except with care `%y`.
129- `sigh-x`, called when an http response comes back on this place. You're given the json of the result, and you should produce either a result or null. Null represents an error. If you didn't create an http request in `read-x`, then this should never be called. Use `++sigh-strange` from `++helpers` to unconditionally signal an error.
131- `sigh-y`, same as `sigh-x` except with care `%y`. Note that a `%y` request must produce an arch, unlike a `%x` request, which may produce data of any mark.
133Filling out the list of places is a lot of grunt work, but most
134places are fairly straightforward, and the `++read-*` helper
135functions are useful. Check out the library source for more
138Besides the list of places, we just need to handle the flow of
139control. `++peek` and `++peer-scry` are the interface we expose
140to the rest of the system while `++sigh-httr` and `++sigh-tang`
141are used to handle responses when we make HTTP requests.
143- `++peek` should usually just produce `~`. If there are cases where we can respond directly to a `.^` request without blocking on anything, we could do it here, but it's generally not worth the hassle since the same logic should be duplicated in `++peer-scry`.
145- `++peer-scry` is where the actual handling for a read request goes. In general, we just need to call `++read` from the `connector` library with the bone, list of places, care, and path. This will match the path to the appropriate place and run either `++read-x` or `++read-y`, depending on the care.
147- If `++read-x` or `++read-y` made a successful API request, then the response will come back on `++sigh-httr`. Here, we just need to parse out the wire and call `++sigh` from the `connector` library with the list of places, care, path, and HTTP result. This will match the path to the appropriate place and run either `++sigh-x` or `++sigh-y`, depending on the care.
149- If `++read-x` or `++read-y` made an API request that failed, then we'll get a stack trace in `++sigh-tang`. Here, we just print it out and move on.
151That's really all there is to the reading portion of API
154One of the most accessbile ways to jump into Arvo programming is
155to just add more places to an existing API connector. It's
156useful, small in scope, and comes in bite-sized chunks since most
157places are less than ten lines of code.
161Listening for events is fairly service-specific. In some
162services, we poll for changes. The Twitter connector has an
163example of this, but note that it predates the `connector`
164library and is thus more complicated than it needs to be, and the
165interface it exposes isn't standard. In Github, we power our
166event streams with webhooks.
168For Github, when someone subscribes to
169`/listen/<user>/<repo>/<events...>`, we want to produce
170well-typed results when they occur.
172We only want to create one webhook per event, so in our state we
173have `hook`, which is a map of event names to the set of bones
174which are subscribed to that event. We must always keep this
177Flow of control starts in `++peer-listen`, where we just call
178`++listen`. In `++listen`, for each event in the list of events,
179we check to see if we have that hook set up (by checking whether
180it exists in `hook`). If so, we call `++update-hook` to add the
181current bone to the set of listeners. Otherwise, we call
182`++create-hook`, which sends a request to Github to set up the
183new webhook. We also create an entry in `hook` with the current
186When we created the webhook, we told Github to send the event to
187`/~/to/gh/gh-<event>.json?anon&wire=/`. This turns into a poke.
188Let's parse out this url. The first `gh` is the app name to
189poke. The next portion, `gh-<event>` is the mark to convert the
190JSON to. For example, when the "issues" event fires, it'll get
191converted to mark `gh-issues`. This mark has a conversion
192routine from `json` to the `issues` type, which is defined in the
193`gh` structure (`/=home=/sur/gh/hoon`). Once this conversion
194happens, the `gh` app will get poked with the correctly marked
197In `++poke`, we check `hook` to find the subscribers to the event
198that was fired, and we update them.
200That's really all there is to it. Webhook flow isn't well
201standardized, so even if your chosen service is powered by
202webhooks, your listening code might look rather different. The
203point is to expose the correct interface.
206## Github API Connector Code
208In case the code in `/=home=/app/gh/hoon` drifts out of sync with
212:: This is a connector for the Github API v3.
214:: You can interact with this in a few different ways:
216:: - .^({type} %gx /=gh={/endpoint}) to read data or
217:: .^(arch %gy /=gh={/endpoint}) to explore the possible
220:: - subscribe to /listen/{owner}/{repo}/{events...} for
221:: webhook-powered event notifications. For event list, see
222:: https://developer.github.com/webhooks/.
224:: This is written with the standard structure for api
225:: connectors, as described in lib/connector.hoon.
229/+ gh-parse, connector
233 +$ move (pair bone card)
235 $% [%diff sub-result]
236 [%them wire (unit hiss)]
237 [%hiss wire [$~ $~] %httr [%hiss hiss]]
240 :: Types of results we produce to subscribers.
245 [$gh-list-issues (list issue:gh)]
246 [$gh-issues issues:gh]
247 [$gh-issue-comment issue-comment:gh]
252 :: Types of webhooks we expect.
255 $% [$gh-issues issues:gh]
256 [$gh-issue-comment issue-comment:gh]
259=+ connector=(connector move sub-result) :: Set up connector library
262 hook=(map @t [id=@t listeners=(set bone)]) :: map events to listeners
264:: ++ prep _`. :: Clear state when code changes
270 ^- (list place:connector)
271 =+ (helpers:connector ost.hid wir "https://api.github.com")
272 => |% :: gh-specific helpers
273 ++ sigh-list-issues-x
275 %+ bind ((ar:jo issue:gh-parse) jon)
276 |= issues=(list issue:gh)
277 gh-list-issues+issues
279 ++ sigh-list-issues-y
281 %+ bind ((ar:jo issue:gh-parse) jon)
282 |= issues=(list issue:gh)
283 :- `(shax (jam issues))
284 (malt (turn issues |=(issue:gh [(rsh 3 2 (scot %ui number)) ~])))
289 read-y=(read-static %issues ~)
294 :* guard=[$issues $~]
296 read-y=(read-static %mine %by-repo ~)
300 ^- place :: /issues/mine
301 :* guard=[$issues $mine $~]
302 read-x=(read-get /issues)
303 read-y=(read-get /issues)
304 sigh-x=sigh-list-issues-x
305 sigh-y=sigh-list-issues-y
307 ^- place :: /issues/by-repo
308 :* guard=[$issues $by-repo $~]
312 =+ /(scot %p our.hid)/home/(scot %da now.hid)/web/plan
313 =+ .^([* acc=(map knot plan-acct)] %cx -)
315 ((read-static usr:(~(got by acc) %github) ~) pax)
319 ^- place :: /issues/by-repo/<user>
320 :* guard=[$issues $by-repo @t $~]
322 read-y=|=(pax=path (get /users/[-.+>.pax]/repos))
326 %+ bind ((ar:jo repository:gh-parse) jon)
327 |= repos=(list repository:gh)
328 [~ (malt (turn repos |=(repository:gh [name ~])))]
330 ^- place :: /issues/by-repo/<user>/<repo>
331 :* guard=[$issues $by-repo @t @t $~]
332 read-x=|=(pax=path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues))
333 read-y=|=(pax=path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues))
334 sigh-x=sigh-list-issues-x
335 sigh-y=sigh-list-issues-y
337 ^- place :: /issues/by-repo/<user>/<repo>
338 :* guard=[$issues $by-repo @t @t @t $~]
340 |=(pax=path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues/[-.+>+>.pax]))
342 read-y=(read-static ~)
345 %+ bind (issue:gh-parse jon)
353:: When a peek on a path blocks, ford turns it into a peer on
354:: /scry/{care}/{path}. You can also just peer to this
357:: We hand control to ++scry.
361 ^- [(list move) _+>.$]
364 (read:connector ost.hid (places %read pax) i.pax t.pax)
366:: HTTP response. We make sure the response is good, then
367:: produce the result (as JSON) to whoever sent the request.
370 |= [way=wire res=httr]
371 ^- [(list move) _+>.$]
372 ?. ?=([$read care @ *] way)
380 (sigh:connector (places ren style pax) ren pax res)
382:: HTTP error. We just print it out, though maybe we should
383:: also produce a result so that the request doesn't hang?
386 |= [way=wire tan=tang]
387 ^- [(list move) _+>.$]
388 %- (slog >%gh-sigh-tang< tan)
389 [[ost.hid %diff null+~]~ +>.$]
391:: We can't actually give the response to pretty much anything
392:: without blocking, so we just block unconditionally.
395 |= [ren=@tas tyl=path]
396 ^- (unit (unit (pair mark *)))
399::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
400:: Webhook-powered event streams (/listen) ::
401::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
403:: To listen to a webhook-powered stream of events, subscribe
404:: to /listen/<user>/<repo>/<events...>
406:: We hand control to ++listen.
410 ^- [(list move) _+>.$]
412 ~& [%bad-listen-path pax]
416:: This core handles event subscription requests by starting or
417:: updating the webhook flow for each event.
424 ++ abet :: Resolve core.
425 ^- [(list move) _+>.$]
428 ++ send-hiss :: Send a hiss
431 =+ wir=`wire`[%x %listen pax]
432 +>.$(mow [[ost.hid %hiss wir `~ %httr [%hiss hiz]] mow])
434 :: Create or update a webhook to listen for a set of events.
438 =+ pax=pax :: TMI-proofing
444 ?: (~(has by hook) i.events)
445 $(+>+ (update-hook i.events), events t.events)
446 $(+>+ (create-hook i.events), events t.events)
454 =+ clean-event=`tape`(turn (trip event) |=(a=@tD ?:(=('_' a) '-' a)))
456 %+ ~(put by hook) (crip clean-event)
458 (~(get by hook) (crip clean-event))
459 *[id=@t listeners=(set bone)]
460 [id (~(put in listeners) ost.hid)]
463 =+ [(trip i.pax) (trip i.t.pax)]
464 "https://api.github.com/repos/{-<}/{->}/hooks"
467 %- taco %- crip %- pojo %- jobe :~
470 events+a+~[s+event] ::(turn `(list ,@t)`t.t.pax |=(a=@t s=a))
474 "http://107.170.195.5:8443/~/to/gh/gh-{-}.json?anon&wire=/"
476 [%'content_type' s+%json]
481 :: Add current bone to the list of subscribers for this event.
486 =+ hok=(~(got by hook) event)
489 %+ ~(put by hook) event
490 hok(listeners (~(put in listeners.hok) ost.hid))
494:: Pokes that aren't caught in more specific arms are handled
495:: here. These should be only from webhooks firing, so if we
496:: get any mark that we shouldn't get from a webhook, we reject
497:: it. Otherwise, we spam out the event to everyone who's
498:: listening for that event.
501 |= response=hook-response
502 ^- [(list move) _+>.$]
503 =+ hook-data=(~(get by hook) (rsh 3 3 -.response))
505 ~& [%strange-hook hook response]
507 :: ~& response=response
509 %+ turn (~(tap in listeners.u.hook-data))
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list