10> Note: This lesson depends on some code that needs to be updated before the examples will work correctly. This will be addressed in the near future. We apologize!
12Most people have lots of data stored in online services, many of
13which have APIs. API connectors allow the user to access this
14data from within their Urbit.
16A security driver allows the user to make authenticated requests
17to the service's API, but the user still needs to know the API
18and send and receive JSON. An API connector puts a layer of
19porcelain over this to allow the user easier control over their
22Each connector should perform two basic functions. First, it
23should expose a tree of the data in the service that's accessible
24through `.^` and FUSE. Second, it should expose event streams as
25actions occur in the service. Let's take a look at how the `%gh`
26app accomplishes both for Github.
28After starting `%gh` (`|start %gh`), let's look at the root of
29the tree that `%gh` exposes:
32~your-urbit:dojo> .^(arch %gy /=gh=)
33[fil=~ dir={[p=~.issues q=~]}]
36`%gh` is currently a skeleton -- it contains examples of all the
37functionality necessary for a connector, but many endpoints
38aren't implemented. As we can see here, only issues are
39implemented. We can explore this tree:
42~your-urbit:dojo> .^(arch %gy /=gh=/issues)
43[fil=~ dir={[p=~.by-repo q=~] [p=~.mine q=~]}]
50.^(issue:gh %gx /=gh=/issues/by-repo/philipcmonktest/testing/7)
52login.user:.^(issue:gh %gx /=gh=/issues/by-repo/philipcmonktest/testing/7)
56The other thing an API connector needs to do is expose event
57streams. One straightforward way to use this capacity is with
61:pipe|connect %gh /listen/philipcmonktest/testing/issues/'issue_comment' %public
64Now creating an issue results in the following message in your
65`%public` `:talk` channel:
68~your-urbit[philipcmonktest@github]: opened issue #11: i found a bug!
71You'll also get notifications of issue status changes and
74As is often the case, the best way to write an API connector is
75to copy an existing one and modify it. We'll dissect the Github
76connector here. This code is in `/=home=/app/gh/hoon`, and it's
77also reproduced at the bottom of this page (in case the code
78drifts out of sync with this doc).
80We'll go over the two parts to an API connector (one-time reading vs listening
81for events) separately.
85A connector exposes a tree of data. Every read request has a
86`care`, which is generally either `%x` or `%y. `%x` is a request
87for a particular piece of data, while `%y` is a request for a
88directory listing. Roughly, `%x` means Unix `cat` and `%y` means
91> Sometimes you wish to expose a tree where a part of the path can't be enumerated. For example, a `%y` of `/issues/by-repo` "should" produce a list of all Github users, but we don't do that because it's too long. Instead, we just produce our own username (from the `web.plan` file). You can still access repos from other users, you just don't see them in the directory listing.
93The usual flow for implementing this tree of data makes heavy use
94of the `connector` library. This library is well documented in
95the source, so check out `/=home=/lib/connector/hoon`.
97The `connector` library needs to be initialized with definitions
98of `move` and `sub-result` (all the types of data that can be
99returned by a place). This can just be a line at the top of the
103=+ connector=(connector move sub-result) :: Set up connector library
106Most of the Github-specific logic is in `++places`, which is a
107list of all the places we can request. A place consists of:
114 sigh-x/$-(jon/json (unit sub-result))
115 sigh-y/$-(jon/json (unit arch))
119- `guard`, the type of the paths we should match. For example, to match `/issues/<user>/<repo>` use `{$issues @t @t $~}`.
121- `read-x`, called when someone tries to read the place with `care` `%x`. Should produce a single move, usually either a `%diff` response if we can immediately answer or a `%hiss` http request if we need to make a request to the api. See the `++read-*` functions in `++helpers` for some common handlers.
123- `read-y`, same as `read-x` except with care `%y`.
125- `sigh-x`, called when an http response comes back on this place. You're given the json of the result, and you should produce either a result or null. Null represents an error. If you didn't create an http request in `read-x`, then this should never be called. Use `++sigh-strange` from `++helpers` to unconditionally signal an error.
127- `sigh-y`, same as `sigh-x` except with care `%y`. Note that a `%y` request must produce an arch, unlike a `%x` request, which may produce data of any mark.
129Filling out the list of places is a lot of grunt work, but most
130places are fairly straightforward, and the `++read-*` helper
131functions are useful. Check out the library source for more
134Besides the list of places, we just need to handle the flow of
135control. `++peek` and `++peer-scry` are the interface we expose
136to the rest of the system while `++sigh-httr` and `++sigh-tang`
137are used to handle responses when we make HTTP requests.
139- `++peek` should usually just produce `~`. If there are cases where we can respond directly to a `.^` request without blocking on anything, we could do it here, but it's generally not worth the hassle since the same logic should be duplicated in `++peer-scry`.
141- `++peer-scry` is where the actual handling for a read request goes. In general, we just need to call `++read` from the `connector` library with the bone, list of places, care, and path. This will match the path to the appropriate place and run either `++read-x` or `++read-y`, depending on the care.
143- If `++read-x` or `++read-y` made a successful API request, then the response will come back on `++sigh-httr`. Here, we just need to parse out the wire and call `++sigh` from the `connector` library with the list of places, care, path, and HTTP result. This will match the path to the appropriate place and run either `++sigh-x` or `++sigh-y`, depending on the care.
145- If `++read-x` or `++read-y` made an API request that failed, then we'll get a stack trace in `++sigh-tang`. Here, we just print it out and move on.
147That's really all there is to the reading portion of API
150One of the most accessible ways to jump into Arvo programming is
151to just add more places to an existing API connector. It's
152useful, small in scope, and comes in bite-sized chunks since most
153places are less than ten lines of code.
157Listening for events is fairly service-specific. In some
158services, we poll for changes. The Twitter connector has an
159example of this, but note that it predates the `connector`
160library and is thus more complicated than it needs to be, and the
161interface it exposes isn't standard. In Github, we power our
162event streams with webhooks.
164For Github, when someone subscribes to
165`/listen/<user>/<repo>/<events...>`, we want to produce
166well-typed results when they occur.
168We only want to create one webhook per event, so in our state we
169have `hook`, which is a map of event names to the set of bones
170which are subscribed to that event. We must always keep this
173Flow of control starts in `++peer-listen`, where we just call
174`++listen`. In `++listen`, for each event in the list of events,
175we check to see if we have that hook set up (by checking whether
176it exists in `hook`). If so, we call `++update-hook` to add the
177current bone to the set of listeners. Otherwise, we call
178`++create-hook`, which sends a request to Github to set up the
179new webhook. We also create an entry in `hook` with the current
182When we created the webhook, we told Github to send the event to
183`/~/to/gh/gh-<event>.json?anon&wire=/`. This turns into a poke.
184Let's parse out this url. The first `gh` is the app name to
185poke. The next portion, `gh-<event>` is the mark to convert the
186JSON to. For example, when the "issues" event fires, it'll get
187converted to mark `gh-issues`. This mark has a conversion
188routine from `json` to the `issues` type, which is defined in the
189`gh` structure (`/=home=/sur/gh/hoon`). Once this conversion
190happens, the `gh` app will get poked with the correctly marked
193In `++poke`, we check `hook` to find the subscribers to the event
194that was fired, and we update them.
196That's really all there is to it. Webhook flow isn't well
197standardized, so even if your chosen service is powered by
198webhooks, your listening code might look rather different. The
199point is to expose the correct interface.
202## Github API Connector Code
204In case the code in `/=home=/app/gh/hoon` drifts out of sync with
208:: This is a connector for the Github API v3.
210:: You can interact with this in a few different ways:
212:: - .^({type} %gx /=gh={/endpoint}) to read data or
213:: .^(arch %gy /=gh={/endpoint}) to explore the possible
216:: - subscribe to /listen/{owner}/{repo}/{events...} for
217:: webhook-powered event notifications. For event list, see
218:: https://developer.github.com/webhooks/.
220:: This is written with the standard structure for api
221:: connectors, as described in lib/connector.hoon.
225/+ gh-parse, connector
229 +$ move (pair bone card)
231 $% [%diff sub-result]
232 [%them wire (unit hiss:eyre)]
233 [%hiss wire [%~ %~] %httr [%hiss hiss:eyre]]
236 :: Types of results we produce to subscribers.
241 [%gh-list-issues (list issue:gh)]
242 [%gh-issues issues:gh]
243 [%gh-issue-comment issue-comment:gh]
248 :: Types of webhooks we expect.
251 $% [%gh-issues issues:gh]
252 [%gh-issue-comment issue-comment:gh]
255=+ connector=(connector move sub-result) :: Set up connector library
258 hook=(map @t [id=@t listeners=(set bone)]) :: map events to listeners
260:: ++ prep _`. :: Clear state when code changes
266 ^- (list place:connector)
267 =+ (helpers:connector ost.hid wir "https://api.github.com")
268 => |% :: gh-specific helpers
269 ++ sigh-list-issues-x
271 %+ bind ((ar:jo issue:gh-parse) jon)
272 |= issues=(list issue:gh)
273 gh-list-issues+issues
275 ++ sigh-list-issues-y
277 %+ bind ((ar:jo issue:gh-parse) jon)
278 |= issues=(list issue:gh)
279 :- `(shax (jam issues))
280 (malt (turn issues |=(issue:gh [(rsh 3 2 (scot %ui number)) ~])))
285 read-y=(read-static %issues ~)
290 :* guard=,[%issues %~]
292 read-y=(read-static %mine %by-repo ~)
296 ^- place :: /issues/mine
297 :* guard=,[%issues %mine %~]
298 read-x=(read-get /issues)
299 read-y=(read-get /issues)
300 sigh-x=sigh-list-issues-x
301 sigh-y=sigh-list-issues-y
303 ^- place :: /issues/by-repo
304 :* guard=,[%issues %by-repo %~]
308 =+ /(scot %p our.hid)/home/(scot %da now.hid)/web/plan
309 =+ .^([* acc=(map knot plan-acct)] %cx -)
311 ((read-static usr:(~(got by acc) %github) ~) pax)
315 ^- place :: /issues/by-repo/<user>
316 :* guard=,[%issues %by-repo @t %~]
318 read-y=|=(pax=path (get /users/[-.+>.pax]/repos))
322 %+ bind ((ar:jo repository:gh-parse) jon)
323 |= repos=(list repository:gh)
324 [~ (malt (turn repos |=(repository:gh [name ~])))]
326 ^- place :: /issues/by-repo/<user>/<repo>
327 :* guard=,[%issues %by-repo @t @t %~]
328 read-x=|=(pax=path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues))
329 read-y=|=(pax=path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues))
330 sigh-x=sigh-list-issues-x
331 sigh-y=sigh-list-issues-y
333 ^- place :: /issues/by-repo/<user>/<repo>
334 :* guard=,[%issues %by-repo @t @t @t %~]
336 |=(pax=path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues/[-.+>+>.pax]))
338 read-y=(read-static ~)
341 %+ bind (issue:gh-parse jon)
349:: When a peek on a path blocks, ford turns it into a peer on
350:: /scry/{care}/{path}. You can also just peer to this
353:: We hand control to ++scry.
357 ^- [(list move) _+>.$]
358 ?> ?=([care:clay *] pax)
360 (read:connector ost.hid (places %read pax) i.pax t.pax)
362:: HTTP response. We make sure the response is good, then
363:: produce the result (as JSON) to whoever sent the request.
366 |= [way=wire res=httr:eyre]
367 ^- [(list move) _+>.$]
368 ?. ?=([%read care:clay @ *] way)
376 (sigh:connector (places ren style pax) ren pax res)
378:: HTTP error. We just print it out, though maybe we should
379:: also produce a result so that the request doesn't hang?
382 |= [way=wire tan=tang]
383 ^- [(list move) _+>.$]
384 %- (slog >%gh-sigh-tang< tan)
385 [[ost.hid %diff null+~]~ +>.$]
387:: We can't actually give the response to pretty much anything
388:: without blocking, so we just block unconditionally.
391 |= [ren=@tas tyl=path]
392 ^- (unit (unit (pair mark *)))
395::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
396:: Webhook-powered event streams (/listen) ::
397::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
399:: To listen to a webhook-powered stream of events, subscribe
400:: to /listen/<user>/<repo>/<events...>
402:: We hand control to ++listen.
406 ^- [(list move) _+>.$]
408 ~& [%bad-listen-path pax]
412:: This core handles event subscription requests by starting or
413:: updating the webhook flow for each event.
420 ++ abet :: Resolve core.
421 ^- [(list move) _+>.$]
424 ++ send-hiss :: Send a hiss
427 =+ wir=`wire`[%x %listen pax]
428 +>.$(mow [[ost.hid %hiss wir `~ %httr [%hiss hiz]] mow])
430 :: Create or update a webhook to listen for a set of events.
434 =+ pax=pax :: TMI-proofing
440 ?: (~(has by hook) i.events)
441 $(+>+ (update-hook i.events), events t.events)
442 $(+>+ (create-hook i.events), events t.events)
450 =+ clean-event=`tape`(turn (trip event) |=(a=@tD ?:(=('_' a) '-' a)))
452 %+ ~(put by hook) (crip clean-event)
454 (~(get by hook) (crip clean-event))
455 *[id=@t listeners=(set bone)]
456 [id (~(put in listeners) ost.hid)]
459 =+ [(trip i.pax) (trip i.t.pax)]
460 "https://api.github.com/repos/{-<}/{->}/hooks"
463 %- as-octs:mimes:html %- crip %- en-json:html %- pairs:enjs:format :~
466 events+a+~[s+event] ::(turn `(list ,@t)`t.t.pax |=(a=@t s=a))
468 %- pairs:enjs:format :~
470 "http://107.170.195.5:8443/~/to/gh/gh-{-}.json?anon&wire=/"
472 [%'content_type' s+%json]
477 :: Add current bone to the list of subscribers for this event.
482 =+ hok=(~(got by hook) event)
485 %+ ~(put by hook) event
486 hok(listeners (~(put in listeners.hok) ost.hid))
490:: Pokes that aren't caught in more specific arms are handled
491:: here. These should be only from webhooks firing, so if we
492:: get any mark that we shouldn't get from a webhook, we reject
493:: it. Otherwise, we spam out the event to everyone who's
494:: listening for that event.
497 |= response=hook-response
498 ^- [(list move) _+>.$]
499 =+ hook-data=(~(get by hook) (rsh 3 3 -.response))
501 ~& [%strange-hook hook response]
503 :: ~& response=response
505 %+ turn ~(tap in listeners.u.hook-data)
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list