docs/arvo/api.md · urbit.org

git.iko.soy/urbit.org

on 8564483 help sign in

urbit.org/docs/arvo/api.md

526 lines 18 kB history raw

1---

2navhome: /docs

3next: true

4sort: 12

5title: API Connectors

6---

8# API Connectors

10Most people have lots of data stored in online services, many of

11which have APIs. API connectors allow the user to access this

12data from within their Urbit.

14A security driver allows the user to make authenticated requests

15to the service's API, but the user still needs to know the API

16and send and receive JSON. An API connector puts a layer of

17porcelain over this to allow the user easier control over their

18data.

20Each connector should perform two basic functions. First, it

21should expose a tree of the data in the service that's accessible

22through `.^` and FUSE. Second, it should expose event streams as

23actions occur in the service. Let's take a look at how the `%gh`

24app accomplishes both for Github.

26After starting `%gh` (`|start %gh`), let's look at the root of

27the tree that `%gh` exposes:

29 ~your-urbit:dojo> .^(arch %gy /=gh=)

30 [fil=~ dir={[p=~.issues q=~]}]

32`%gh` is currently a skeleton -- it contains examples of all the

33functionality necessary for a connector, but many endpoints

34aren't implemented. As we can see here, only issues are

35implemented. We can explore this tree:

37 ~your-urbit:dojo> .^(arch %gy /=gh=/issues)

38 [fil=~ dir={[p=~.by-repo q=~] [p=~.mine q=~]}]

40And eventually:

42 /- gh

43 .^(issue:gh %gx /=gh=/issues/by-repo/philipcmonktest/testing/7)

44 <issue...>

45 login.user:.^(issue:gh %gx /=gh=/issues/by-repo/philipcmonktest/testing/7)

46 'philipcmonktest'

48The other thing an API connector needs to do is expose event

49streams. One straightforward way to use this capacity is with

50the `:pipe` app:

52 :pipe|connect %gh /listen/philipcmonktest/testing/issues/'issue_comment' %public

54Now creating an issue results in the following message in your

55`%public` `:talk` channel:

57 ~your-urbit[philipcmonktest@github]: opened issue #11: i found a bug!

59You'll also get notifications of issue status changes and

60comments.

62As is often the case, the best way to write an API connector is

63to copy an existing one and modify it. We'll dissect the Github

64connector here. This code is in `/=home=/app/gh/hoon`, and it's

65also reproduced at the bottom of this page (in case the code

66drifts out of sync with this doc).

68We'll go over the two parts to an API connector (one-time reading vs listening

69for events) separately.

71## Reading

73A connector exposes a tree of data. Every read request has a

74`care`, which is generally either `%x` or `%y. `%x` is a request

75for a particular piece of data, while `%y` is a request for a

76directory listing. Roughly, `%x` means Unix `cat` and `%y` means

77Unix `ls`.

79> Sometimes you wish to expose a tree where a part of the path

80> can't be enumerated. For example, a `%y` of `/issues/by-repo`

81> "should" produce a list of all Github users, but we don't do

82> that because it's too long. Instead, we just produce our own

83> username (from the `web.plan` file). You can still access

84> repos from other users, you just don't see them in the

85> directory listing.

87The usual flow for implementing this tree of data makes heavy use

88of the `connector` library. This library is well documented in

89the source, so check out `/=home=/lib/connector/hoon`.

91The `connector` library needs to be initialized with definitions

92of `move` and `sub-result` (all the types of data that can be

93returned by a place). This can just be a line at the top of the

94main core:

96 =+ connector=(connector move sub-result) :: Set up connector library

98Most of the Github-specific logic is in `++places`, which is a

99list of all the places we can request. A place consists of:

100

101 ++ place

102 $: guard/mold

103 read-x/$-(path move)

104 read-y/$-(path move)

105 sigh-x/$-(jon/json (unit sub-result))

106 sigh-y/$-(jon/json (unit arch))

107 ==

108

109- `guard`, the type of the paths we should match. For example,

110 to match `/issues/<user>/<repo>` use `{$issues @t @t $~}`.

111

112- `read-x`, called when someone tries to read the place with

113 `care` `%x`. Should produce a single move, usually either a

114 `%diff` response if we can immediately answer or a `%hiss`

115 http request if we need to make a request to the api. See the

116 `++read-*` functions in `++helpers` for some common handlers.

117

118- `read-y`, same as `read-x` except with care `%y`.

119

120- `sigh-x`, called when an http response comes back on this

121 place. You're given the json of the result, and you should

122 produce either a result or null. Null represents an error.

123 If you didn't create an http request in `read-x`, then this

124 should never be called. Use `++sigh-strange` from `++helpers`

125 to unconditionally signal an error.

126

127- `sigh-y`, same as `sigh-x` except with care `%y`. Note that a

128 `%y` request must produce an arch, unlike a `%x` request,

129 which may produce data of any mark.

130

131Filling out the list of places is a lot of grunt work, but most

132places are fairly straightforward, and the `++read-*` helper

133functions are useful. Check out the library source for more

134information on those.

135

136Besides the list of places, we just need to handle the flow of

137control. `++peek` and `++peer-scry` are the interface we expose

138to the rest of the system while `++sigh-httr` and `++sigh-tang`

139are used to handle responses when we make HTTP requests.

140

141- `++peek` should usually just produce `~`. If there are cases

142 where we can respond directly to a `.^` request without

143 blocking on anything, we could do it here, but it's generally

144 not worth the hassle since the same logic should be duplicated

145 in `++peer-scry`.

146

147- `++peer-scry` is where the actual handling for a read request

148 goes. In general, we just need to call `++read` from the

149 `connector` library with the bone, list of places, care, and

150 path. This will match the path to the appropriate place and

151 run either `++read-x` or `++read-y`, depending on the care.

152

153- If `++read-x` or `++read-y` made a successful API request, then

154 the response will come back on `++sigh-httr`. Here, we just

155 need to parse out the wire and call `++sigh` from the

156 `connector` library with the list of places, care, path, and

157 HTTP result. This will match the path to the appropriate place

158 and run either `++sigh-x` or `++sigh-y`, depending on the care.

159

160- If `++read-x` or `++read-y` made an API request that failed,

161 then we'll get a stack trace in `++sigh-tang`. Here, we just

162 print it out and move on.

163

164That's really all there is to the reading portion of API

165connectors.

166

167One of the most accessbile ways to jump into Arvo programming is

168to just add more places to an existing API connector. It's

169useful, small in scope, and comes in bite-sized chunks since most

170places are less than ten lines of code.

171

172## Listening

173

174Listening for events is fairly service-specific. In some

175services, we poll for changes. The Twitter connector has an

176example of this, but note that it predates the `connector`

177library and is thus more complicated than it needs to be, and the

178interface it exposes isn't standard. In Github, we power our

179event streams with webhooks.

180

181For Github, when someone subscribes to

182`/listen/<user>/<repo>/<events...>`, we want to produce

183well-typed results when they occur.

184

185We only want to create one webhook per event, so in our state we

186have `hook`, which is a map of event names to the set of bones

187which are subscribed to that event. We must always keep this

188up-to-date.

189

190Flow of control starts in `++peer-listen`, where we just call

191`++listen`. In `++listen`, for each event in the list of events,

192we check to see if we have that hook set up (by checking whether

193it exists in `hook`). If so, we call `++update-hook` to add the

194current bone to the set of listeners. Otherwise, we call

195`++create-hook`, which sends a request to Github to set up the

196new webhook. We also create an entry in `hook` with the current

197bone.

198

199When we created the webhook, we told Github to send the event to

200`/~/to/gh/gh-<event>.json?anon&wire=/`. This turns into a poke.

201Let's parse out this url. The first `gh` is the app name to

202poke. The next portion, `gh-<event>` is the mark to convert the

203JSON to. For example, when the "issues" event fires, it'll get

204converted to mark `gh-issues`. This mark has a conversion

205routine from `json` to the `issues` type, which is defined in the

206`gh` structure (`/=home=/sur/gh/hoon`). Once this conversion

207happens, the `gh` app will get poked with the correctly marked

208data in `++poke`.

209

210In `++poke`, we check `hook` to find the subscribers to the event

211that was fired, and we update them.

212

213That's really all there is to it. Webhook flow isn't well

214standardized, so even if your chosen service is powered by

215webhooks, your listening code might look rather different. The

216point is to expose the correct interface.

217

218

219## Github API Connector Code

220

221In case the code in `/=home=/app/gh/hoon` drifts out of sync with

222this doc:

223

224```

225:: This is a connector for the Github API v3.

226::

227:: You can interact with this in a few different ways:

228::

229:: - .^({type} %gx /=gh={/endpoint}) to read data or

230:: .^(arch %gy /=gh={/endpoint}) to explore the possible

231:: endpoints.

232::

233:: - subscribe to /listen/{owner}/{repo}/{events...} for

234:: webhook-powered event notifications. For event list, see

235:: https://developer.github.com/webhooks/.

236::

237:: This is written with the standard structure for api

238:: connectors, as described in lib/connector.hoon.

239::

240/? 314

241/- gh, plan-acct

242/+ gh-parse, connector

243::

244!:

245=> |%

246 ++ move (pair bone card)

247 ++ card

248 $% {$diff sub-result}

249 {$them wire (unit hiss)}

250 {$hiss wire {$~ $~} $httr {$hiss hiss}}

251 ==

252 ::

253 :: Types of results we produce to subscribers.

254 ::

255 ++ sub-result

256 $% {$arch arch}

257 {$gh-issue issue:gh}

258 {$gh-list-issues (list issue:gh)}

259 {$gh-issues issues:gh}

260 {$gh-issue-comment issue-comment:gh}

261 {$json json}

262 {$null $~}

263 ==

264 ::

265 :: Types of webhooks we expect.

266 ::

267 ++ hook-response

268 $% {$gh-issues issues:gh}

269 {$gh-issue-comment issue-comment:gh}

270 ==

271 --

272=+ connector=(connector move sub-result) :: Set up connector library

273::

274|_ $: hid/bowl

275 hook/(map @t {id/@t listeners/(set bone)}) :: map events to listeners

276 ==

277:: ++ prep _`. :: Clear state when code changes

278::

279:: List of endpoints

280::

281++ places

282 |= wir/wire

283 ^- (list place:connector)

284 =+ (helpers:connector ost.hid wir "https://api.github.com")

285 => |% :: gh-specific helpers

286 ++ sigh-list-issues-x

287 |= jon/json

288 %+ bind ((ar:jo issue:gh-parse) jon)

289 |= issues/(list issue:gh)

290 gh-list-issues+issues

291 ::

292 ++ sigh-list-issues-y

293 |= jon/json

294 %+ bind ((ar:jo issue:gh-parse) jon)

295 |= issues/(list issue:gh)

296 :- `(shax (jam issues))

297 (malt (turn issues |=(issue:gh [(rsh 3 2 (scot %ui number)) ~])))

298 --

299 :~ ^- place :: /

300 :* guard=$~

301 read-x=read-null

302 read-y=(read-static %issues ~)

303 sigh-x=sigh-strange

304 sigh-y=sigh-strange

305 ==

306 ^- place :: /issues

307 :* guard={$issues $~}

308 read-x=read-null

309 read-y=(read-static %mine %by-repo ~)

310 sigh-x=sigh-strange

311 sigh-y=sigh-strange

312 ==

313 ^- place :: /issues/mine

314 :* guard={$issues $mine $~}

315 read-x=(read-get /issues)

316 read-y=(read-get /issues)

317 sigh-x=sigh-list-issues-x

318 sigh-y=sigh-list-issues-y

319 ==

320 ^- place :: /issues/by-repo

321 :* guard={$issues $by-repo $~}

322 read-x=read-null

323 ^= read-y

324 |= pax/path

325 =+ /(scot %p our.hid)/home/(scot %da now.hid)/web/plan

326 =+ .^({* acc/(map knot plan-acct)} %cx -)

327 ::

328 ((read-static usr:(~(got by acc) %github) ~) pax)

329 sigh-x=sigh-strange

330 sigh-y=sigh-strange

331 ==

332 ^- place :: /issues/by-repo/<user>

333 :* guard={$issues $by-repo @t $~}

334 read-x=read-null

335 read-y=|=(pax/path (get /users/[-.+>.pax]/repos))

336 sigh-x=sigh-strange

337 ^= sigh-y

338 |= jon/json

339 %+ bind ((ar:jo repository:gh-parse) jon)

340 |= repos/(list repository:gh)

341 [~ (malt (turn repos |=(repository:gh [name ~])))]

342 ==

343 ^- place :: /issues/by-repo/<user>/<repo>

344 :* guard={$issues $by-repo @t @t $~}

345 read-x=|=(pax/path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues))

346 read-y=|=(pax/path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues))

347 sigh-x=sigh-list-issues-x

348 sigh-y=sigh-list-issues-y

349 ==

350 ^- place :: /issues/by-repo/<user>/<repo>

351 :* guard={$issues $by-repo @t @t @t $~}

352 ^= read-x

353 |=(pax/path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues/[-.+>+>.pax]))

354 ::

355 read-y=(read-static ~)

356 ^= sigh-x

357 |= jon/json

358 %+ bind (issue:gh-parse jon)

359 |= issue/issue:gh

360 gh-issue+issue

361 ::

362 sigh-y=sigh-strange

363 ==

364 ==

365::

366:: When a peek on a path blocks, ford turns it into a peer on

367:: /scry/{care}/{path}. You can also just peer to this

368:: directly.

369::

370:: We hand control to ++scry.

371::

372++ peer-scry

373 |= pax/path

374 ^- {(list move) _+>.$}

375 ?> ?=({care *} pax)

376 :_ +>.$ :_ ~

377 (read:connector ost.hid (places %read pax) i.pax t.pax)

378::

379:: HTTP response. We make sure the response is good, then

380:: produce the result (as JSON) to whoever sent the request.

381::

382++ sigh-httr

383 |= {way/wire res/httr}

384 ^- {(list move) _+>.$}

385 ?. ?=({$read care @ *} way)

386 ~& res=res

387 [~ +>.$]

388 =* style i.way

389 =* ren i.t.way

390 =* pax t.t.way

391 :_ +>.$ :_ ~

392 :+ ost.hid %diff

393 (sigh:connector (places ren style pax) ren pax res)

394::

395:: HTTP error. We just print it out, though maybe we should

396:: also produce a result so that the request doesn't hang?

397::

398++ sigh-tang

399 |= {way/wire tan/tang}

400 ^- {(list move) _+>.$}

401 %- (slog >%gh-sigh-tang< tan)

402 [[ost.hid %diff null+~]~ +>.$]

403::

404:: We can't actually give the response to pretty much anything

405:: without blocking, so we just block unconditionally.

406::

407++ peek

408 |= {ren/@tas tyl/path}

409 ^- (unit (unit (pair mark *)))

410 ~ ::``noun/[ren tyl]

411::

412::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

413:: Webhook-powered event streams (/listen) ::

414::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

415::

416:: To listen to a webhook-powered stream of events, subscribe

417:: to /listen/<user>/<repo>/<events...>

418::

419:: We hand control to ++listen.

420::

421++ peer-listen

422 |= pax/path

423 ^- {(list move) _+>.$}

424 ?. ?=({@ @ *} pax)

425 ~& [%bad-listen-path pax]

426 [~ +>.$]

427 (listen pax)

428::

429:: This core handles event subscription requests by starting or

430:: updating the webhook flow for each event.

431::

432++ listen

433 |= pax/path

434 =| mow/(list move)

435 =< abet:listen

436 |%

437 ++ abet :: Resolve core.

438 ^- {(list move) _+>.$}

439 [(flop mow) +>.$]

440 ::

441 ++ send-hiss :: Send a hiss

442 |= hiz/hiss

443 ^+ +>

444 =+ wir=`wire`[%x %listen pax]

445 +>.$(mow [[ost.hid %hiss wir `~ %httr [%hiss hiz]] mow])

446 ::

447 :: Create or update a webhook to listen for a set of events.

448 ::

449 ++ listen

450 ^+ .

451 =+ pax=pax :: TMI-proofing

452 ?> ?=({@ @ *} pax)

453 =+ events=t.t.pax

454 |- ^+ +>+.$

455 ?~ events

456 +>+.$

457 ?: (~(has by hook) i.events)

458 $(+>+ (update-hook i.events), events t.events)

459 $(+>+ (create-hook i.events), events t.events)

460 ::

461 :: Set up a webhook.

462 ::

463 ++ create-hook

464 |= event/@t

465 ^+ +>

466 ?> ?=({@ @ *} pax)

467 =+ clean-event=`tape`(turn (trip event) |=(a/@tD ?:(=('_' a) '-' a)))

468 =. hook

469 %+ ~(put by hook) (crip clean-event)

470 =+ %+ fall

471 (~(get by hook) (crip clean-event))

472 *{id/@t listeners/(set bone)}

473 [id (~(put in listeners) ost.hid)]

474 %- send-hiss

475 :* %+ scan

476 =+ [(trip i.pax) (trip i.t.pax)]

477 "https://api.github.com/repos/{-<}/{->}/hooks"

478 auri:epur

479 %post ~ ~

480 %- taco %- crip %- pojo %- jobe :~

481 name+s+%web

482 active+b+&

483 events+a+~[s+event] ::(turn `(list ,@t)`t.t.pax |=(a=@t s/a))

484 :- %config

485 %- jobe :~

486 =+ =+ clean-event

487 "http://107.170.195.5:8443/~/to/gh/gh-{-}.json?anon&wire=/"

488 [%url s+(crip -)]

489 [%'content_type' s+%json]

490 ==

491 ==

492 ==

493 ::

494 :: Add current bone to the list of subscribers for this event.

495 ::

496 ++ update-hook

497 |= event/@t

498 ^+ +>

499 =+ hok=(~(got by hook) event)

500 %_ +>.$

501 hook

502 %+ ~(put by hook) event

503 hok(listeners (~(put in listeners.hok) ost.hid))

504 ==

505 --

506::

507:: Pokes that aren't caught in more specific arms are handled

508:: here. These should be only from webhooks firing, so if we

509:: get any mark that we shouldn't get from a webhook, we reject

510:: it. Otherwise, we spam out the event to everyone who's

511:: listening for that event.

512::

513++ poke

514 |= response/hook-response

515 ^- {(list move) _+>.$}

516 =+ hook-data=(~(get by hook) (rsh 3 3 -.response))

517 ?~ hook-data

518 ~& [%strange-hook hook response]

519 [~ +>.$]

520 :: ~& response=response

521 :_ +>.$

522 %+ turn (~(tap in listeners.u.hook-data))

523 |= ost/bone

524 [ost %diff response]

525--

526```