docs/programming/arvo/system/api.md · urbit.org

git.iko.soy/urbit.org

on 01ac191 help sign in

urbit.org/docs/programming/arvo/system/api.md

525 lines 18 kB history raw

1---

2next: true

3sort: 12

4title: API Connectors

5---

7# API Connectors

9Most people have lots of data stored in online services, many of

10which have APIs. API connectors allow the user to access this

11data from within their Urbit.

13A security driver allows the user to make authenticated requests

14to the service's API, but the user still needs to know the API

15and send and receive JSON. An API connector puts a layer of

16porcelain over this to allow the user easier control over their

17data.

19Each connector should perform two basic functions. First, it

20should expose a tree of the data in the service that's accessible

21through `.^` and FUSE. Second, it should expose event streams as

22actions occur in the service. Let's take a look at how the `%gh`

23app accomplishes both for Github.

25After starting `%gh` (`|start %gh`), let's look at the root of

26the tree that `%gh` exposes:

28 ~your-urbit:dojo> .^(arch %gy /=gh=)

29 [fil=~ dir={[p=~.issues q=~]}]

31`%gh` is currently a skeleton -- it contains examples of all the

32functionality necessary for a connector, but many endpoints

33aren't implemented. As we can see here, only issues are

34implemented. We can explore this tree:

36 ~your-urbit:dojo> .^(arch %gy /=gh=/issues)

37 [fil=~ dir={[p=~.by-repo q=~] [p=~.mine q=~]}]

39And eventually:

41 /- gh

42 .^(issue:gh %gx /=gh=/issues/by-repo/philipcmonktest/testing/7)

43 <issue...>

44 login.user:.^(issue:gh %gx /=gh=/issues/by-repo/philipcmonktest/testing/7)

45 'philipcmonktest'

47The other thing an API connector needs to do is expose event

48streams. One straightforward way to use this capacity is with

49the `:pipe` app:

51 :pipe|connect %gh /listen/philipcmonktest/testing/issues/'issue_comment' %public

53Now creating an issue results in the following message in your

54`%public` `:talk` channel:

56 ~your-urbit[philipcmonktest@github]: opened issue #11: i found a bug!

58You'll also get notifications of issue status changes and

59comments.

61As is often the case, the best way to write an API connector is

62to copy an existing one and modify it. We'll dissect the Github

63connector here. This code is in `/=home=/app/gh/hoon`, and it's

64also reproduced at the bottom of this page (in case the code

65drifts out of sync with this doc).

67We'll go over the two parts to an API connector (one-time reading vs listening

68for events) separately.

70## Reading

72A connector exposes a tree of data. Every read request has a

73`care`, which is generally either `%x` or `%y. `%x` is a request

74for a particular piece of data, while `%y` is a request for a

75directory listing. Roughly, `%x` means Unix `cat` and `%y` means

76Unix `ls`.

78> Sometimes you wish to expose a tree where a part of the path

79> can't be enumerated. For example, a `%y` of `/issues/by-repo`

80> "should" produce a list of all Github users, but we don't do

81> that because it's too long. Instead, we just produce our own

82> username (from the `web.plan` file). You can still access

83> repos from other users, you just don't see them in the

84> directory listing.

86The usual flow for implementing this tree of data makes heavy use

87of the `connector` library. This library is well documented in

88the source, so check out `/=home=/lib/connector/hoon`.

90The `connector` library needs to be initialized with definitions

91of `move` and `sub-result` (all the types of data that can be

92returned by a place). This can just be a line at the top of the

93main core:

95 =+ connector=(connector move sub-result) :: Set up connector library

97Most of the Github-specific logic is in `++places`, which is a

98list of all the places we can request. A place consists of:

100 ++ place

101 $: guard/mold

102 read-x/$-(path move)

103 read-y/$-(path move)

104 sigh-x/$-(jon/json (unit sub-result))

105 sigh-y/$-(jon/json (unit arch))

106 ==

107

108- `guard`, the type of the paths we should match. For example,

109 to match `/issues/<user>/<repo>` use `{$issues @t @t $~}`.

110

111- `read-x`, called when someone tries to read the place with

112 `care` `%x`. Should produce a single move, usually either a

113 `%diff` response if we can immediately answer or a `%hiss`

114 http request if we need to make a request to the api. See the

115 `++read-*` functions in `++helpers` for some common handlers.

116

117- `read-y`, same as `read-x` except with care `%y`.

118

119- `sigh-x`, called when an http response comes back on this

120 place. You're given the json of the result, and you should

121 produce either a result or null. Null represents an error.

122 If you didn't create an http request in `read-x`, then this

123 should never be called. Use `++sigh-strange` from `++helpers`

124 to unconditionally signal an error.

125

126- `sigh-y`, same as `sigh-x` except with care `%y`. Note that a

127 `%y` request must produce an arch, unlike a `%x` request,

128 which may produce data of any mark.

129

130Filling out the list of places is a lot of grunt work, but most

131places are fairly straightforward, and the `++read-*` helper

132functions are useful. Check out the library source for more

133information on those.

134

135Besides the list of places, we just need to handle the flow of

136control. `++peek` and `++peer-scry` are the interface we expose

137to the rest of the system while `++sigh-httr` and `++sigh-tang`

138are used to handle responses when we make HTTP requests.

139

140- `++peek` should usually just produce `~`. If there are cases

141 where we can respond directly to a `.^` request without

142 blocking on anything, we could do it here, but it's generally

143 not worth the hassle since the same logic should be duplicated

144 in `++peer-scry`.

145

146- `++peer-scry` is where the actual handling for a read request

147 goes. In general, we just need to call `++read` from the

148 `connector` library with the bone, list of places, care, and

149 path. This will match the path to the appropriate place and

150 run either `++read-x` or `++read-y`, depending on the care.

151

152- If `++read-x` or `++read-y` made a successful API request, then

153 the response will come back on `++sigh-httr`. Here, we just

154 need to parse out the wire and call `++sigh` from the

155 `connector` library with the list of places, care, path, and

156 HTTP result. This will match the path to the appropriate place

157 and run either `++sigh-x` or `++sigh-y`, depending on the care.

158

159- If `++read-x` or `++read-y` made an API request that failed,

160 then we'll get a stack trace in `++sigh-tang`. Here, we just

161 print it out and move on.

162

163That's really all there is to the reading portion of API

164connectors.

165

166One of the most accessbile ways to jump into Arvo programming is

167to just add more places to an existing API connector. It's

168useful, small in scope, and comes in bite-sized chunks since most

169places are less than ten lines of code.

170

171## Listening

172

173Listening for events is fairly service-specific. In some

174services, we poll for changes. The Twitter connector has an

175example of this, but note that it predates the `connector`

176library and is thus more complicated than it needs to be, and the

177interface it exposes isn't standard. In Github, we power our

178event streams with webhooks.

179

180For Github, when someone subscribes to

181`/listen/<user>/<repo>/<events...>`, we want to produce

182well-typed results when they occur.

183

184We only want to create one webhook per event, so in our state we

185have `hook`, which is a map of event names to the set of bones

186which are subscribed to that event. We must always keep this

187up-to-date.

188

189Flow of control starts in `++peer-listen`, where we just call

190`++listen`. In `++listen`, for each event in the list of events,

191we check to see if we have that hook set up (by checking whether

192it exists in `hook`). If so, we call `++update-hook` to add the

193current bone to the set of listeners. Otherwise, we call

194`++create-hook`, which sends a request to Github to set up the

195new webhook. We also create an entry in `hook` with the current

196bone.

197

198When we created the webhook, we told Github to send the event to

199`/~/to/gh/gh-<event>.json?anon&wire=/`. This turns into a poke.

200Let's parse out this url. The first `gh` is the app name to

201poke. The next portion, `gh-<event>` is the mark to convert the

202JSON to. For example, when the "issues" event fires, it'll get

203converted to mark `gh-issues`. This mark has a conversion

204routine from `json` to the `issues` type, which is defined in the

205`gh` structure (`/=home=/sur/gh/hoon`). Once this conversion

206happens, the `gh` app will get poked with the correctly marked

207data in `++poke`.

208

209In `++poke`, we check `hook` to find the subscribers to the event

210that was fired, and we update them.

211

212That's really all there is to it. Webhook flow isn't well

213standardized, so even if your chosen service is powered by

214webhooks, your listening code might look rather different. The

215point is to expose the correct interface.

216

217

218## Github API Connector Code

219

220In case the code in `/=home=/app/gh/hoon` drifts out of sync with

221this doc:

222

223```

224:: This is a connector for the Github API v3.

225::

226:: You can interact with this in a few different ways:

227::

228:: - .^({type} %gx /=gh={/endpoint}) to read data or

229:: .^(arch %gy /=gh={/endpoint}) to explore the possible

230:: endpoints.

231::

232:: - subscribe to /listen/{owner}/{repo}/{events...} for

233:: webhook-powered event notifications. For event list, see

234:: https://developer.github.com/webhooks/.

235::

236:: This is written with the standard structure for api

237:: connectors, as described in lib/connector.hoon.

238::

239/? 314

240/- gh, plan-acct

241/+ gh-parse, connector

242::

243!:

244=> |%

245 ++ move (pair bone card)

246 ++ card

247 $% {$diff sub-result}

248 {$them wire (unit hiss)}

249 {$hiss wire {$~ $~} $httr {$hiss hiss}}

250 ==

251 ::

252 :: Types of results we produce to subscribers.

253 ::

254 ++ sub-result

255 $% {$arch arch}

256 {$gh-issue issue:gh}

257 {$gh-list-issues (list issue:gh)}

258 {$gh-issues issues:gh}

259 {$gh-issue-comment issue-comment:gh}

260 {$json json}

261 {$null $~}

262 ==

263 ::

264 :: Types of webhooks we expect.

265 ::

266 ++ hook-response

267 $% {$gh-issues issues:gh}

268 {$gh-issue-comment issue-comment:gh}

269 ==

270 --

271=+ connector=(connector move sub-result) :: Set up connector library

272::

273|_ $: hid/bowl

274 hook/(map @t {id/@t listeners/(set bone)}) :: map events to listeners

275 ==

276:: ++ prep _`. :: Clear state when code changes

277::

278:: List of endpoints

279::

280++ places

281 |= wir/wire

282 ^- (list place:connector)

283 =+ (helpers:connector ost.hid wir "https://api.github.com")

284 => |% :: gh-specific helpers

285 ++ sigh-list-issues-x

286 |= jon/json

287 %+ bind ((ar:jo issue:gh-parse) jon)

288 |= issues/(list issue:gh)

289 gh-list-issues+issues

290 ::

291 ++ sigh-list-issues-y

292 |= jon/json

293 %+ bind ((ar:jo issue:gh-parse) jon)

294 |= issues/(list issue:gh)

295 :- `(shax (jam issues))

296 (malt (turn issues |=(issue:gh [(rsh 3 2 (scot %ui number)) ~])))

297 --

298 :~ ^- place :: /

299 :* guard=$~

300 read-x=read-null

301 read-y=(read-static %issues ~)

302 sigh-x=sigh-strange

303 sigh-y=sigh-strange

304 ==

305 ^- place :: /issues

306 :* guard={$issues $~}

307 read-x=read-null

308 read-y=(read-static %mine %by-repo ~)

309 sigh-x=sigh-strange

310 sigh-y=sigh-strange

311 ==

312 ^- place :: /issues/mine

313 :* guard={$issues $mine $~}

314 read-x=(read-get /issues)

315 read-y=(read-get /issues)

316 sigh-x=sigh-list-issues-x

317 sigh-y=sigh-list-issues-y

318 ==

319 ^- place :: /issues/by-repo

320 :* guard={$issues $by-repo $~}

321 read-x=read-null

322 ^= read-y

323 |= pax/path

324 =+ /(scot %p our.hid)/home/(scot %da now.hid)/web/plan

325 =+ .^({* acc/(map knot plan-acct)} %cx -)

326 ::

327 ((read-static usr:(~(got by acc) %github) ~) pax)

328 sigh-x=sigh-strange

329 sigh-y=sigh-strange

330 ==

331 ^- place :: /issues/by-repo/<user>

332 :* guard={$issues $by-repo @t $~}

333 read-x=read-null

334 read-y=|=(pax/path (get /users/[-.+>.pax]/repos))

335 sigh-x=sigh-strange

336 ^= sigh-y

337 |= jon/json

338 %+ bind ((ar:jo repository:gh-parse) jon)

339 |= repos/(list repository:gh)

340 [~ (malt (turn repos |=(repository:gh [name ~])))]

341 ==

342 ^- place :: /issues/by-repo/<user>/<repo>

343 :* guard={$issues $by-repo @t @t $~}

344 read-x=|=(pax/path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues))

345 read-y=|=(pax/path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues))

346 sigh-x=sigh-list-issues-x

347 sigh-y=sigh-list-issues-y

348 ==

349 ^- place :: /issues/by-repo/<user>/<repo>

350 :* guard={$issues $by-repo @t @t @t $~}

351 ^= read-x

352 |=(pax/path (get /repos/[-.+>.pax]/[-.+>+.pax]/issues/[-.+>+>.pax]))

353 ::

354 read-y=(read-static ~)

355 ^= sigh-x

356 |= jon/json

357 %+ bind (issue:gh-parse jon)

358 |= issue/issue:gh

359 gh-issue+issue

360 ::

361 sigh-y=sigh-strange

362 ==

363 ==

364::

365:: When a peek on a path blocks, ford turns it into a peer on

366:: /scry/{care}/{path}. You can also just peer to this

367:: directly.

368::

369:: We hand control to ++scry.

370::

371++ peer-scry

372 |= pax/path

373 ^- {(list move) _+>.$}

374 ?> ?=({care *} pax)

375 :_ +>.$ :_ ~

376 (read:connector ost.hid (places %read pax) i.pax t.pax)

377::

378:: HTTP response. We make sure the response is good, then

379:: produce the result (as JSON) to whoever sent the request.

380::

381++ sigh-httr

382 |= {way/wire res/httr}

383 ^- {(list move) _+>.$}

384 ?. ?=({$read care @ *} way)

385 ~& res=res

386 [~ +>.$]

387 =* style i.way

388 =* ren i.t.way

389 =* pax t.t.way

390 :_ +>.$ :_ ~

391 :+ ost.hid %diff

392 (sigh:connector (places ren style pax) ren pax res)

393::

394:: HTTP error. We just print it out, though maybe we should

395:: also produce a result so that the request doesn't hang?

396::

397++ sigh-tang

398 |= {way/wire tan/tang}

399 ^- {(list move) _+>.$}

400 %- (slog >%gh-sigh-tang< tan)

401 [[ost.hid %diff null+~]~ +>.$]

402::

403:: We can't actually give the response to pretty much anything

404:: without blocking, so we just block unconditionally.

405::

406++ peek

407 |= {ren/@tas tyl/path}

408 ^- (unit (unit (pair mark *)))

409 ~ ::``noun/[ren tyl]

410::

411::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

412:: Webhook-powered event streams (/listen) ::

413::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

414::

415:: To listen to a webhook-powered stream of events, subscribe

416:: to /listen/<user>/<repo>/<events...>

417::

418:: We hand control to ++listen.

419::

420++ peer-listen

421 |= pax/path

422 ^- {(list move) _+>.$}

423 ?. ?=({@ @ *} pax)

424 ~& [%bad-listen-path pax]

425 [~ +>.$]

426 (listen pax)

427::

428:: This core handles event subscription requests by starting or

429:: updating the webhook flow for each event.

430::

431++ listen

432 |= pax/path

433 =| mow/(list move)

434 =< abet:listen

435 |%

436 ++ abet :: Resolve core.

437 ^- {(list move) _+>.$}

438 [(flop mow) +>.$]

439 ::

440 ++ send-hiss :: Send a hiss

441 |= hiz/hiss

442 ^+ +>

443 =+ wir=`wire`[%x %listen pax]

444 +>.$(mow [[ost.hid %hiss wir `~ %httr [%hiss hiz]] mow])

445 ::

446 :: Create or update a webhook to listen for a set of events.

447 ::

448 ++ listen

449 ^+ .

450 =+ pax=pax :: TMI-proofing

451 ?> ?=({@ @ *} pax)

452 =+ events=t.t.pax

453 |- ^+ +>+.$

454 ?~ events

455 +>+.$

456 ?: (~(has by hook) i.events)

457 $(+>+ (update-hook i.events), events t.events)

458 $(+>+ (create-hook i.events), events t.events)

459 ::

460 :: Set up a webhook.

461 ::

462 ++ create-hook

463 |= event/@t

464 ^+ +>

465 ?> ?=({@ @ *} pax)

466 =+ clean-event=`tape`(turn (trip event) |=(a/@tD ?:(=('_' a) '-' a)))

467 =. hook

468 %+ ~(put by hook) (crip clean-event)

469 =+ %+ fall

470 (~(get by hook) (crip clean-event))

471 *{id/@t listeners/(set bone)}

472 [id (~(put in listeners) ost.hid)]

473 %- send-hiss

474 :* %+ scan

475 =+ [(trip i.pax) (trip i.t.pax)]

476 "https://api.github.com/repos/{-<}/{->}/hooks"

477 auri:epur

478 %post ~ ~

479 %- taco %- crip %- pojo %- jobe :~

480 name+s+%web

481 active+b+&

482 events+a+~[s+event] ::(turn `(list ,@t)`t.t.pax |=(a=@t s/a))

483 :- %config

484 %- jobe :~

485 =+ =+ clean-event

486 "http://107.170.195.5:8443/~/to/gh/gh-{-}.json?anon&wire=/"

487 [%url s+(crip -)]

488 [%'content_type' s+%json]

489 ==

490 ==

491 ==

492 ::

493 :: Add current bone to the list of subscribers for this event.

494 ::

495 ++ update-hook

496 |= event/@t

497 ^+ +>

498 =+ hok=(~(got by hook) event)

499 %_ +>.$

500 hook

501 %+ ~(put by hook) event

502 hok(listeners (~(put in listeners.hok) ost.hid))

503 ==

504 --

505::

506:: Pokes that aren't caught in more specific arms are handled

507:: here. These should be only from webhooks firing, so if we

508:: get any mark that we shouldn't get from a webhook, we reject

509:: it. Otherwise, we spam out the event to everyone who's

510:: listening for that event.

511::

512++ poke

513 |= response/hook-response

514 ^- {(list move) _+>.$}

515 =+ hook-data=(~(get by hook) (rsh 3 3 -.response))

516 ?~ hook-data

517 ~& [%strange-hook hook response]

518 [~ +>.$]

519 :: ~& response=response

520 :_ +>.$

521 %+ turn (~(tap in listeners.u.hook-data))

522 |= ost/bone

523 [ost %diff response]

524--

525```