@@ -1,234 +0,0 @@
1−+++
2−title = "Network Explorer: Data Pipeline & HTTP API"
3−date = "2021-03-11"
4−
5−[taxonomies]
6−grant_type = [ "Bounty" ]
7−grant_category = [ "App Dev" ]
8−
9−[extra]
10−image = ""
11−description = "Build a public, open HTTP API for exploring metrics about the Urbit network."
12−reward = 5
13−assignee = "Christopher King"
14−id = "1101665172"
15−completed = false
16−canceled = true
17−link = ""
18−+++
19−
20−## Overview
21−
22−
23−
24−### Goal
25−
26−Build a public, open HTTP API for exploring metrics about the Urbit network.
27−
28−### Background
29−
30−The project known as the **Network Explorer** has been in the works at Tlon for
31−quite some time. The idea is to build a publicly-accessible tool (hosted on
32−https://network.urbit.org) that provides key metrics and insights into the state
33−of the Urbit network. A series of high-fidelity designs have been produced.
34−
35−### Scope
36−
37−This bounty covers the creation of the:
38−
39−- **data pipeline**: we're concerned with data from the PKI (Azimuth) and
40− the Urbit network itself. These two datasets will need to be aggregated and
41− indexed for efficient querying.
42−- **HTTP API**: the interface for querying data from a client, like the Network
43− Explorer.
44−
45−The Network Explorer interface will be covered in a separate bounty.
46−
47−## Implementation
48−
49−### Data Model
50−
51−Let's start with the data model. There are two sources of data that we need:
52−
53−- **PKI (Azimuth)**: The public-key infrastructure, currently known as Azimuth,
54− is our primary source of data. This information is currently found on Ethereum
55−- **Radar output**: [`radar`](https://github.com/urbit/urbit/blob/2c412d658a74dca3b3f584b92f8b390f38a05ba1/pkg/arvo/app/radar.hoon) is a program run from within an Urbit
56− ship that pings (`|hi`) every Urbit ship and records the results. This data is exposed in a single JSON file that is added to over time as the script runs.
57−
58−Most of the required information in the Network Explorer will come from Azimuth.
59−It's important to know though that the source of PKI information will likely
60−change in the near future in light of [ongoing work to reduce the
61−cost](https://groups.google.com/a/urbit.org/g/dev/c/p6rP_WsxLS0) of bringing
62−nodes onto the network.
63−
64−The below [entity-relationship
65−diagram](https://en.wikipedia.org/wiki/Entity%E2%80%93relationship_model)
66−defines a data model in terms of a relational database:
67−
68−
69−
70−Some notes about the above diagram:
71−
72−- The data model should be _agnostic to any particular PKI storage
73− infrastructure_ to future-proof it against PKI changes.
74−- For non-`ping` data, it should be possible to compute the table of `node` from
75− a time-ordered traversal of `pki_event` (kind of like Arvo!). This means that
76− many fields listed on the `node` table could instead be derived from analysis
77− of the `pki_event` table. The implementer will be responsible for determining
78− which fields should be represented on the `node` directly and which should be
79− determined from `pki_event` for performance purposes.
80−- We don't have data for comets or moons, because they're not registered on
81− Azimuth and `radar` only checks for activity from ships with keys that have
82− been set on Azimuth. There will likely be a future in which we'll have a way
83− to provide data about these ship types though.
84−
85−### Data Ingestion
86−
87−Data from Azimuth and `radar` will need to be regularly extracted, parsed, and
88−persisted to the local database for querying.
89−
90−For Azimuth, using a tool like [infura](https://infura.io) should be used.. Tlon
91−currently uses infura, so it should be possible to use a free account for
92−development and Tlon's pre-existing service in production.
93−
94−The output of the `radar` script is saved to a JSON file that can be found
95−[here](http://35.247.74.19:8080/~radar.json). This file is updated as new ping
96−information is obtained, so data ingestion from this service should happen on
97−some regular schedule.
98−
99−### API
100−
101−The following are are necessary API endpoints for the Network Explorer UI to
102−function properly. It's likely that some details have been missed, so apply your
103−own judgement when designing the API.
104−
105−This interface is meant to be agnostic to the API mechanism you choose, which
106−could be either REST or GraphQL.
107−
108−#### get-node
109−
110−Produce the details for a single node.
111−
112−**Parameters**
113−
114−- `urbit_id`: `@p`
115−
116−**Returns**
117−
118−- `urbit_id`: `@p`
119−- `node_type`: `#{galaxy,star,planet,moon,comet}`
120−- `status`: `#{locked,unlocked,spawned,activated,online}`
121−- `continuity`: `int`, current continuity
122−- `key_revision`: `int`, current revision of keys
123−- `num_owners`: `int`, number of owners
124−- `sponsors`: `[node]` (sponsor and sponsor's sponsor, if applicable)
125−- `kids`: `[node]` (nodes sponsored by this one)
126−- `proxy_addresses`:
127− - `ownership`
128− - `spawn`
129− - `transfer`
130− - `voting`
131− - `management`
132−
133−#### get-nodes
134−
135−Produce a list of nodes, optionally with parameters. Absence of parameters queries all nodes.
136−
137−**Parameters**
138−
139−- `q`: `string` (optional: query term to search for)
140−- `node_types`: `[node_type]` (optional: produce just events for specific node types)
141−- `limit`: `int` (number of results to return)
142−- `offset`: `int` (number of results to offset by)
143−
144−**Returns**
145−An array of `node` in ascending order by `point`.
146−
147−#### get-pki-events
148−
149−Produce the stream of PKI events for a specific Urbit ID, or globally in descending order of `time`.
150−
151−**Parameters**
152−
153−- `urbit_id`: `@p` (optional: unspecified implies global)
154−- `since`: `timestamp` (optional: unspecified implies all events)
155−- `node_types`: `[node_type]` (optional: produce just events for specific node types)
156−- `limit`: `int` (number of results to return)
157−- `offset`: `int` (number of results to offset by\*\*
158−
159−**Returns**
160−An array of `pki_event` in descending order of `time`.
161−
162−#### get-activity
163−
164−Produce the network activity of a node or nodes.
165−
166−**Parameters**
167−
168−- `urbit_id`: `@p` (optional: specific ship to pull activity for)
169−- `since`: `timestamp` (optional: activity since provided time)
170−- `until`: `timestamp` (optional: activity up until the provided time)
171−
172−If `urbit_id` is unspecified, implies all ships.
173−
174−If `since` and `until` are unspecified, produces all activity for all time. If
175−only `since` is provided, produces activity from `since` to the present. If only
176−`until` is provided, produces activity for all of time up until `until`. If both
177−are provided, queries activity within a range.
178−
179−**Returns**
180−Array of:
181−
182−- `urbit_id`: `@p`
183−- `date`: `date` (no time)
184−- `active`: `boolean` (active or inactive)
185−
186−### Technology Requirements/Guidelines
187−
188−That which is not explicitly specified here is up to the choice of the
189−implementer.
190−
191−- **Database**: PostgreSQL
192−- **Ethereum Interaction**: Infura
193−- **HTTP API methodology**: GraphQL or REST
194−- **ORM**: None. Implementer should be sufficiently familiar with SQL.
195−- **API Hosting**: The API will be hosted on `api.network.urbit.org` or
196− `network.urbit.org/api`.
197−
198−## Requirements
199−
200−- 2+ years of experience with developing HTTP APIs
201−- 2+ years of experience working with relational databases
202−- Code hosted at https://github.com/urbit/network-api
203−- Attend regular (weekly or bi-weekly) checkin meetings with the director of
204− urbit.org to provide status updates
205−
206−## Resources
207−
208−- A complete set of high-fidelity designs in Figma will be made available to the
209− worker.
210−- The director of urbit.org will ensure access to other resources or personnel
211− as needed.
212−
213−## Milestones
214−
215−### Data ETL
216−
217−2 stars
218−The database has been designed and data from both sources is ingested
219−(extracted, transformed and loaded) on a regular schedule. Analysis can be
220−performed at the database layer.
221−
222−### HTTP API
223−
224−2 stars
225−An HTTP API is written per the above specification and deployed to
226−`https://api.network.urbit.org`.
227−
228−### UI Integration
229−
230−1 stars
231−The Network Explorer UI will be a separate proposal, but integration with an
232−interface inevitably surfaces oversights in the original design. An additional
233−star will be awarded for providing ongoing maintenance as-needed during
234−front-end integration.