@@ -1,233 +0,0 @@
1−+++
2−title = "Network Explorer: Data Pipeline and HTTP API"
3−date = "2021-03-11"
4−
5−[taxonomies]
6−grant_type = [ "Bounty" ]
7−grant_category = [ "App Dev: Other" ]
8−
9−[extra]
10−image = ""
11−description = "Build a public, open HTTP API for exploring metrics about the Urbit network."
12−reward = 5
13−assignee = "Christopher King"
14−id = ""
15−completed = false
16−link = ""
17−+++
18−
19−## Overview
20−
21−
22−
23−### Goal
24−
25−Build a public, open HTTP API for exploring metrics about the Urbit network.
26−
27−### Background
28−
29−The project known as the **Network Explorer** has been in the works at Tlon for
30−quite some time. The idea is to build a publicly-accessible tool (hosted on
31−https://network.urbit.org) that provides key metrics and insights into the state
32−of the Urbit network. A series of high-fidelity designs have been produced.
33−
34−### Scope
35−
36−This bounty covers the creation of the:
37−
38−- **data pipeline**: we're concerned with data from the PKI (Azimuth) and
39− the Urbit network itself. These two datasets will need to be aggregated and
40− indexed for efficient querying.
41−- **HTTP API**: the interface for querying data from a client, like the Network
42− Explorer.
43−
44−The Network Explorer interface will be covered in a separate bounty.
45−
46−## Implementation
47−
48−### Data Model
49−
50−Let's start with the data model. There are two sources of data that we need:
51−
52−- **PKI (Azimuth)**: The public-key infrastructure, currently known as Azimuth,
53− is our primary source of data. This information is currently found on Ethereum
54−- **Radar output**: [`radar`](https://github.com/urbit/urbit/blob/2c412d658a74dca3b3f584b92f8b390f38a05ba1/pkg/arvo/app/radar.hoon) is a program run from within an Urbit
55− ship that pings (`|hi`) every Urbit ship and records the results. This data is exposed in a single JSON file that is added to over time as the script runs.
56−
57−Most of the required information in the Network Explorer will come from Azimuth.
58−It's important to know though that the source of PKI information will likely
59−change in the near future in light of [ongoing work to reduce the
60−cost](https://groups.google.com/a/urbit.org/g/dev/c/p6rP_WsxLS0) of bringing
61−nodes onto the network.
62−
63−The below [entity-relationship
64−diagram](https://en.wikipedia.org/wiki/Entity%E2%80%93relationship_model)
65−defines a data model in terms of a relational database:
66−
67−
68−
69−Some notes about the above diagram:
70−
71−- The data model should be _agnostic to any particular PKI storage
72− infrastructure_ to future-proof it against PKI changes.
73−- For non-`ping` data, it should be possible to compute the table of `node` from
74− a time-ordered traversal of `pki_event` (kind of like Arvo!). This means that
75− many fields listed on the `node` table could instead be derived from analysis
76− of the `pki_event` table. The implementer will be responsible for determining
77− which fields should be represented on the `node` directly and which should be
78− determined from `pki_event` for performance purposes.
79−- We don't have data for comets or moons, because they're not registered on
80− Azimuth and `radar` only checks for activity from ships with keys that have
81− been set on Azimuth. There will likely be a future in which we'll have a way
82− to provide data about these ship types though.
83−
84−### Data Ingestion
85−
86−Data from Azimuth and `radar` will need to be regularly extracted, parsed, and
87−persisted to the local database for querying.
88−
89−For Azimuth, using a tool like [infura](https://infura.io) should be used.. Tlon
90−currently uses infura, so it should be possible to use a free account for
91−development and Tlon's pre-existing service in production.
92−
93−The output of the `radar` script is saved to a JSON file that can be found
94−[here](http://35.247.74.19:8080/~radar.json). This file is updated as new ping
95−information is obtained, so data ingestion from this service should happen on
96−some regular schedule.
97−
98−### API
99−
100−The following are are necessary API endpoints for the Network Explorer UI to
101−function properly. It's likely that some details have been missed, so apply your
102−own judgement when designing the API.
103−
104−This interface is meant to be agnostic to the API mechanism you choose, which
105−could be either REST or GraphQL.
106−
107−#### get-node
108−
109−Produce the details for a single node.
110−
111−**Parameters**
112−
113−- `urbit_id`: `@p`
114−
115−**Returns**
116−
117−- `urbit_id`: `@p`
118−- `node_type`: `#{galaxy,star,planet,moon,comet}`
119−- `status`: `#{locked,unlocked,spawned,activated,online}`
120−- `continuity`: `int`, current continuity
121−- `key_revision`: `int`, current revision of keys
122−- `num_owners`: `int`, number of owners
123−- `sponsors`: `[node]` (sponsor and sponsor's sponsor, if applicable)
124−- `kids`: `[node]` (nodes sponsored by this one)
125−- `proxy_addresses`:
126− - `ownership`
127− - `spawn`
128− - `transfer`
129− - `voting`
130− - `management`
131−
132−#### get-nodes
133−
134−Produce a list of nodes, optionally with parameters. Absence of parameters queries all nodes.
135−
136−**Parameters**
137−
138−- `q`: `string` (optional: query term to search for)
139−- `node_types`: `[node_type]` (optional: produce just events for specific node types)
140−- `limit`: `int` (number of results to return)
141−- `offset`: `int` (number of results to offset by)
142−
143−**Returns**
144−An array of `node` in ascending order by `point`.
145−
146−#### get-pki-events
147−
148−Produce the stream of PKI events for a specific Urbit ID, or globally in descending order of `time`.
149−
150−**Parameters**
151−
152−- `urbit_id`: `@p` (optional: unspecified implies global)
153−- `since`: `timestamp` (optional: unspecified implies all events)
154−- `node_types`: `[node_type]` (optional: produce just events for specific node types)
155−- `limit`: `int` (number of results to return)
156−- `offset`: `int` (number of results to offset by\*\*
157−
158−**Returns**
159−An array of `pki_event` in descending order of `time`.
160−
161−#### get-activity
162−
163−Produce the network activity of a node or nodes.
164−
165−**Parameters**
166−
167−- `urbit_id`: `@p` (optional: specific ship to pull activity for)
168−- `since`: `timestamp` (optional: activity since provided time)
169−- `until`: `timestamp` (optional: activity up until the provided time)
170−
171−If `urbit_id` is unspecified, implies all ships.
172−
173−If `since` and `until` are unspecified, produces all activity for all time. If
174−only `since` is provided, produces activity from `since` to the present. If only
175−`until` is provided, produces activity for all of time up until `until`. If both
176−are provided, queries activity within a range.
177−
178−**Returns**
179−Array of:
180−
181−- `urbit_id`: `@p`
182−- `date`: `date` (no time)
183−- `active`: `boolean` (active or inactive)
184−
185−### Technology Requirements/Guidelines
186−
187−That which is not explicitly specified here is up to the choice of the
188−implementer.
189−
190−- **Database**: PostgreSQL
191−- **Ethereum Interaction**: Infura
192−- **HTTP API methodology**: GraphQL or REST
193−- **ORM**: None. Implementer should be sufficiently familiar with SQL.
194−- **API Hosting**: The API will be hosted on `api.network.urbit.org` or
195− `network.urbit.org/api`.
196−
197−## Requirements
198−
199−- 2+ years of experience with developing HTTP APIs
200−- 2+ years of experience working with relational databases
201−- Code hosted at https://github.com/urbit/network-api
202−- Attend regular (weekly or bi-weekly) checkin meetings with the director of
203− urbit.org to provide status updates
204−
205−## Resources
206−
207−- A complete set of high-fidelity designs in Figma will be made available to the
208− worker.
209−- The director of urbit.org will ensure access to other resources or personnel
210− as needed.
211−
212−## Milestones
213−
214−### Data ETL
215−
216−2 stars
217−The database has been designed and data from both sources is ingested
218−(extracted, transformed and loaded) on a regular schedule. Analysis can be
219−performed at the database layer.
220−
221−### HTTP API
222−
223−2 stars
224−An HTTP API is written per the above specification and deployed to
225−`https://api.network.urbit.org`.
226−
227−### UI Integration
228−
229−1 star
230−The Network Explorer UI will be a separate proposal, but integration with an
231−interface inevitably surfaces oversights in the original design. An additional
232−star will be awarded for providing ongoing maintenance as-needed during
233−front-end integration.