@@ -0,0 +1,227 @@
1++++
2+title = "Network Explorer: Data Pipeline and HTTP API"
3+date = 2021-03-12
4+[taxonomies]
5+grant_type = ["bounties"]
6+grant_category = ["App Dev: Other"]
7+[extra]
8+image = ""
9+description = "Build a public, open HTTP API for exploring metrics about the Urbit network."
10+reward = 5
11+assignee = "Christopher King"
12+id = ""
13+completed = false
14+link = ""
15++++
16+
17+## Overview
18+
19+
20+
21+### Goal
22+
23+Build a public, open HTTP API for exploring metrics about the Urbit network.
24+
25+### Background
26+
27+The project known as the **Network Explorer** has been in the works at Tlon for
28+quite some time. The idea is to build a publicly-accessible tool (hosted on
29+https://network.urbit.org) that provides key metrics and insights into the state
30+of the Urbit network. A series of high-fidelity designs have been produced.
31+
32+### Scope
33+
34+This bounty covers the creation of the:
35+
36+- **data pipeline**: we're concerned with data from the PKI (Azimuth) and
37+ the Urbit network itself. These two datasets will need to be aggregated and
38+ indexed for efficient querying.
39+- **HTTP API**: the interface for querying data from a client, like the Network
40+ Explorer.
41+
42+The Network Explorer interface will be covered in a separate bounty.
43+
44+## Implementation
45+
46+### Data Model
47+
48+Let's start with the data model. There are two sources of data that we need:
49+
50+- **PKI (Azimuth)**: The public-key infrastructure, currently known as Azimuth,
51+ is our primary source of data. This information is currently found on Ethereum
52+- **Radar output**: [`radar`](https://github.com/urbit/urbit/blob/2c412d658a74dca3b3f584b92f8b390f38a05ba1/pkg/arvo/app/radar.hoon) is a program run from within an Urbit
53+ ship that pings (`|hi`) every Urbit ship and records the results. This data is exposed in a single JSON file that is added to over time as the script runs.
54+
55+Most of the required information in the Network Explorer will come from Azimuth.
56+It's important to know though that the source of PKI information will likely
57+change in the near future in light of [ongoing work to reduce the
58+cost](https://groups.google.com/a/urbit.org/g/dev/c/p6rP_WsxLS0) of bringing
59+nodes onto the network.
60+
61+The below [entity-relationship
62+diagram](https://en.wikipedia.org/wiki/Entity%E2%80%93relationship_model)
63+defines a data model in terms of a relational database:
64+
65+
66+
67+Some notes about the above diagram:
68+
69+* The data model should be *agnostic to any particular PKI storage
70+ infrastructure* to future-proof it against PKI changes.
71+* For non-`ping` data, it should be possible to compute the table of `node` from
72+ a time-ordered traversal of `pki_event` (kind of like Arvo!). This means that
73+ many fields listed on the `node` table could instead be derived from analysis
74+ of the `pki_event` table. The implementer will be responsible for determining
75+ which fields should be represented on the `node` directly and which should be
76+ determined from `pki_event` for performance purposes.
77+* We don't have data for comets or moons, because they're not registered on
78+ Azimuth and `radar` only checks for activity from ships with keys that have
79+ been set on Azimuth. There will likely be a future in which we'll have a way
80+ to provide data about these ship types though.
81+
82+### Data Ingestion
83+
84+Data from Azimuth and `radar` will need to be regularly extracted, parsed, and
85+persisted to the local database for querying.
86+
87+For Azimuth, using a tool like [infura](https://infura.io) should be used.. Tlon
88+currently uses infura, so it should be possible to use a free account for
89+development and Tlon's pre-existing service in production.
90+
91+The output of the `radar` script is saved to a JSON file that can be found
92+[here](http://35.247.74.19:8080/~radar.json). This file is updated as new ping
93+information is obtained, so data ingestion from this service should happen on
94+some regular schedule.
95+
96+### API
97+
98+The following are are necessary API endpoints for the Network Explorer UI to
99+function properly. It's likely that some details have been missed, so apply your
100+own judgement when designing the API.
101+
102+This interface is meant to be agnostic to the API mechanism you choose, which
103+could be either REST or GraphQL.
104+
105+#### get-node
106+
107+Produce the details for a single node.
108+
109+**Parameters**
110+- `urbit_id`: `@p`
111+
112+**Returns**
113+- `urbit_id`: `@p`
114+- `node_type`: `#{galaxy,star,planet,moon,comet}`
115+- `status`: `#{locked,unlocked,spawned,activated,online}`
116+- `continuity`: `int`, current continuity
117+- `key_revision`: `int`, current revision of keys
118+- `num_owners`: `int`, number of owners
119+- `sponsors`: `[node]` (sponsor and sponsor's sponsor, if applicable)
120+- `kids`: `[node]` (nodes sponsored by this one)
121+- `proxy_addresses`:
122+ - `ownership`
123+ - `spawn`
124+ - `transfer`
125+ - `voting`
126+ - `management`
127+
128+#### get-nodes
129+
130+Produce a list of nodes, optionally with parameters. Absence of parameters queries all nodes.
131+
132+**Parameters**
133+- `q`: `string` (optional: query term to search for)
134+- `node_types`: `[node_type]` (optional: produce just events for specific node types)
135+- `limit`: `int` (number of results to return)
136+- `offset`: `int` (number of results to offset by)
137+
138+**Returns**
139+An array of `node` in ascending order by `point`.
140+
141+#### get-pki-events
142+
143+Produce the stream of PKI events for a specific Urbit ID, or globally in descending order of `time`.
144+
145+**Parameters**
146+- `urbit_id`: `@p` (optional: unspecified implies global)
147+- `since`: `timestamp` (optional: unspecified implies all events)
148+- `node_types`: `[node_type]` (optional: produce just events for specific node types)
149+- `limit`: `int` (number of results to return)
150+- `offset`: `int` (number of results to offset by**
151+
152+**Returns**
153+An array of `pki_event` in descending order of `time`.
154+
155+#### get-activity
156+
157+Produce the network activity of a node or nodes.
158+
159+**Parameters**
160+- `urbit_id`: `@p` (optional: specific ship to pull activity for)
161+- `since`: `timestamp` (optional: activity since provided time)
162+- `until`: `timestamp` (optional: activity up until the provided time)
163+
164+If `urbit_id` is unspecified, implies all ships.
165+
166+If `since` and `until` are unspecified, produces all activity for all time. If
167+only `since` is provided, produces activity from `since` to the present. If only
168+`until` is provided, produces activity for all of time up until `until`. If both
169+are provided, queries activity within a range.
170+
171+**Returns**
172+Array of:
173+- `urbit_id`: `@p`
174+- `date`: `date` (no time)
175+- `active`: `boolean` (active or inactive)
176+
177+### Technology Requirements/Guidelines
178+
179+That which is not explicitly specified here is up to the choice of the
180+implementer.
181+
182+* **Database**: PostgreSQL
183+* **Ethereum Interaction**: Infura
184+* **HTTP API methodology**: GraphQL or REST
185+* **ORM**: None. Implementer should be sufficiently familiar with SQL.
186+* **API Hosting**: The API will be hosted on `api.network.urbit.org` or
187+ `network.urbit.org/api`.
188+
189+## Requirements
190+
191+- 2+ years of experience with developing HTTP APIs
192+- 2+ years of experience working with relational databases
193+- Code hosted at https://github.com/urbit/network-api
194+- Attend regular (weekly or bi-weekly) checkin meetings with the director of
195+ urbit.org to provide status updates
196+
197+## Resources
198+
199+- A complete set of high-fidelity designs in Figma will be made available to the
200+ worker.
201+- The director of urbit.org will ensure access to other resources or personnel
202+ as needed.
203+
204+
205+## Milestones
206+
207+
208+### Data ETL
209+2 stars
210+The database has been designed and data from both sources is ingested
211+(extracted, transformed and loaded) on a regular schedule. Analysis can be
212+performed at the database layer.
213+
214+### HTTP API
215+2 stars
216+An HTTP API is written per the above specification and deployed to
217+`https://api.network.urbit.org`.
218+
219+### UI Integration
220+1 star
221+The Network Explorer UI will be a separate proposal, but integration with an
222+interface inevitably surfaces oversights in the original design. An additional
223+star will be awarded for providing ongoing maintenance as-needed during
224+front-end integration.
225+
226+
227+