@@ -0,0 +1,227 @@
2+title = "Network Explorer: Data Pipeline and HTTP API"
5+grant_type = ["bounties"]
6+grant_category = ["App Dev: Other"]
9+description = "Build a public, open HTTP API for exploring metrics about the Urbit network."
11+assignee = "Christopher King"
19+
23+Build a public, open HTTP API for exploring metrics about the Urbit network.
27+The project known as the **Network Explorer** has been in the works at Tlon for
28+quite some time. The idea is to build a publicly-accessible tool (hosted on
29+https://network.urbit.org) that provides key metrics and insights into the state
30+of the Urbit network. A series of high-fidelity designs have been produced.
34+This bounty covers the creation of the:
36+- **data pipeline**: we're concerned with data from the PKI (Azimuth) and
37+ the Urbit network itself. These two datasets will need to be aggregated and
38+ indexed for efficient querying.
39+- **HTTP API**: the interface for querying data from a client, like the Network
42+The Network Explorer interface will be covered in a separate bounty.
48+Let's start with the data model. There are two sources of data that we need:
50+- **PKI (Azimuth)**: The public-key infrastructure, currently known as Azimuth,
51+ is our primary source of data. This information is currently found on Ethereum
52+- **Radar output**: [`radar`](https://github.com/urbit/urbit/blob/2c412d658a74dca3b3f584b92f8b390f38a05ba1/pkg/arvo/app/radar.hoon) is a program run from within an Urbit
53+ ship that pings (`|hi`) every Urbit ship and records the results. This data is exposed in a single JSON file that is added to over time as the script runs.
55+Most of the required information in the Network Explorer will come from Azimuth.
56+It's important to know though that the source of PKI information will likely
57+change in the near future in light of [ongoing work to reduce the
58+cost](https://groups.google.com/a/urbit.org/g/dev/c/p6rP_WsxLS0) of bringing
59+nodes onto the network.
61+The below [entity-relationship
62+diagram](https://en.wikipedia.org/wiki/Entity%E2%80%93relationship_model)
63+defines a data model in terms of a relational database:
65+
67+Some notes about the above diagram:
69+* The data model should be *agnostic to any particular PKI storage
70+ infrastructure* to future-proof it against PKI changes.
71+* For non-`ping` data, it should be possible to compute the table of `node` from
72+ a time-ordered traversal of `pki_event` (kind of like Arvo!). This means that
73+ many fields listed on the `node` table could instead be derived from analysis
74+ of the `pki_event` table. The implementer will be responsible for determining
75+ which fields should be represented on the `node` directly and which should be
76+ determined from `pki_event` for performance purposes.
77+* We don't have data for comets or moons, because they're not registered on
78+ Azimuth and `radar` only checks for activity from ships with keys that have
79+ been set on Azimuth. There will likely be a future in which we'll have a way
80+ to provide data about these ship types though.
84+Data from Azimuth and `radar` will need to be regularly extracted, parsed, and
85+persisted to the local database for querying.
87+For Azimuth, using a tool like [infura](https://infura.io) should be used.. Tlon
88+currently uses infura, so it should be possible to use a free account for
89+development and Tlon's pre-existing service in production.
91+The output of the `radar` script is saved to a JSON file that can be found
92+[here](http://35.247.74.19:8080/~radar.json). This file is updated as new ping
93+information is obtained, so data ingestion from this service should happen on
94+some regular schedule.
98+The following are are necessary API endpoints for the Network Explorer UI to
99+function properly. It's likely that some details have been missed, so apply your
100+own judgement when designing the API.
102+This interface is meant to be agnostic to the API mechanism you choose, which
103+could be either REST or GraphQL.
107+Produce the details for a single node.
114+- `node_type`: `#{galaxy,star,planet,moon,comet}`
115+- `status`: `#{locked,unlocked,spawned,activated,online}`
116+- `continuity`: `int`, current continuity
117+- `key_revision`: `int`, current revision of keys
118+- `num_owners`: `int`, number of owners
119+- `sponsors`: `[node]` (sponsor and sponsor's sponsor, if applicable)
120+- `kids`: `[node]` (nodes sponsored by this one)
130+Produce a list of nodes, optionally with parameters. Absence of parameters queries all nodes.
133+- `q`: `string` (optional: query term to search for)
134+- `node_types`: `[node_type]` (optional: produce just events for specific node types)
135+- `limit`: `int` (number of results to return)
136+- `offset`: `int` (number of results to offset by)
139+An array of `node` in ascending order by `point`.
143+Produce the stream of PKI events for a specific Urbit ID, or globally in descending order of `time`.
146+- `urbit_id`: `@p` (optional: unspecified implies global)
147+- `since`: `timestamp` (optional: unspecified implies all events)
148+- `node_types`: `[node_type]` (optional: produce just events for specific node types)
149+- `limit`: `int` (number of results to return)
150+- `offset`: `int` (number of results to offset by**
153+An array of `pki_event` in descending order of `time`.
157+Produce the network activity of a node or nodes.
160+- `urbit_id`: `@p` (optional: specific ship to pull activity for)
161+- `since`: `timestamp` (optional: activity since provided time)
162+- `until`: `timestamp` (optional: activity up until the provided time)
164+If `urbit_id` is unspecified, implies all ships.
166+If `since` and `until` are unspecified, produces all activity for all time. If
167+only `since` is provided, produces activity from `since` to the present. If only
168+`until` is provided, produces activity for all of time up until `until`. If both
169+are provided, queries activity within a range.
174+- `date`: `date` (no time)
175+- `active`: `boolean` (active or inactive)
177+### Technology Requirements/Guidelines
179+That which is not explicitly specified here is up to the choice of the
182+* **Database**: PostgreSQL
183+* **Ethereum Interaction**: Infura
184+* **HTTP API methodology**: GraphQL or REST
185+* **ORM**: None. Implementer should be sufficiently familiar with SQL.
186+* **API Hosting**: The API will be hosted on `api.network.urbit.org` or
187+ `network.urbit.org/api`.
191+- 2+ years of experience with developing HTTP APIs
192+- 2+ years of experience working with relational databases
193+- Code hosted at https://github.com/urbit/network-api
194+- Attend regular (weekly or bi-weekly) checkin meetings with the director of
195+ urbit.org to provide status updates
199+- A complete set of high-fidelity designs in Figma will be made available to the
201+- The director of urbit.org will ensure access to other resources or personnel
210+The database has been designed and data from both sources is ingested
211+(extracted, transformed and loaded) on a regular schedule. Analysis can be
212+performed at the database layer.
216+An HTTP API is written per the above specification and deployed to
217+`https://api.network.urbit.org`.
221+The Network Explorer UI will be a separate proposal, but integration with an
222+interface inevitably surfaces oversights in the original design. An additional
223+star will be awarded for providing ongoing maintenance as-needed during
224+front-end integration.