urbit/ARCH.md

# Urbit Bitcoin Architecture

## Intro
Urbit Bitcoin allows selected Urbit ships to inject an outside resource, a Bitcoin full node, into Urbit as a service.

This architecture is, by Urbit standards, odd. The oddness arises mainly from the asymmetry of full nodes: only a few nodes are providers/full nodes, and they have to keep remote clients updated as to the state of the blockchain. The system also requires providers to run a node side-by-side with their Urbit, although this can mostly be abstracted away as HTTP calls out.

My goal in designing this was to isolate the architecture's awkwardness as much as possible to specific chokepoints, and to keep the non-provider portions as clean state machine primitives.

## System Components
### Outside Dependencies (for `btc-provider`)
These dependencies only apply to a provider running a full node with the `btc-provider` agent.
- Fully sync'd Bitcoin full node with running RPC. Be sure to have settings:
```
server=1
rpcallowip=127.0.0.1
rpcport=8332
```
- Fully sync'd ElectRS
- Custom HTTP API proxy. This is what `btc-provider` calls. It is necessary because ElectRS does not accept HTTP calls, and its API endpoints chain several RPC calls into one for convenience. It also abstracts out the multiple Bitcoin/ElectRS RPCs.

### Gall Agents
- `btc-wallet-store`: holds wallets and watches their addresses
  * tracks whether a wallet has been scanned
  * generates receiving addresses and change addresses
  * can take address state input from any agent on its own ship
- `btc-wallet-hook`: requests BTC state from provider and forwards it
  * subscribes to wallet-store for any address requests.
  * pokes wallet-store with new address info
- `btc-provider`: 
- helper BTC libraries for address and transaction generation.

## Address Watching Logic

### in `btc-wallet-store`
Every time that `btc-wallet-store`
- receives a new address in `update-address` or
- generates an address
it runs the following logic:
1. Is the address unused (no prior history)? If yes, request info for it again. This is always true for newly generated addresses.
2. Does the address have any UTXOs with fewer than `confs` (the wallet variable for confs required, default=6). If ys, request info for it.
3. If neither true (it's used and UTXOs are all confirmed), do nothing.

### in `btc-wallet-hook`
On receiving an `%address-info` request from the store:
- is the `last-block` older than the most recent block I've seen? 
  * If yes, send `%address-info` request to the provider and add to `reqs` (watchlist)
  * If no, just add to `reqs`

When provider sends a new status update:
- are we now connected and were previously disconnected?
  * if yes, retry all reqs and tx information requests
- is the latest block newer than our previous one?
  * if yes, retry all older `reqs`

## btc-wallet-store
Intentionally very limited in function. It's a primitive for tracking wallet state, including available addresses an existing/watched addresses.

Addresses are put in a watchlist if they have UTXOs *or* have been previously used. "Used" means either existing in transactions already, or having been generated by the store's `gen-address` gate, which generates and watches a new address, and increments the next "free" address.

You add a wallet to the store by adding that wallet's xpub. The store scans that xpub's change and non-change addresses in batches, by sending the address batches to its subscribers and taking pokes back with info for each address.  The scan is done when store sees `max-gap` consecutive unused addresses.

Any source on the ship can poke the store with address info. This allows the possibility of creating import programs for pre-existing wallet data, or large amounts of wallet data. Currently, the only program that interacts with it is `btc-wallet-hook`.

Incoming data:
- requested address info
- unsolicited address updates (checked against watch list)
- requests to generate and watch new addresses

Outgoing data:
- requests for address info on unscanned address batches (sends to each new subscriber on /requests)
- newly generated/watched addresses

## btc-wallet-hook
I don't like the name "hook" here, but can't think of anything better atm. It's closer to a non-wallet-state manager on top of the wallet-store; potentially just one of many.

Incoming data:
- responses from `btc-provider`
- connectivity status from `btc-provider`
- address lookup requests from `btc-wallet-store`
- newly generated/watched addresses from `btc-wallet-store`

Outgoing data:
- pokes `btc-wallet-store` with address updates
- pokes `btc-wallet-store` with address generation requests
- pokes `btc-provider` with address lookup requests

Error conditions:
Disconnected provider: when it receives a message that this is the case, it stops sending outgoing address info requests until the provider says it's back up. Once we receive a connected message, all pending requests are retried.

## btc-provider
Layers on top of both BTC and ElectRS *TODO explain why the latter*

Incoming data:

Outgoing data:

Error conditions:

## Resource Usage

### Provider
- machine: requires hard drive and processing to run a BTC full node and ElectRS indexer.
- network: sends out *all* addresses in each new block to *all* clients

### Wallet
- machine: processes all addresses for each block to see whether they are being watched.
- network: receives all addresses for each block

## Needed Extensions
- Invoice generator that asks for addresses on behalf of ships and tracks whether they've made payments. `wallet-hook` could probably be renamed to `wallet-manager` and extended for this purpose.
- `btc-provider` should push out address state in each block
- `btc-wallet-store` should watch the next ~20 addresses in each wallet account, so that it can detect BTC sent to the wallet if the wallet is also manageed/generates addresses in an outside-Urbit program.
- use Branch-and-Bound UTXO selection algo

## Possible Improvements/Changes
- Do away with `btc-wallet-hook` altogether in its current form, and instead make `btc-provider` both a server (as it is now) and also a client. Pros: less between-agent data. Cons: complicates the otherwise simple provider module. PrA better solution might be to split just the connectivity parts of `btc-wallet-hook` into a local provider
- Multiple Providers
- Gossip network for both pulling and pushing address updates to lower network usage on the providers.