shrub/ARCH.md

Urbit Bitcoin Architecture

Intro

This architecture is, by Urbit standards, awkward. The awkwardness arises mainly from the asymmetry of full nodes: only a few nodes are providers/full nodes, and they have to keep remote clients updated as to the state of the blockchain. The system also requires providers to run a node side-by-side with their Urbit, although this can mostly be abstracted away as HTTP calls out.

My goal in designing this was to isolate the architecture's awkwardness as much as possible to specific chokepoints, and to keep the non-provider portions as clean state machine primitives.

System Components

Outside Dependencies (for btc-provider)

These dependencies only apply to a provider running a full node with the btc-provider agent.

• Fully sync'd Bitcoin full node with running RPC. Be sure to have settings:

server=1
rpcallowip=127.0.0.1
rpcport=8332

• Fully sync'd ElectRS
• Custom HTTP API proxy. This is what btc-provider calls. It is necessary because ElectRS does not accept HTTP calls, and its API endpoints chain several RPC calls into one for convenience. It also abstracts out the multiple Bitcoin/ElectRS RPCs.

Gall Agents

• btc-wallet-store: holds wallets and watches their addresses
  • tracks whether a wallet has been scanned
  • generates receiving addresses and change addresses
  • can take address state input from any agent on its own ship
• btc-wallet-hook: requests BTC state from provider and forwards it
  • subscribes to wallet-store for any address requests.
  • pokes wallet-store with new address info
• btc-provider:
• helper BTC libraries for address and transaction generation.

btc-wallet-store

Intentionally very limited in function. It's a primitive for tracking wallet state, including available addresses an existing/watched addresses.

Addresses are put in a watchlist if they have UTXOs or have been previously used. "Used" means either existing in transactions already, or having been generated by the store's gen-address gate, which generates and watches a new address, and increments the next "free" address.

You add a wallet to the store by adding that wallet's xpub. The store scans that xpub's change and non-change addresses in batches, by sending the address batches to its subscribers and taking pokes back with info for each address. The scan is done when store sees max-gap consecutive unused addresses.

Any source on the ship can poke the store with address info. This allows the possibility of creating import programs for pre-existing wallet data, or large amounts of wallet data. Currently, the only program that interacts with it is btc-wallet-hook.

Incoming data:

• requested address info
• unsolicited address updates (checked against watch list)
• requests to generate and watch new addresses

Outgoing data:

• requests for address info on unscanned address batches (sends to each new subscriber on /requests)
• newly generated/watched addresses

btc-wallet-hook

I don't like the name "hook" here, but can't think of anything better atm. It's closer to a non-wallet-state manager on top of the wallet-store; potentially just one of many.

Incoming data:

• responses from btc-provider
• connectivity status from btc-provider
• address lookup requests from btc-wallet-store
• newly generated/watched addresses from btc-wallet-store

Outgoing data:

• pokes btc-wallet-store with address updates
• pokes btc-wallet-store with address generation requests
• pokes btc-provider with address lookup requests

Error conditions: Disconnected provider: when it receives a message that this is the case, it stops sending outgoing address info requests until the provider says it's back up. Once we receive a connected message, all pending requests are retried.

btc-provider

Layers on top of both BTC and ElectRS TODO explain why the latter

Incoming data:

Outgoing data:

Error conditions:

Resource Usage

Provider

• machine: requires hard drive and processing to run a BTC full node and ElectRS indexer.
• network: sends out all addresses in each new block to all clients

Wallet

• machine: processes all addresses for each block to see whether they are being watched.
• network: receives all addresses for each block

Needed Extensions

• Invoice generator that asks for addresses on behalf of ships and tracks whether they've made payments. wallet-hook could probably be renamed to wallet-manager and extended for this purpose.
• btc-provider should push out address state in each block
• btc-wallet-store should watch the next ~20 addresses in each wallet account, so that it can detect BTC sent to the wallet if the wallet is also managed/generates addresses in an outside-Urbit program.

Possible Improvements/Changes

• Do away with btc-wallet-hook altogether in its current form, and instead make btc-provider both a server (as it is now) and also a client. Pros: less between-agent data. Cons: complicates the otherwise simple provider module. PrA better solution might be to split just the connectivity parts of btc-wallet-hook into a local provider
• Multiple Providers
• Gossip network for both pulling and pushing address updates to lower network usage on the providers.