monolith works e2e

This commit is contained in:
timlucmiptev 2021-02-14 14:17:58 +04:00 committed by ixv
parent 1521ec8393
commit a191d8fb8b
7 changed files with 81 additions and 181 deletions

120
ARCH.md
View File

@ -1,120 +0,0 @@
# Urbit Bitcoin Architecture
## Intro
Urbit Bitcoin allows selected Urbit ships to inject an outside resource, a Bitcoin full node, into Urbit as a service.
This architecture is, by Urbit standards, odd. The oddness arises mainly from the asymmetry of full nodes: only a few nodes are providers/full nodes, and they have to keep remote clients updated as to the state of the blockchain. The system also requires providers to run a node side-by-side with their Urbit, although this can mostly be abstracted away as HTTP calls out.
My goal in designing this was to isolate the architecture's awkwardness as much as possible to specific chokepoints, and to keep the non-provider portions as clean state machine primitives.
## System Components
### Outside Dependencies (for `btc-provider`)
These dependencies only apply to a provider running a full node with the `btc-provider` agent.
- Fully sync'd Bitcoin full node with running RPC. Be sure to have settings:
```
server=1
rpcallowip=127.0.0.1
rpcport=8332
```
- Fully sync'd ElectRS
- Custom HTTP API proxy. This is what `btc-provider` calls. It is necessary because ElectRS does not accept HTTP calls, and its API endpoints chain several RPC calls into one for convenience. It also abstracts out the multiple Bitcoin/ElectRS RPCs.
### Gall Agents
- `btc-wallet-store`: holds wallets and watches their addresses
* tracks whether a wallet has been scanned
* generates receiving addresses and change addresses
* can take address state input from any agent on its own ship
- `btc-wallet-hook`: requests BTC state from provider and forwards it
* subscribes to wallet-store for any address requests.
* pokes wallet-store with new address info
- `btc-provider`:
- helper BTC libraries for address and transaction generation.
## Address Watching Logic
### in `btc-wallet-store`
Every time that `btc-wallet-store`
- receives a new address in `update-address` or
- generates an address
it runs the following logic:
1. Is the address unused (no prior history)? If yes, request info for it again. This is always true for newly generated addresses.
2. Does the address have any UTXOs with fewer than `confs` (the wallet variable for confs required, default=6). If ys, request info for it.
3. If neither true (it's used and UTXOs are all confirmed), do nothing.
### in `btc-wallet-hook`
On receiving an `%address-info` request from the store:
- is the `last-block` older than the most recent block I've seen?
* If yes, send `%address-info` request to the provider and add to `reqs` (watchlist)
* If no, just add to `reqs`
When provider sends a new status update:
- are we now connected and were previously disconnected?
* if yes, retry all reqs and tx information requests
- is the latest block newer than our previous one?
* if yes, retry all older `reqs`
## btc-wallet-store
Intentionally very limited in function. It's a primitive for tracking wallet state, including available addresses an existing/watched addresses.
Addresses are put in a watchlist if they have UTXOs *or* have been previously used. "Used" means either existing in transactions already, or having been generated by the store's `gen-address` gate, which generates and watches a new address, and increments the next "free" address.
You add a wallet to the store by adding that wallet's xpub. The store scans that xpub's change and non-change addresses in batches, by sending the address batches to its subscribers and taking pokes back with info for each address. The scan is done when store sees `max-gap` consecutive unused addresses.
Any source on the ship can poke the store with address info. This allows the possibility of creating import programs for pre-existing wallet data, or large amounts of wallet data. Currently, the only program that interacts with it is `btc-wallet-hook`.
Incoming data:
- requested address info
- unsolicited address updates (checked against watch list)
- requests to generate and watch new addresses
Outgoing data:
- requests for address info on unscanned address batches (sends to each new subscriber on /requests)
- newly generated/watched addresses
## btc-wallet-hook
I don't like the name "hook" here, but can't think of anything better atm. It's closer to a non-wallet-state manager on top of the wallet-store; potentially just one of many.
Incoming data:
- responses from `btc-provider`
- connectivity status from `btc-provider`
- address lookup requests from `btc-wallet-store`
- newly generated/watched addresses from `btc-wallet-store`
Outgoing data:
- pokes `btc-wallet-store` with address updates
- pokes `btc-wallet-store` with address generation requests
- pokes `btc-provider` with address lookup requests
Error conditions:
Disconnected provider: when it receives a message that this is the case, it stops sending outgoing address info requests until the provider says it's back up. Once we receive a connected message, all pending requests are retried.
## btc-provider
Layers on top of both BTC and ElectRS *TODO explain why the latter*
Incoming data:
Outgoing data:
Error conditions:
## Resource Usage
### Provider
- machine: requires hard drive and processing to run a BTC full node and ElectRS indexer.
- network: sends out *all* addresses in each new block to *all* clients
### Wallet
- machine: processes all addresses for each block to see whether they are being watched.
- network: receives all addresses for each block
## Needed Extensions
- Invoice generator that asks for addresses on behalf of ships and tracks whether they've made payments. `wallet-hook` could probably be renamed to `wallet-manager` and extended for this purpose.
- `btc-provider` should push out address state in each block
- `btc-wallet-store` should watch the next ~20 addresses in each wallet account, so that it can detect BTC sent to the wallet if the wallet is also manageed/generates addresses in an outside-Urbit program.
- use Branch-and-Bound UTXO selection algo
## Possible Improvements/Changes
- Do away with `btc-wallet-hook` altogether in its current form, and instead make `btc-provider` both a server (as it is now) and also a client. Pros: less between-agent data. Cons: complicates the otherwise simple provider module. PrA better solution might be to split just the connectivity parts of `btc-wallet-hook` into a local provider
- Multiple Providers
- Gossip network for both pulling and pushing address updates to lower network usage on the providers.

30
DEMO.md
View File

@ -11,12 +11,14 @@ Runs the full node API services.
## Start Agents and set XPUBs
On `~zod`. Uses "abandon abandon..." mnemonic
```
=network %main
=network %testnet
|commit %home
|start %btc-provider
|start %btc-wallet
:btc-provider|command [%set-credentials api-url='http://localhost:50002' %main]
:btc-wallet|command [%set-provider ~zod %main]
:btc-provider|command [%set-credentials api-url='http://localhost:50002' network]
:btc-wallet|command [%set-provider ~zod network]
:btc-provider|command [%add-whitelist %users `(set ship)`(sy ~[~dopzod])]
=fprint [%4 0xbeef.dead]
@ -26,10 +28,12 @@ On `~zod`. Uses "abandon abandon..." mnemonic
On `~dopzod`. Uses "absurd sick..." mnemonic from PRIVATE.scratch.md
```
=network %main
=network %testnet
|commit %home
|start %btc-wallet
:btc-wallet|command [%set-provider ~zod %main]
:btc-wallet|command [%set-provider ~zod network]
=fprint [%4 0xdead.beef]
=xpubmain 'zpub6r8dKyWJ31XF6n69KKeEwLjVC5ruqAbiJ4QCqLsrV36Mvx9WEjUaiPNPGFLHNCCqgCdy6iZC8ZgHsm6a1AUTVBMVbKGemNcWFcwBGSjJKbD'
@ -38,10 +42,12 @@ On `~dopzod`. Uses "absurd sick..." mnemonic from PRIVATE.scratch.md
### Add Wallets
On both `~zod`/`dopzod`, choose depending on whether you're on test or main
```
:btc-wallet|command [%add-wallet xpubmain fprint ~ [~ 8] [~ 6]]
:btc-wallet|command [%add-wallet xpubtest fprint ~ [~ 8] [~ 6]]
Using 1 confirmation for testing.
```
:btc-wallet|command [%add-wallet xpubmain fprint ~ [~ 8] [~ 1]]
:btc-wallet|command [%add-wallet xpubtest fprint ~ [~ 8] [~ 1]]
```
## Check Balance
@ -57,33 +63,33 @@ On both `~zod`/`dopzod`, choose depending on whether you're on test or main
`~dopzod`
```
:btc-wallet-hook|command [%req-pay-address ~zod 30.000 feyb=10
:btc-wallet|command [%req-pay-address ~zod 80.000 feyb=100]
```
### Check State on ~zod/~dopzod
`~dopzod`: outgoing
```
:btc-wallet-hook +dbug [%state 'poym']
:btc-wallet +dbug [%state 'poym']
```
`~zod`: incoming
```
:btc-wallet-hook +dbug [%state 'piym']
:btc-wallet +dbug [%state 'piym']
```
### Idempotent
`~dopzod`
```
:btc-wallet-hook|command [%req-pay-address ~zod 3.000 feyb=100
:btc-wallet|command [%req-pay-address ~zod 3.000 feyb=100]
```
Or can change amount:
```
:btc-wallet-hook|command [%req-pay-address ~zod 3.000 feyb=100
:btc-wallet|command [%req-pay-address ~zod 3.000 feyb=100]
```
### Broadcast the Signed TX
```
:btc-wallet-hook|command [%broadcast-tx tx]
:btc-wallet|command [%broadcast-tx tx]
```

View File

@ -35,7 +35,6 @@
feybs=(map ship sats)
=piym
=poym
=pend-piym
==
::
+$ card card:agent:gall
@ -68,7 +67,6 @@
*(map ship sats)
*^piym
*^poym
*^pend-piym
==
==
++ on-save
@ -82,7 +80,6 @@
++ on-poke
|= [=mark =vase]
^- (quip card _this)
?> (team:title our.bowl src.bowl)
=^ cards state
?+ mark (on-poke:def mark vase)
%btc-wallet-action
@ -135,6 +132,7 @@
++ handle-command
|= comm=command
^- (quip card _state)
?> =(our.bowl src.bowl)
?- -.comm
%set-provider
=* sub-card
@ -146,16 +144,16 @@
==
::
%set-current-wallet
?~ (find ~[xpub.comm] scanned-wallets)
`state
`state(curr-xpub `xpub.comm)
(set-curr-xpub xpub.comm)
::
%add-wallet
?~ (~(has by walts) xpub.comm)
((slog ~[leaf+"xpub already in wallet"]) `state)
=/ w=walt (from-xpub +.comm)
=. walts (~(put by walts) xpub.comm w)
(init-batches xpub.comm (dec max-gap.w))
=^ c1 state (init-batches xpub.comm (dec max-gap.w))
=^ c2 state (set-curr-xpub xpub.comm)
[(weld c1 c2) state]
::
%delete-wallet
=* cw curr-xpub.state
@ -171,7 +169,7 @@
?< ?=(%pawn (clan:title payee.comm))
?< is-broadcasting
:_ state(poym ~, feybs (~(put by feybs) payee.comm feyb.comm))
~[(poke-us payee.comm [%gen-pay-address value.comm])]
~[(poke-us payee.comm [%gen-pay-address value.comm])]
::
%broadcast-tx
?> =(src.bowl our.bowl)
@ -211,8 +209,8 @@
?> =(src.bowl our.bowl)
=^ cards state
?. included.ti.act
`state
?: (~(has by pend-piym) txid.ti.act)
`state
?: (~(has by pend.piym) txid.ti.act)
(piym-to-history ti.act)
?: (poym-has-txid txid.ti.act)
(poym-to-history ti.act)
@ -253,7 +251,7 @@
?^ cards [cards state]
=+ f=(fam src.bowl)
=+ n=(~(gut by num-fam.piym) f 0)
?~ curr-xpub ~|("btc-wallet-hook: no curr-xpub set" !!)
?~ curr-xpub ~|("btc-walle: no curr-xpub set" !!)
?: (gte n fam-limit.params)
~|("More than {<fam-limit.params>} addresses for moons + planet" !!)
=. state state(num-fam.piym (~(put by num-fam.piym) f +(n)))
@ -284,7 +282,7 @@
:_ state(poym tb)
?~ tb ~
%+ turn txis.u.tb
|=(=txi (poke-provider [%tx-info txid.utxo.txi]))
|=(=txi (poke-provider [%raw-tx txid.utxo.txi]))
::
++ generate-txbu
|= [=xpub payee=(unit ship) feyb=sats txos=(list txo)]
@ -295,7 +293,7 @@
=/ [tb=(unit txbu) chng=(unit sats)]
%~ with-change sut
[u.uw eny.bowl block.btc-state payee feyb txos]
?~ tb ((slog leaf+"insufficient balance") [tb state])
?~ tb ((slog ~[leaf+"insufficient balance or not enough confirmed balance"]) [tb state])
:: if no change, return txbu; else add change output to txbu
::
?~ chng [tb state]
@ -308,7 +306,7 @@
:: %expect-payment
:: - check that payment is in piym
:: - replace pend.payment with incoming txid (lock)
:: - add txid to pend-piym
:: - add txid to pend.piym
:: - request tx-info from provider
::
%expect-payment
@ -330,7 +328,7 @@
==
::
:: +handle-provider-status: handle connectivity updates from provider
:: - retry pend-piym on any %connected event, since we're checking mempool
:: - retry pend.piym on any %connected event, since we're checking mempool
:: - if status is %connected, retry all pending address lookups
:: - only retry all if previously disconnected
:: - if block is updated, retry all address reqs
@ -400,18 +398,21 @@
^- _state
|^
=/ h (~(get by history) txid.ti)
=/ addrs=(set address)
=/ our-addrs=(set address) :: all our addresses in inputs/outputs of tx
%- sy
%+ turn (weld inputs.ti outputs.ti)
|=(=val:tx address.val)
=/ w (first-matching-wallet ~(tap in addrs))
?~ w state
?~ h :: addresses in wallets, but tx not in history
%+ skim
%+ turn (weld inputs.ti outputs.ti)
|=(=val:tx address.val)
is-our-address
?: =(0 ~(wyt in our-addrs)) state
=/ =xpub
xpub.w:(need (address-coords (snag 0 ~(tap in our-addrs)) ~(val by walts)))
?~ h :: addresses in wallets, but tx not in history
=. history
%+ ~(put by history) txid.ti
(mk-hest xpub.u.w addrs)
(mk-hest xpub our-addrs)
state
?. included.ti
?. included.ti :: tx in history, but not in mempool/blocks
state(history (~(del by history) txid.ti))
%_ state
history
@ -421,29 +422,29 @@
::
++ mk-hest
:: has tx-info
|= [=xpub addrs=(set address)]
|= [=xpub our-addrs=(set address)]
^- hest
:* xpub
txid.ti
confs.ti
recvd.ti
(turn inputs.ti |=(v=val:tx (our-ship addrs v)))
(turn outputs.ti |=(v=val:tx (our-ship addrs v)))
(turn inputs.ti |=(v=val:tx (is-our-ship our-addrs v)))
(turn outputs.ti |=(v=val:tx (is-our-ship our-addrs v)))
==
::
++ our-ship
++ is-our-ship
|= [as=(set address:btc) v=val:tx:btc]
^- [=val:tx s=(unit ship)]
[v ?:((~(has in as) address.v) `our.bowl ~)]
::
++ first-matching-wallet
|= addrs=(list address)
^- (unit walt)
|- ?~ addrs ~
=/ ac (address-coords i.addrs ~(val by walts))
?^ ac `w.u.ac
$(addrs t.addrs)
++ is-our-address
|=(a=address ?=(^ (address-coords a ~(val by walts))))
--
++ set-curr-xpub
|= =xpub
^- (quip card _state)
?~ (find ~[xpub] scanned-wallets) `state
`state(curr-xpub `xpub)
::
::
:: Scan Logic
@ -551,7 +552,8 @@
=. scans (~(del by scans) [xpub %0])
=. scans (~(del by scans) [xpub %1])
%- (slog ~[leaf+"Scanned xpub {<xpub>}"])
`state(walts (~(put by walts) xpub w(scanned %.y)))
=. state state(walts (~(put by walts) xpub w(scanned %.y)))
(set-curr-xpub xpub)
:: +check-scan: initiate a scan if one hasn't started
:: check status of scan if one is running
::
@ -626,18 +628,18 @@
$(idx +(idx), txos t.txos)
--
:: +piym-to-history
:: - checks whether txid in pend-piym
:: - checks whether txid in pend.piym
:: - checks whether ti has a matching value output to piym
:: - if no match found, just deletes pend-piym with this tx
:: - if no match found, just deletes pend.piym with this tx
:: stops peer from spamming txids
:: - returns card that adds hest to history
::
++ piym-to-history
|= ti=info:tx
|^ ^- (quip card _state)
=+ pay=(~(get by pend-piym) txid.ti)
=+ pay=(~(get by pend.piym) txid.ti)
?~ pay `state
:: if no matching output in piym, delete from pend-piym to stop DDOS of txids
:: if no matching output in piym, delete from pend.piym to stop DDOS of txids
::
=+ vout=(get-vout value.u.pay)
?~ vout
@ -659,14 +661,14 @@
++ del-pend-piym
|= txid=hexb
^- _state
state(pend-piym (~(del by pend-piym) txid.ti))
state(pend.piym (~(del by pend.piym) txid.ti))
::
++ del-all-piym
|= [txid=hexb payer=ship]
^- _state
=+ nf=(~(gut by num-fam.piym) payer 1)
%= state
pend-piym (~(del by pend-piym) txid)
pend.piym (~(del by pend.piym) txid)
ps.piym (~(del by ps.piym) payer)
num-fam.piym (~(put by num-fam.piym) payer (dec nf))
==
@ -695,17 +697,17 @@
^- ship
?. =(%earl (clan:title s)) s
(sein:title our.bowl now.bowl s)
:: +update-pend-piym
:: +update-pend.piym
:: - set pend.payment to txid (lock)
:: - add txid to pend-piym
:: - add txid to pend.piym
::
++ update-pend-piym
|= [txid=hexb p=payment]
^- _state
?~ pend.p ~|("update-pend-piym: empty pend.payment" !!)
?~ pend.p ~|("update-pend-piym: no pending payment" !!)
%= state
ps.piym (~(put by ps.piym) payer.p p)
pend-piym (~(put by pend-piym) txid p)
pend.piym (~(put by pend.piym) txid p)
==
::
:: +update-poym-txis:
@ -769,7 +771,7 @@
::
++ retry-pend-piym
^- (list card)
%+ turn ~(tap in ~(key by pend-piym))
%+ turn ~(tap in ~(key by pend.piym))
|=(=txid (poke-provider [%tx-info txid]))
::
++ poke-provider

View File

@ -292,7 +292,7 @@
::
++ with-change
^- [tb=(unit txbu) chng=(unit sats)]
=+ tb=select-utxos
=/ tb=(unit txbu) select-utxos
?~ tb [~ ~]
=+ excess=~(fee txb u.tb) :: (inputs - outputs)
=/ new-fee=sats :: cost of this tx + one more output

View File

@ -0,0 +1,12 @@
/- *btc-wallet
|_ act=action
++ grad %noun
++ grow
|%
++ noun act
--
++ grab
|%
++ noun action
--
--

View File

@ -48,6 +48,7 @@ const run = async(network, mnemonics, psbtStr) => {
const validate = await psbt.validateSignaturesOfAllInputs();
await psbt.finalizeAllInputs();
const hex = psbt.extractTransaction().toHex();
console.log(bitcoin.Transaction.fromHex(hex).getId())
console.log({ validate, hex});
}
return;

View File

@ -6,8 +6,7 @@
+$ block @ud
+$ btc-state [=block fee=(unit sats) t=@da]
+$ payment [pend=(unit txid) =xpub =address payer=ship value=sats]
+$ piym [ps=(map ship payment) num-fam=(map ship @ud)]
+$ pend-piym (map txid payment)
+$ piym [ps=(map ship payment) pend=(map txid payment) num-fam=(map ship @ud)]
+$ poym (unit txbu)
::
+$ command