diff --git a/ARCH.md b/ARCH.md deleted file mode 100644 index d3507e6ab..000000000 --- a/ARCH.md +++ /dev/null @@ -1,120 +0,0 @@ -# Urbit Bitcoin Architecture - -## Intro -Urbit Bitcoin allows selected Urbit ships to inject an outside resource, a Bitcoin full node, into Urbit as a service. - -This architecture is, by Urbit standards, odd. The oddness arises mainly from the asymmetry of full nodes: only a few nodes are providers/full nodes, and they have to keep remote clients updated as to the state of the blockchain. The system also requires providers to run a node side-by-side with their Urbit, although this can mostly be abstracted away as HTTP calls out. - -My goal in designing this was to isolate the architecture's awkwardness as much as possible to specific chokepoints, and to keep the non-provider portions as clean state machine primitives. - -## System Components -### Outside Dependencies (for `btc-provider`) -These dependencies only apply to a provider running a full node with the `btc-provider` agent. -- Fully sync'd Bitcoin full node with running RPC. Be sure to have settings: -``` -server=1 -rpcallowip=127.0.0.1 -rpcport=8332 -``` -- Fully sync'd ElectRS -- Custom HTTP API proxy. This is what `btc-provider` calls. It is necessary because ElectRS does not accept HTTP calls, and its API endpoints chain several RPC calls into one for convenience. It also abstracts out the multiple Bitcoin/ElectRS RPCs. - -### Gall Agents -- `btc-wallet-store`: holds wallets and watches their addresses - * tracks whether a wallet has been scanned - * generates receiving addresses and change addresses - * can take address state input from any agent on its own ship -- `btc-wallet-hook`: requests BTC state from provider and forwards it - * subscribes to wallet-store for any address requests. - * pokes wallet-store with new address info -- `btc-provider`: -- helper BTC libraries for address and transaction generation. - -## Address Watching Logic - -### in `btc-wallet-store` -Every time that `btc-wallet-store` -- receives a new address in `update-address` or -- generates an address -it runs the following logic: -1. Is the address unused (no prior history)? If yes, request info for it again. This is always true for newly generated addresses. -2. Does the address have any UTXOs with fewer than `confs` (the wallet variable for confs required, default=6). If ys, request info for it. -3. If neither true (it's used and UTXOs are all confirmed), do nothing. - -### in `btc-wallet-hook` -On receiving an `%address-info` request from the store: -- is the `last-block` older than the most recent block I've seen? - * If yes, send `%address-info` request to the provider and add to `reqs` (watchlist) - * If no, just add to `reqs` - -When provider sends a new status update: -- are we now connected and were previously disconnected? - * if yes, retry all reqs and tx information requests -- is the latest block newer than our previous one? - * if yes, retry all older `reqs` - -## btc-wallet-store -Intentionally very limited in function. It's a primitive for tracking wallet state, including available addresses an existing/watched addresses. - -Addresses are put in a watchlist if they have UTXOs *or* have been previously used. "Used" means either existing in transactions already, or having been generated by the store's `gen-address` gate, which generates and watches a new address, and increments the next "free" address. - -You add a wallet to the store by adding that wallet's xpub. The store scans that xpub's change and non-change addresses in batches, by sending the address batches to its subscribers and taking pokes back with info for each address. The scan is done when store sees `max-gap` consecutive unused addresses. - -Any source on the ship can poke the store with address info. This allows the possibility of creating import programs for pre-existing wallet data, or large amounts of wallet data. Currently, the only program that interacts with it is `btc-wallet-hook`. - -Incoming data: -- requested address info -- unsolicited address updates (checked against watch list) -- requests to generate and watch new addresses - -Outgoing data: -- requests for address info on unscanned address batches (sends to each new subscriber on /requests) -- newly generated/watched addresses - -## btc-wallet-hook -I don't like the name "hook" here, but can't think of anything better atm. It's closer to a non-wallet-state manager on top of the wallet-store; potentially just one of many. - -Incoming data: -- responses from `btc-provider` -- connectivity status from `btc-provider` -- address lookup requests from `btc-wallet-store` -- newly generated/watched addresses from `btc-wallet-store` - -Outgoing data: -- pokes `btc-wallet-store` with address updates -- pokes `btc-wallet-store` with address generation requests -- pokes `btc-provider` with address lookup requests - -Error conditions: -Disconnected provider: when it receives a message that this is the case, it stops sending outgoing address info requests until the provider says it's back up. Once we receive a connected message, all pending requests are retried. - -## btc-provider -Layers on top of both BTC and ElectRS *TODO explain why the latter* - -Incoming data: - -Outgoing data: - -Error conditions: - -## Resource Usage - -### Provider -- machine: requires hard drive and processing to run a BTC full node and ElectRS indexer. -- network: sends out *all* addresses in each new block to *all* clients - -### Wallet -- machine: processes all addresses for each block to see whether they are being watched. -- network: receives all addresses for each block - -## Needed Extensions -- Invoice generator that asks for addresses on behalf of ships and tracks whether they've made payments. `wallet-hook` could probably be renamed to `wallet-manager` and extended for this purpose. -- `btc-provider` should push out address state in each block -- `btc-wallet-store` should watch the next ~20 addresses in each wallet account, so that it can detect BTC sent to the wallet if the wallet is also manageed/generates addresses in an outside-Urbit program. -- use Branch-and-Bound UTXO selection algo - -## Possible Improvements/Changes -- Do away with `btc-wallet-hook` altogether in its current form, and instead make `btc-provider` both a server (as it is now) and also a client. Pros: less between-agent data. Cons: complicates the otherwise simple provider module. PrA better solution might be to split just the connectivity parts of `btc-wallet-hook` into a local provider -- Multiple Providers -- Gossip network for both pulling and pushing address updates to lower network usage on the providers. - diff --git a/DEMO.md b/DEMO.md index 220d34925..0a51e8709 100644 --- a/DEMO.md +++ b/DEMO.md @@ -11,12 +11,14 @@ Runs the full node API services. ## Start Agents and set XPUBs On `~zod`. Uses "abandon abandon..." mnemonic ``` +=network %main +=network %testnet |commit %home |start %btc-provider |start %btc-wallet -:btc-provider|command [%set-credentials api-url='http://localhost:50002' %main] -:btc-wallet|command [%set-provider ~zod %main] +:btc-provider|command [%set-credentials api-url='http://localhost:50002' network] +:btc-wallet|command [%set-provider ~zod network] :btc-provider|command [%add-whitelist %users `(set ship)`(sy ~[~dopzod])] =fprint [%4 0xbeef.dead] @@ -26,10 +28,12 @@ On `~zod`. Uses "abandon abandon..." mnemonic On `~dopzod`. Uses "absurd sick..." mnemonic from PRIVATE.scratch.md ``` +=network %main +=network %testnet |commit %home |start %btc-wallet -:btc-wallet|command [%set-provider ~zod %main] +:btc-wallet|command [%set-provider ~zod network] =fprint [%4 0xdead.beef] =xpubmain 'zpub6r8dKyWJ31XF6n69KKeEwLjVC5ruqAbiJ4QCqLsrV36Mvx9WEjUaiPNPGFLHNCCqgCdy6iZC8ZgHsm6a1AUTVBMVbKGemNcWFcwBGSjJKbD' @@ -38,10 +42,12 @@ On `~dopzod`. Uses "absurd sick..." mnemonic from PRIVATE.scratch.md ### Add Wallets On both `~zod`/`dopzod`, choose depending on whether you're on test or main -``` -:btc-wallet|command [%add-wallet xpubmain fprint ~ [~ 8] [~ 6]] -:btc-wallet|command [%add-wallet xpubtest fprint ~ [~ 8] [~ 6]] +Using 1 confirmation for testing. +``` +:btc-wallet|command [%add-wallet xpubmain fprint ~ [~ 8] [~ 1]] + +:btc-wallet|command [%add-wallet xpubtest fprint ~ [~ 8] [~ 1]] ``` ## Check Balance @@ -57,33 +63,33 @@ On both `~zod`/`dopzod`, choose depending on whether you're on test or main `~dopzod` ``` -:btc-wallet-hook|command [%req-pay-address ~zod 30.000 feyb=10 +:btc-wallet|command [%req-pay-address ~zod 80.000 feyb=100] ``` ### Check State on ~zod/~dopzod `~dopzod`: outgoing ``` -:btc-wallet-hook +dbug [%state 'poym'] +:btc-wallet +dbug [%state 'poym'] ``` `~zod`: incoming ``` -:btc-wallet-hook +dbug [%state 'piym'] +:btc-wallet +dbug [%state 'piym'] ``` ### Idempotent `~dopzod` ``` -:btc-wallet-hook|command [%req-pay-address ~zod 3.000 feyb=100 +:btc-wallet|command [%req-pay-address ~zod 3.000 feyb=100] ``` Or can change amount: ``` -:btc-wallet-hook|command [%req-pay-address ~zod 3.000 feyb=100 +:btc-wallet|command [%req-pay-address ~zod 3.000 feyb=100] ``` ### Broadcast the Signed TX ``` -:btc-wallet-hook|command [%broadcast-tx tx] +:btc-wallet|command [%broadcast-tx tx] ``` diff --git a/app/btc-wallet.hoon b/app/btc-wallet.hoon index 0f232ff71..81ec57317 100644 --- a/app/btc-wallet.hoon +++ b/app/btc-wallet.hoon @@ -35,7 +35,6 @@ feybs=(map ship sats) =piym =poym - =pend-piym == :: +$ card card:agent:gall @@ -68,7 +67,6 @@ *(map ship sats) *^piym *^poym - *^pend-piym == == ++ on-save @@ -82,7 +80,6 @@ ++ on-poke |= [=mark =vase] ^- (quip card _this) - ?> (team:title our.bowl src.bowl) =^ cards state ?+ mark (on-poke:def mark vase) %btc-wallet-action @@ -135,6 +132,7 @@ ++ handle-command |= comm=command ^- (quip card _state) + ?> =(our.bowl src.bowl) ?- -.comm %set-provider =* sub-card @@ -146,16 +144,16 @@ == :: %set-current-wallet - ?~ (find ~[xpub.comm] scanned-wallets) - `state - `state(curr-xpub `xpub.comm) + (set-curr-xpub xpub.comm) :: %add-wallet ?~ (~(has by walts) xpub.comm) ((slog ~[leaf+"xpub already in wallet"]) `state) =/ w=walt (from-xpub +.comm) =. walts (~(put by walts) xpub.comm w) - (init-batches xpub.comm (dec max-gap.w)) + =^ c1 state (init-batches xpub.comm (dec max-gap.w)) + =^ c2 state (set-curr-xpub xpub.comm) + [(weld c1 c2) state] :: %delete-wallet =* cw curr-xpub.state @@ -171,7 +169,7 @@ ?< ?=(%pawn (clan:title payee.comm)) ?< is-broadcasting :_ state(poym ~, feybs (~(put by feybs) payee.comm feyb.comm)) - ~[(poke-us payee.comm [%gen-pay-address value.comm])] + ~[(poke-us payee.comm [%gen-pay-address value.comm])] :: %broadcast-tx ?> =(src.bowl our.bowl) @@ -211,8 +209,8 @@ ?> =(src.bowl our.bowl) =^ cards state ?. included.ti.act - `state - ?: (~(has by pend-piym) txid.ti.act) + `state + ?: (~(has by pend.piym) txid.ti.act) (piym-to-history ti.act) ?: (poym-has-txid txid.ti.act) (poym-to-history ti.act) @@ -253,7 +251,7 @@ ?^ cards [cards state] =+ f=(fam src.bowl) =+ n=(~(gut by num-fam.piym) f 0) - ?~ curr-xpub ~|("btc-wallet-hook: no curr-xpub set" !!) + ?~ curr-xpub ~|("btc-walle: no curr-xpub set" !!) ?: (gte n fam-limit.params) ~|("More than {} addresses for moons + planet" !!) =. state state(num-fam.piym (~(put by num-fam.piym) f +(n))) @@ -284,7 +282,7 @@ :_ state(poym tb) ?~ tb ~ %+ turn txis.u.tb - |=(=txi (poke-provider [%tx-info txid.utxo.txi])) + |=(=txi (poke-provider [%raw-tx txid.utxo.txi])) :: ++ generate-txbu |= [=xpub payee=(unit ship) feyb=sats txos=(list txo)] @@ -295,7 +293,7 @@ =/ [tb=(unit txbu) chng=(unit sats)] %~ with-change sut [u.uw eny.bowl block.btc-state payee feyb txos] - ?~ tb ((slog leaf+"insufficient balance") [tb state]) + ?~ tb ((slog ~[leaf+"insufficient balance or not enough confirmed balance"]) [tb state]) :: if no change, return txbu; else add change output to txbu :: ?~ chng [tb state] @@ -308,7 +306,7 @@ :: %expect-payment :: - check that payment is in piym :: - replace pend.payment with incoming txid (lock) - :: - add txid to pend-piym + :: - add txid to pend.piym :: - request tx-info from provider :: %expect-payment @@ -330,7 +328,7 @@ == :: :: +handle-provider-status: handle connectivity updates from provider -:: - retry pend-piym on any %connected event, since we're checking mempool +:: - retry pend.piym on any %connected event, since we're checking mempool :: - if status is %connected, retry all pending address lookups :: - only retry all if previously disconnected :: - if block is updated, retry all address reqs @@ -400,18 +398,21 @@ ^- _state |^ =/ h (~(get by history) txid.ti) - =/ addrs=(set address) + =/ our-addrs=(set address) :: all our addresses in inputs/outputs of tx %- sy - %+ turn (weld inputs.ti outputs.ti) - |=(=val:tx address.val) - =/ w (first-matching-wallet ~(tap in addrs)) - ?~ w state - ?~ h :: addresses in wallets, but tx not in history + %+ skim + %+ turn (weld inputs.ti outputs.ti) + |=(=val:tx address.val) + is-our-address + ?: =(0 ~(wyt in our-addrs)) state + =/ =xpub + xpub.w:(need (address-coords (snag 0 ~(tap in our-addrs)) ~(val by walts))) + ?~ h :: addresses in wallets, but tx not in history =. history %+ ~(put by history) txid.ti - (mk-hest xpub.u.w addrs) + (mk-hest xpub our-addrs) state - ?. included.ti + ?. included.ti :: tx in history, but not in mempool/blocks state(history (~(del by history) txid.ti)) %_ state history @@ -421,29 +422,29 @@ :: ++ mk-hest :: has tx-info - |= [=xpub addrs=(set address)] + |= [=xpub our-addrs=(set address)] ^- hest :* xpub txid.ti confs.ti recvd.ti - (turn inputs.ti |=(v=val:tx (our-ship addrs v))) - (turn outputs.ti |=(v=val:tx (our-ship addrs v))) + (turn inputs.ti |=(v=val:tx (is-our-ship our-addrs v))) + (turn outputs.ti |=(v=val:tx (is-our-ship our-addrs v))) == :: - ++ our-ship + ++ is-our-ship |= [as=(set address:btc) v=val:tx:btc] ^- [=val:tx s=(unit ship)] [v ?:((~(has in as) address.v) `our.bowl ~)] :: - ++ first-matching-wallet - |= addrs=(list address) - ^- (unit walt) - |- ?~ addrs ~ - =/ ac (address-coords i.addrs ~(val by walts)) - ?^ ac `w.u.ac - $(addrs t.addrs) + ++ is-our-address + |=(a=address ?=(^ (address-coords a ~(val by walts)))) -- +++ set-curr-xpub + |= =xpub + ^- (quip card _state) + ?~ (find ~[xpub] scanned-wallets) `state + `state(curr-xpub `xpub) :: :: :: Scan Logic @@ -551,7 +552,8 @@ =. scans (~(del by scans) [xpub %0]) =. scans (~(del by scans) [xpub %1]) %- (slog ~[leaf+"Scanned xpub {}"]) - `state(walts (~(put by walts) xpub w(scanned %.y))) + =. state state(walts (~(put by walts) xpub w(scanned %.y))) + (set-curr-xpub xpub) :: +check-scan: initiate a scan if one hasn't started :: check status of scan if one is running :: @@ -626,18 +628,18 @@ $(idx +(idx), txos t.txos) -- :: +piym-to-history -:: - checks whether txid in pend-piym +:: - checks whether txid in pend.piym :: - checks whether ti has a matching value output to piym -:: - if no match found, just deletes pend-piym with this tx +:: - if no match found, just deletes pend.piym with this tx :: stops peer from spamming txids :: - returns card that adds hest to history :: ++ piym-to-history |= ti=info:tx |^ ^- (quip card _state) - =+ pay=(~(get by pend-piym) txid.ti) + =+ pay=(~(get by pend.piym) txid.ti) ?~ pay `state - :: if no matching output in piym, delete from pend-piym to stop DDOS of txids + :: if no matching output in piym, delete from pend.piym to stop DDOS of txids :: =+ vout=(get-vout value.u.pay) ?~ vout @@ -659,14 +661,14 @@ ++ del-pend-piym |= txid=hexb ^- _state - state(pend-piym (~(del by pend-piym) txid.ti)) + state(pend.piym (~(del by pend.piym) txid.ti)) :: ++ del-all-piym |= [txid=hexb payer=ship] ^- _state =+ nf=(~(gut by num-fam.piym) payer 1) %= state - pend-piym (~(del by pend-piym) txid) + pend.piym (~(del by pend.piym) txid) ps.piym (~(del by ps.piym) payer) num-fam.piym (~(put by num-fam.piym) payer (dec nf)) == @@ -695,17 +697,17 @@ ^- ship ?. =(%earl (clan:title s)) s (sein:title our.bowl now.bowl s) -:: +update-pend-piym +:: +update-pend.piym :: - set pend.payment to txid (lock) -:: - add txid to pend-piym +:: - add txid to pend.piym :: ++ update-pend-piym |= [txid=hexb p=payment] ^- _state - ?~ pend.p ~|("update-pend-piym: empty pend.payment" !!) + ?~ pend.p ~|("update-pend-piym: no pending payment" !!) %= state ps.piym (~(put by ps.piym) payer.p p) - pend-piym (~(put by pend-piym) txid p) + pend.piym (~(put by pend.piym) txid p) == :: :: +update-poym-txis: @@ -769,7 +771,7 @@ :: ++ retry-pend-piym ^- (list card) - %+ turn ~(tap in ~(key by pend-piym)) + %+ turn ~(tap in ~(key by pend.piym)) |=(=txid (poke-provider [%tx-info txid])) :: ++ poke-provider diff --git a/lib/btc-wallet.hoon b/lib/btc-wallet.hoon index bcf5294f8..9fdb951d6 100644 --- a/lib/btc-wallet.hoon +++ b/lib/btc-wallet.hoon @@ -292,7 +292,7 @@ :: ++ with-change ^- [tb=(unit txbu) chng=(unit sats)] - =+ tb=select-utxos + =/ tb=(unit txbu) select-utxos ?~ tb [~ ~] =+ excess=~(fee txb u.tb) :: (inputs - outputs) =/ new-fee=sats :: cost of this tx + one more output diff --git a/mar/btc-wallet/action.hoon b/mar/btc-wallet/action.hoon new file mode 100644 index 000000000..6d50771ea --- /dev/null +++ b/mar/btc-wallet/action.hoon @@ -0,0 +1,12 @@ +/- *btc-wallet +|_ act=action +++ grad %noun +++ grow + |% + ++ noun act + -- +++ grab + |% + ++ noun action + -- +-- diff --git a/psbt_sign.js b/psbt_sign.js index e186f85bf..7484174e9 100644 --- a/psbt_sign.js +++ b/psbt_sign.js @@ -48,6 +48,7 @@ const run = async(network, mnemonics, psbtStr) => { const validate = await psbt.validateSignaturesOfAllInputs(); await psbt.finalizeAllInputs(); const hex = psbt.extractTransaction().toHex(); + console.log(bitcoin.Transaction.fromHex(hex).getId()) console.log({ validate, hex}); } return; diff --git a/sur/btc-wallet.hoon b/sur/btc-wallet.hoon index 2383e9c3e..e3b9695c9 100644 --- a/sur/btc-wallet.hoon +++ b/sur/btc-wallet.hoon @@ -6,8 +6,7 @@ +$ block @ud +$ btc-state [=block fee=(unit sats) t=@da] +$ payment [pend=(unit txid) =xpub =address payer=ship value=sats] -+$ piym [ps=(map ship payment) num-fam=(map ship @ud)] -+$ pend-piym (map txid payment) ++$ piym [ps=(map ship payment) pend=(map txid payment) num-fam=(map ship @ud)] +$ poym (unit txbu) :: +$ command