architecture writeup; basic scrys

ARCH.md

@@ -31,17 +31,22 @@ Incoming data:
 - requests to generate and watch new addresses
 
 Outgoing data:
-- requests for address info
+- requests for address info on unscanned address batches (sends to each new subscriber on /requests)
 - newly generated/watched addresses
 
 ## btc-wallet-hook
 I don't like the name "hook" here, but can't think of anything better atm. It's closer to a non-wallet-state manager on top of the wallet-store; potentially just one of many.
 
-It interfaces with hte 
+Incoming data:
+- responses from `btc-provider`
+- connectivity status from `btc-provider`
+- address lookup requests from `btc-wallet-store`
+- newlyy generated/watched addresses from `btc-wallet-store`
 
 Outgoing data:
-
-Incoming data:
+- pokes `btc-wallet-store` with address updates
+- pokes `btc-wallet-store` with address generation requests
+- pokes `btc-provider` with address lookup requests
 
 Error conditions:
 Disconnected provider: when it receives a message that this is the case, it stops sending outgoing address info requests until the provider says it's back up. Once we receive a connected message, all pending requests are retried.
@@ -71,7 +76,7 @@ Error conditions:
 - `btc-wallet-store` should watch the next ~20 addresses in each wallet account, so that it can detect BTC sent to the wallet if the wallet is also managed/generates addresses in an outside-Urbit program.
 
 ## Possible Improvements/Changes
-- Do away with `btc-wallet-hook` altogether in its current form, and instead make `btc-provider` both a server (as it is now) and also a client. Pros: conceptually clean; less between-agent data. Cons: complicates the otherwise simple provider codebase.
+- Do away with `btc-wallet-hook` altogether in its current form, and instead make `btc-provider` both a server (as it is now) and also a client. Pros: less between-agent data. Cons: complicates the otherwise simple provider module. PrA better solution might be to split just the connectivity parts of `btc-wallet-hook` into a local provider
 - Multiple Providers
 - Gossip network for both pulling and pushing address updates to lower network usage on the providers.
 

BTC.scratch.md

@@ -3,7 +3,7 @@ The below requires norsyr's fix to `decompress-point` in order to work.
 
 ## Set Credentials and Ping Servers
 ```
-=rpc-pass '9cd2f14f9855ea5eec587935887e83c39494a35f036a9ef30abcb12df6ccf0fe'
+=rpc-pass 'e09acb2144ec8a5d32fe6b3c098590c10935be19395e6a997ae5eaca89ea0133'
 :btc-provider|command [%set-credentials [rpc-url='http://localhost:8332' rpc-user='__cookie__' rpc-pass] [rpc-url='http://localhost:50002']]
 
 :btc-provider|action ['addr0' %address-info [%bech32 'bc1q59u5epktervh6fxqay2dlph0wxu9hjnx6v8n66']]

WALLET.scratch.md

@@ -63,13 +63,13 @@ abandon abandon abandon abandon abandon abandon abandon abandon abandon abandon
 With max-gap=3
 ```
 :btc-wallet-hook|action [%set-provider ~dopzod]
-=btc -build-file %/lib/btc/hoon
 =scan-xpub 'zpub6rFR7y4Q2AijBEqTUquhVz398htDFrtymD9xYYfG1m4wAcvPhXNfE3EfH1r1ADqtfSdVCToUG868RvUUkgDKf31mGDtKsAYz2oz2AGutZYs'
 :btc-wallet-store|action [%add-wallet scan-xpub ~ [~ 3]]
 :btc-wallet-store +dbug
 :: shows scans with the xpub and {0 1 2} todos
 
 ::  %0 account has no used
+=btc -build-file %/lib/btc/hoon
 :btc-wallet-store|action [%watch-address scan-xpub %0 1 *(set utxo:btc) used=%.n]
 :btc-wallet-store|action [%watch-address scan-xpub %0 2 *(set utxo:btc) used=%.n]
 :btc-wallet-store|action [%watch-address scan-xpub %0 0 *(set utxo:btc) used=%.n]
@@ -80,43 +80,13 @@ With max-gap=3
 :: dbug should show re-filled scans: [xpub %1]
 ```
 
+## scrys
+```
+.^((list @t) %gx /=btc-wallet-store=/scanned/noun)
+```
+
 ## Algos
 
-### Scan addresses
-#### in wallet-store
-* maps:
-  - scans ([xpub chyg] -> waltscan)
-  
-* start scan
-  - params: xpub
-  - get `nixt` from wallet
-  - choose the next N indices WITHOUT generating
-  - store their indexes in batch
-  - new entry in scans for this xpub+both chyg
-  - send address+ idx [xpub chyg] to wallet-hook for processing
-  
-* on `:watch-address` action
-  - watch-address in the wallet IF used
-  - delete `idx` from todo.batch
-  - if the `todo`s in this xpub+chyg batch are empty, check whether wallet is scanned
-
-#### in wallet-hook
-* types
-  - req-id=@t: hash160 of (cat xpub chyg)
-  - pend/fail: (req-id -> [=idx key=[xpub chyg]])
-  - timeouts: (req-id -> @da) -- store Behns for each req
-
-* send address-watch req
-  - send address to provider with req-id
-  - set a Behn for 30s, put in timeouts
-
-* on response from server
-  - insert the address into the wallet
-  
-* on error
-
-* on timeout
-
 ### Monitor addresses
 - nixt also stores next 50 addresses for each account.
 - every update-address call also checks those

app/btc-provider.hoon

@@ -119,7 +119,7 @@
   =/  req=request:http
     (gen-request host-info ract)
   [%pass (mk-wire act ract) %arvo %i %request req out]
-::  wire structure: /action-tas/rpc-action-tas/req-id/(address, if rpc-action %erpc)/now
+::  wire structure: /action-tas/rpc-action-tas/req-id/(address, if rpc-action %erpc)/now 
 ::
 ++  mk-wire
   |=  [act=action ract=action:rpc]
@@ -145,7 +145,6 @@
   =^  conn-err  state
     (connection-error status)
   ?^  conn-err
-    ~&  >>>  conn-err
     :_  state(connected.host-info %.n)
     ~[(send-status %disconnected) (send-update [%| u.conn-err])]
   =/  rpc-resp=response:rpc:jstd

app/btc-wallet-store.hoon

@@ -66,12 +66,26 @@
   ?>  (team:title our.bowl src.bowl)
   ?+  pax  (on-watch:def pax)
       [%requests *]
-    `this
+      ::  TODO: run req-scan on all scans
+    :_  this
+    %-  zing
+    %~  val  by
+    %-  ~(urn by scans)
+      |*  [k=[=xpub:btc =chyg] b=batch]
+      ^-  (list card)
+      (req-scan ~ b xpub.k chyg.k)
+    ::
       [%updates *]
     `this
   ==
+++  on-peek
+  |=  pax=path
+  ^-  (unit (unit cage))
+  ?+  pax  (on-peek:def pax)
+      [%x %scanned ~]
+    ``noun+!>(scanned-wallets)
+  ==
 ++  on-leave  on-leave:def
-++  on-peek   on-peek:def
 ++  on-agent  on-agent:def
 ++  on-arvo   on-arvo:def
 ++  on-fail   on-fail:def
@@ -111,12 +125,12 @@
 ::    - if it isn't, refill it with idxs to scan
 ::
 ++  req-scan
-  |=  [b=batch =xpub =chyg]
+  |=  [pax=(list path) b=batch =xpub =chyg]
   ^-  (list card)
   =/  w=_walt  (~(got by walts) xpub)
   %+  turn  ~(tap in todo.b)
   |=  =idx
-  :*  %give  %fact  ~[/requests]
+  :*  %give  %fact  pax
       %btc-wallet-store-request
       !>([%scan-address (~(mk-address w chyg) idx) xpub chyg idx])
   ==
@@ -140,7 +154,7 @@
   ^-  (quip card _state)
   =/  b=batch
     [(sy (gulf 0 endpoint)) endpoint %.n]
-  :-  (weld (req-scan b xpub %0) (req-scan b xpub %1))
+  :-  (weld (req-scan ~[/requests] b xpub %0) (req-scan ~[/requests] b xpub %1))
   state(scans (insert-batches xpub b b))
 ::  if the batch is done but the wallet isn't done scanning, returns new address requests and updated batch
 ::
@@ -157,7 +171,7 @@
         (add endpoint.b max-gap.st.w)
         %.n
     ==
-  :-  (req-scan newb xpub chyg)
+  :-  (req-scan ~[/requests] newb xpub chyg)
   newb
 ::
 ++  iter-scan
@@ -210,4 +224,11 @@
   ?:  empty:(scan-status xpub chyg)
     (run-scan xpub)
   `state
+::
+++  scanned-wallets
+  ^-  (list xpub)
+  %+  murn  ~(tap by walts)
+  |=  [=xpub:btc w=_walt]
+  ^-  (unit xpub:btc)
+  ?:  scanned.st.w  `xpub  ~
 --