ilyakooo0/urbit
1
0
Fork 0
mirror of https://github.com/ilyakooo0/urbit.git synced 2024-12-01 03:23:09 +03:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #5647 from urbit/roller/batch-split

Browse Source 
roller: add batch-trimming functionality
This commit is contained in:
Philip Monk 2022-05-03 14:34:12 -07:00 committed by GitHub
commit 71d4f72ce1
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
5 changed files with 304 additions and 68 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

283
pkg/arvo/app/roller.hoon
View File

@ -31,7 +31,7 @@
::
|%
+$ app-state
$: %4
$: %6
:: pending: the next l2 txs to be sent
:: sending: l2 txs awaiting l2 confirmation, ordered by nonce
:: finding: sig+raw-tx hash reverse lookup for txs in sending map
@ -263,8 +263,8 @@
[~d7 7 ~m5 ~m1]
=, old-state
:* %1
pending sending finding history
ship-quota next-nonce next-batch
pending ^-((tree [l1-tx-pointer old-send-tx-4]) sending)
finding history ship-quota next-nonce next-batch
pre own pk slice quota derive
frequency endpoint contract chain-id
resend-time update-rate
@ -294,7 +294,7 @@
resend-time update-rate
==
=? old-state ?=(%3 -.old-state)
^- app-state
^- state-4
=, old-state
=/ fallback-gas-price=@ud 10.000.000.000
:* %4
@ -305,10 +305,61 @@
frequency endpoint contract chain-id
resend-time update-rate fallback-gas-price
==
?> ?=(%4 -.old-state)
=? old-state ?=(%4 -.old-state)
^- state-5
=/ new-sending=(tree [l1-tx-pointer old-send-tx-5])
%+ run:ors:dice sending.old-state
|= old=old-send-tx-4
^- old-send-tx-5
old(txs (turn txs.old (lead |)))
=, old-state
:* %5
pending new-sending finding history
ship-quota allowances
next-nonce next-batch next-slice
pre own spo pk slice quota derive
frequency endpoint contract chain-id
resend-time update-rate fallback-gas-price
==
=? old-state ?=(%5 -.old-state)
^- app-state
=/ new-sending=(tree [l1-tx-pointer send-tx])
%+ run:ors:dice sending.old-state
|= old=old-send-tx-5
^- send-tx
%= old
txs
%+ turn txs.old
|= [force=? =raw-tx:naive]
=/ sign-address=(unit @ux)
(extract-address:lib raw-tx pre chain-id)
:_ [force raw-tx]
?. ?=(^ sign-address)
0x0
u.sign-address
==
=, old-state
:* %6
pending new-sending finding history
ship-quota allowances
next-nonce next-batch next-slice
pre own spo pk slice quota derive
frequency endpoint contract chain-id
resend-time update-rate fallback-gas-price
==
?> ?=(%6 -.old-state)
[cards this(state old-state)]
::
++ app-states $%(state-0 state-1 state-2 state-3 app-state)
++ app-states
$% state-0
state-1
state-2
state-3
state-4
state-5
app-state
==
::
++ state-0
$: %0
pending=(list pend-tx)
@ -330,7 +381,7 @@
++ state-1
$: %1
pending=(list pend-tx)
sending=(tree [l1-tx-pointer send-tx])
sending=(tree [l1-tx-pointer old-send-tx-4])
finding=(map keccak ?(%confirmed %failed [=time l1-tx-pointer]))
history=(map address:ethereum (tree hist-tx))
ship-quota=(map ship @ud)
@ -353,7 +404,7 @@
++ state-2
$: %2
pending=(list pend-tx)
sending=(tree [l1-tx-pointer send-tx])
sending=(tree [l1-tx-pointer old-send-tx-4])
finding=(map keccak ?(%confirmed %failed [=time l1-tx-pointer]))
history=(map address:ethereum (tree hist-tx))
ship-quota=(map ship @ud)
@ -373,10 +424,11 @@
resend-time=@dr
update-rate=@dr
==
::
++ state-3
$: %3
pending=(list pend-tx)
sending=(tree [l1-tx-pointer send-tx])
sending=(tree [l1-tx-pointer old-send-tx-4])
finding=(map keccak ?(%confirmed %failed [=time l1-tx-pointer]))
history=(map address:ethereum (tree hist-tx))
ship-quota=(map ship @ud)
@ -398,6 +450,65 @@
resend-time=@dr
update-rate=@dr
==
::
+$ old-send-tx-4 [next-gas-price=@ud sent=? txs=(list =raw-tx:naive)]
::
++ state-4
$: %4
pending=(list pend-tx)
sending=(tree [l1-tx-pointer old-send-tx-4])
finding=(map keccak ?(%confirmed %failed [=time l1-tx-pointer]))
history=(map address:ethereum (tree hist-tx))
ship-quota=(map ship @ud)
allowances=(map ship (unit @ud))
next-nonce=(unit @ud)
next-batch=time
next-slice=time
pre=^state:naive
own=owners
spo=sponsors
pk=@
slice=@dr
quota=@ud
derive=?
frequency=@dr
endpoint=(unit @t)
contract=@ux
chain-id=@
resend-time=@dr
update-rate=@dr
fallback-gas-price=@ud
==
::
+$ old-send-tx-5
[next-gas-price=@ud sent=? txs=(list [force=? =raw-tx:naive])]
::
++ state-5
$: %5
pending=(list pend-tx)
sending=(tree [l1-tx-pointer old-send-tx-5])
finding=(map keccak ?(%confirmed %failed [=time l1-tx-pointer]))
history=(map address:ethereum (tree hist-tx))
ship-quota=(map ship @ud)
allowances=(map ship (unit @ud))
next-nonce=(unit @ud)
next-batch=time
next-slice=time
pre=^state:naive
own=owners
spo=sponsors
pk=@
slice=@dr
quota=@ud
derive=?
frequency=@dr
endpoint=(unit @t)
contract=@ux
chain-id=@
resend-time=@dr
update-rate=@dr
fallback-gas-price=@ud
==
--
::
++ on-poke
@ -498,9 +609,20 @@
::
[%resend @ @ ~]
=/ [address=@ux nonce=@ud]
[(slav %ux i.t.wire) (rash i.t.t.wire dem)]
?+ +<.sign-arvo (on-arvo:def wire sign-arvo)
%wake [(send-roll:do address nonce) this]
[(slav %ux i.t.wire) (slav %ud i.t.t.wire)]
?+ +<.sign-arvo (on-arvo:def wire sign-arvo)
%wake
=/ cards=(list card) (send-roll:do address nonce)
=? sending
?& ?=(~ cards)
(has:ors:dice sending [address nonce])
=(0 (lent txs:(got:ors:dice sending [address nonce])))
==
~& > "empty sending, removing {<[nonce address]>}"
=^ * sending
(del:ors:dice sending [address nonce])
sending
[cards this]
==
==
::
@ -572,7 +694,7 @@
|= [address=@t nonce=@t =sign:agent:gall]
^- (quip card _this)
=/ [address=@ux nonce=@ud]
[(slav %ux address) (rash nonce dem)]
[(slav %ux address) (slav %ud nonce)]
?- -.sign
%poke-ack
?~ p.sign
@ -602,7 +724,7 @@
[cards this]
::
%thread-done
=+ !<(result=(each @ud [term @t]) q.cage.sign)
=+ !<(result=(each [@ud @ud] [term @t]) q.cage.sign)
=^ cards state
(on-batch-result:do address nonce result)
[cards this]
@ -674,7 +796,7 @@
++ refresh
|= [nonce=@t =sign:agent:gall]
^- (quip card _this)
=/ failed-nonce=@ud (rash nonce dem)
=/ failed-nonce=@ud (slav %ud nonce)
?- -.sign
%poke-ack
?~ p.sign
@ -776,16 +898,17 @@
::
=* key key.i.sorted
=* val val.i.sorted
=+ txs=(turn txs.val |=(=raw-tx:naive [| 0x0 *time raw-tx]))
=/ txs=(list pend-tx)
%+ turn txs.val
|=([addr=@ux force=? =raw-tx:naive] force^addr^*time^raw-tx)
=^ [new-valid=_txs nups=_ups] state
(apply-txs txs %sending `nonce.key)
=/ new-sending
(turn new-valid |=([force=? addr=@ux * =raw-tx:naive] addr^force^raw-tx))
:: we only hear updates for this nonce if it has been sent
::
=. valid ::=? valid sent.val
%^ put:ors:dice valid
key
:: TODO: too much functional hackery?
val(txs (turn new-valid (cork tail (cork tail tail))))
(put:ors:dice valid key val(txs new-sending))
$(sorted t.sorted, ups (welp ups nups))
::
++ apply-txs
@ -809,8 +932,14 @@
?: gud [~ history]
=. time.tx
?: ?=(%pending type) time.tx
=+ wer=(~(got by finding) keccak)
?>(?=(^ wer) time.wer)
?~ wer=(~(get by finding) keccak)
~& >>> "missing %sending tx in finding"^[ship raw-tx]
now.bowl
?@ u.wer
~& >>> "weird tx in finding gud: {<gud>} {<u.wer>}"^[ship raw-tx]
now.bowl
time.u.wer
~? =(0x0 address.tx) %weird-null-tx-address^'apply-txs'
(update-history:dice history [tx]~ %failed)
=? finding !gud (~(put by finding) keccak %failed)
=. updates :(welp up-2 up-1 updates)
@ -1031,7 +1160,8 @@
sending
%^ put:ors:dice sending
[address nonce]
[0 | (turn pending (cork tail (cork tail tail)))]
:+ 0 |
(turn pending |=([force=? addr=@ux * =raw-tx:naive] addr^force^raw-tx))
::
finding
%- ~(gas by finding)
@ -1084,11 +1214,11 @@
:: If %.y, the roller has been trying to send a batch for a whole frequency.
::
:: The cause of not sending the previous batch can happen because
:: of thread failure (see line 1251) or because the private key loaded onto
:: of thread failure or because the private key loaded onto
:: the roller was used for something other than signing L2 batches right
:: after the send-batch thread started.
::
:: After reaching this state, any subsequents attempts have failed (L: 1251)
:: After reaching this state, any subsequents attempts have failed
:: (prior to updating the sending nonce if we hit the on-out-of-sync case)
:: which would possibly require a manual intervention (e.g. changing the
:: ethereum node URL, adding funds to the roller's address, manually bumping
@ -1139,7 +1269,7 @@
::
++ process-l2-txs
%+ roll txs.q
|= [=raw-tx:naive nif=_finding sih=_history]
|= [[@ @ =raw-tx:naive] nif=_finding sih=_history]
=/ =keccak (hash-raw-tx:lib raw-tx)
|^
?~ val=(~(get by nif) keccak)
@ -1205,17 +1335,18 @@
nonce
fallback-gas-price
::
=< [next-gas-price txs]
(got:ors:dice sending [address nonce])
=< [next-gas-price (turn txs (cork tail tail))]
[. (got:ors:dice sending [address nonce])]
==
:: +on-batch-result: await resend after thread success or failure
::
++ on-batch-result
|= [=address:ethereum nonce=@ud result=(each @ud [term @t])]
|= [=address:ethereum nonce=@ud result=(each [@ud @ud] [term @t])]
^- (quip card _state)
|^
:: print error if there was one
::
~? ?=(%| -.result) [dap.bowl %send-error +.p.result]
~? ?=(%| -.result) [dap.bowl %send-error nonce+nonce +.p.result]
:: if this nonce was removed from the queue by a
:: previous resend-with-higher-gas thread, it's done
::
@ -1230,6 +1361,24 @@
(del:ors:dice sending [address nonce])
`state
=/ =send-tx (got:ors:dice sending [address nonce])
:: if the number of txs sent is less than the ones in sending, we remove
:: them from the latest sending batch and add them on top of the pending list
::
=/ n-txs=@ud ?:(?=(%& -.result) -.p.result (lent txs.send-tx))
=/ not-sent=(list [=address:naive force=? =raw-tx:naive])
(slag n-txs txs.send-tx)
=/ partial-send=? &(?=(%& -.result) (lth n-txs (lent txs.send-tx)))
=? txs.send-tx partial-send
(oust [n-txs (lent txs.send-tx)] txs.send-tx)
=? pending partial-send
(fix-not-sent-pending not-sent)
=/ [nif=_finding sih=_history]
(fix-not-sent-status not-sent)
=: finding nif
history sih
==
~? partial-send [%extracting-txs-from-batch (lent not-sent)]
::
=? sending ?| ?=(%& -.result)
?=([%| %crash *] result)
==
@ -1238,15 +1387,22 @@
:: update gas price for this tx in state
::
?: ?=(%& -.result)
send-tx(next-gas-price p.result, sent &)
:: if the thread crashed, we don't know the gas used,
:: so we udpate it manually, same as the thread would do
send-tx(next-gas-price +.p.result, sent &)
:: if the thread crashed, we don't know the gas used, so we udpate it
:: manually, same as the thread would do. this has the problem of causing
:: the batch to be blocked if the thread keeps crashing, and we don't have
:: enough funds to pay.
::
%_ send-tx
next-gas-price
?: =(0 next-gas-price.send-tx)
fallback-gas-price
(add next-gas-price.send-tx 5.000.000.000)
:: on the other hand if the thread fails because +fetch-gas-price fails
:: (e.g. API change), and our fallback gas price is too low, the batch will
:: also be blocked, if we don't increase the next-gas-price, so either way
:: the batch will be stuck because of another underlying issue.
::
%_ send-tx
next-gas-price
?: =(0 next-gas-price.send-tx)
fallback-gas-price
(add next-gas-price.send-tx 5.000.000.000)
==
:_ state
?: ?& !sent.send-tx
@ -1270,6 +1426,44 @@
%+ wait:b:sys
/resend/(scot %ux address)/(scot %ud nonce)
(add resend-time now.bowl)
::
++ fix-not-sent-pending
|= not-sent=(list [=address:naive force=? =raw-tx:naive])
=; txs=(list pend-tx)
(weld txs pending)
:: TODO: this would not be needed if txs.send-tx was a (list pend-tx)
::
%+ murn not-sent
|= [=address:naive force=? =raw-tx:naive]
=/ =keccak (hash-raw-tx:lib raw-tx)
?~ wer=(~(get by finding) keccak)
~& >>> %missing-tx-in-finding
~
?@ u.wer
~& >>> %missing-tx-in-finding
~
`[force address time.u.wer raw-tx]
::
++ fix-not-sent-status
|= not-sent=(list [=address:naive force=? =raw-tx:naive])
%+ roll not-sent
|= [[@ @ =raw-tx:naive] nif=_finding sih=_history]
=/ =keccak (hash-raw-tx:lib raw-tx)
?~ val=(~(get by nif) keccak)
[nif sih]
?. ?=(^ u.val)
[nif sih]
=* time time.u.val
=* address address.u.val
=* ship ship.from.tx.raw-tx
=/ l2-tx (l2-tx +<.tx.raw-tx)
=/ =roll-tx [ship %pending keccak l2-tx]
=+ txs=(~(got by sih) address)
=. txs +:(del:orh:dice txs time)
:- (~(del by nif) keccak)
%+ ~(put by sih) address
(put:orh:dice txs [time roll-tx])
--
:: +on-naive-diff: process l2 tx confirmations
::
++ on-naive-diff
@ -1309,7 +1503,7 @@
?~ sen=(get:ors:dice sending [address nonce])
~? lverb [dap.bowl %weird-double-remove nonce+nonce]
sending
?~ nin=(find [raw-tx.diff]~ txs.u.sen)
?~ nin=(find [raw-tx.diff]~ (turn txs.u.sen (cork tail tail)))
~? lverb [dap.bowl %weird-unknown nonce+nonce]
sending
=. txs.u.sen (oust [u.nin 1] txs.u.sen)
@ -1331,6 +1525,7 @@
:: ~? !forced [dap.bowl %aggregated-tx-failed-anyway err.diff]
%failed
::
~? =(0x0 tx-address) %weird-null-tx-address^'on-naive-diff'
=^ updates history
%^ update-history:dice
history
@ -1507,11 +1702,17 @@
|= wat=@t
?~ who=(slaw %p wat) [~ ~]
=/ [exceeded=? next-quota=@ud] (quota-exceeded u.who)
=/ allow=(unit (unit @ud)) (~(get by allowances) u.who)
:+ ~ ~
:- %atom
!> ^- @ud
?: exceeded 0
(sub quota.state (dec next-quota))
?: exceeded 0
=/ max-quota=@ quota.state
?: &(?=(^ allow) ?=(~ u.allow))
max-quota
=? max-quota &(?=(^ allow) ?=(^ u.allow))
u.u.allow
(sub max-quota (dec next-quota))
::
++ allowance
|= wat=@t

19
pkg/arvo/lib/azimuth-roll-rpc.hoon
View File

@ -179,6 +179,12 @@
^- (unit @ud)
?~ nonce=(~(get by params) 'nonce') ~
(ni u.nonce)
::
++ force
|= params=(map @t json)
^- (unit ?)
?~ force=(~(get by params) 'force') ~
(bo u.force)
--
::
++ to-json
@ -520,15 +526,20 @@
++ process-rpc
|= [id=@t params=(map @t json) action=l2-tx over-quota=$-(@p ?)]
^- [(unit cage) response:rpc]
?. =((lent ~(tap by params)) 4)
?. ?| =((lent ~(tap by params)) 4)
=((lent ~(tap by params)) 5)
==
[~ ~(params error:json-rpc id)]
=? params =((lent ~(tap by params)) 4)
(~(put by params) 'force' b+|)
=+ ^- $: sig=(unit @)
from=(unit [=ship proxy:naive])
addr=(unit @ux)
force=(unit ?)
==
=, from-json
[(sig params) (from params) (address params)]
?: |(?=(~ sig) ?=(~ from) ?=(~ addr))
[(sig params) (from params) (address params) (force params)]
?: |(?=(~ sig) ?=(~ from) ?=(~ addr) ?=(~ force))
[~ ~(parse error:json-rpc id)]
?: (over-quota ship.u.from)
`[%error id '-32002' 'Max tx quota exceeded']
@ -537,7 +548,7 @@
=+ (gen-tx-octs:lib u.tx)
:_ [%result id (hex:to-json 32 (hash-tx:lib p q))]
%- some
roller-action+!>([%submit | u.addr u.sig %don u.tx])
roller-action+!>([%submit u.force u.addr u.sig %don u.tx])
::
++ nonce
|= [id=@t params=(map @t json) scry=$-([ship proxy:naive] (unit @))]

4
pkg/arvo/lib/dice.hoon
View File

@ -68,13 +68,13 @@
?. (verify-sig-and-nonce:naive verifier chain-t nas raw-tx)
=+ [force ~ nas indices]
?. verb -
~& >>> [verb+verb %verify-sig-and-nonce %failed tx.raw-tx] -
~& >>> [force+force %verify-sig-and-nonce %failed tx.raw-tx] -
=^ effects-1 points.nas
(increment-nonce:naive nas from.tx.raw-tx)
?~ nex=(receive-tx:naive nas tx.raw-tx)
=+ [force ~ ?:(force nas cache) indices]
?. verb -
~& >>> [verb+verb %receive-tx %failed] -
~& >>> [force+force %receive-tx %failed] -
=* new-nas +.u.nex
=/ effects (welp effects-1 -.u.nex)
=^ updates indices

8
pkg/arvo/sur/dice.hoon
View File

@ -85,7 +85,13 @@
+$ hist-tx [p=time q=roll-tx]
+$ roll-tx [=ship =status hash=keccak type=l2-tx]
+$ pend-tx [force=? =address:naive =time =raw-tx:naive]
+$ send-tx [next-gas-price=@ud sent=? txs=(list raw-tx:naive)]
+$ send-tx
$: next-gas-price=@ud
sent=?
:: TODO: make txs as (list pend-tx)?
::
txs=(list [=address:naive force=? =raw-tx:naive])
==
+$ part-tx
$% [%raw raw=octs]
[%don =tx:naive]

58
pkg/arvo/ted/roller/send.hoon
View File

@ -13,12 +13,26 @@
=/ =address:ethereum (address-from-prv:key:ethereum pk)
;< expected-nonce=@ud bind:m
(get-next-nonce:ethio endpoint address)
=/ batch-data=octs
%+ cad:naive 3
%- flop
%+ roll txs
|= [=raw-tx:naive out=(list octs)]
[raw.raw-tx 65^sig.raw-tx out]
:: Infura enforces a max calldata size (32, 64, 128 Kb?) so we calculate how
:: many txs are included in a batch of that size, and only send those
::
=/ max-calldata=@ud 128.000
=/ [n-txs=@ud batch-data=octs]
=| n-txs=@ud
=| size=@ud
=| out=(list octs)
|- ^- [@ud octs]
?~ txs
[n-txs (cad:naive 3 (flop out))]
=* raw-tx i.txs
=. size :(add 65 p.raw.raw-tx size)
?: (gth size max-calldata)
[n-txs (cad:naive 3 (flop out))]
%_ $
n-txs +(n-txs)
txs t.txs
out [raw.raw-tx 65^sig.raw-tx out]
==
:: if the batch is malformed, emit error to kick it out of sending
::
?~ (parse-roll:naive q.batch-data)
@ -26,16 +40,15 @@
:: if chain expects a different nonce, don't send this transaction
::
?. =(nonce expected-nonce)
~& >>> [%unexpected-nonce nonce expected+expected-nonce]
%- pure:m
!> ^- [%.n @tas @t]
:+ %.n
%not-sent
?: (lth expected-nonce nonce)
:: if ahead, it will use the same next-gas-price when resending
:: if ahead, use the same next-gas-price when resending
::
%ahead-nonce
:: if behind, start out-of-sync flow
:: if behind, start out-of-sync flow if batch was not sent before
::
%behind-nonce
:: if a gas-price of 0 was specified, fetch the recommended one
@ -49,22 +62,23 @@
:: gasLimit = G_transaction + G_txdatanonzero × dataByteLength
:: where
:: G_transaction = 21000 gas (base fee)
:: + G_txdatanonzero = 68 gas
:: + G_txdatanonzero = 16 gas (previously 68; see EIP-2028)
:: * dataByteLength = (65 + raw) * (lent txs) bytes
::
:: TODO: enforce max number of tx in batch?
:: 1.000 gas are added to the base fee as extra, for emitting the log
::
=/ gas-limit=@ud (add 21.000 (mul 68 p.batch-data))
:: if we cannot pay for the transaction, don't bother sending it out
::
=/ max-cost=@ud (mul gas-limit use-gas-price)
=/ gas-limit=@ud (add 22.000 (mul 16 p.batch-data))
=/ max-cost=@ud (mul gas-limit use-gas-price)
;< balance=@ud bind:m
(get-balance:ethio endpoint address)
?: (gth max-cost balance)
:: if we cannot pay for the transaction, don't bother sending it out
::
(pure:m !>(%.n^[%not-sent %insufficient-roller-balance]))
::
::NOTE this fails the thread if sending fails, which in the app gives us
:: the "retry with same gas price" behavior we want
::
;< =response:rpc bind:m
%+ send-batch endpoint
=; tx=transaction:rpc:ethereum
@ -80,19 +94,23 @@
:: log batch tx-hash to getTransactionReceipt(tx-hash)
::
~? &(?=(%result -.response) ?=(%s -.res.response))
^- [nonce=@ud batch-hash=@t gas=@ud]
nonce^(so:dejs:format res.response)^use-gas-price
^- [nonce=@ud batch-hash=@t gas=@ud sent-txs=@ud bytes=@ud]
:* nonce
(so:dejs:format res.response)
use-gas-price
n-txs
p.batch-data
==
%- pure:m
!> ^- (each @ud [term @t])
!> ^- (each [@ud @ud] [term @t])
:: TODO: capture if the tx fails (e.g. Runtime Error: revert)
:: check that tx-hash in +.response is non-zero?
:: enforce max here, or in app?
::
?+ -.response %.n^[%error 'unexpected rpc response']
%error %.n^[%error message.response]
:: add five gwei to gas price of next attempt
::
%result %.y^(add use-gas-price 5.000.000.000)
%result %.y^[n-txs (add use-gas-price 5.000.000.000)]
==
::
::TODO should be distilled further, partially added to strandio?