urbit/shrub
1
1
Fork 0
mirror of https://github.com/urbit/shrub.git synced 2024-11-23 20:26:54 +03:00
Code Issues Packages Projects Releases Wiki Activity

azimuth: fetch snapshot on boot

Browse Source 
Previously, the initial Azimuth snapshot was stored in Clay and shipped
in the pill.  This causes several problems:

- It bloats the pill
- Updating the snapshot added large blobs to Clay's state.  Even now
  that tombstoning is possible, you don't want to have to do that
  regularly.
- As a result, the snapshot was never updated.
- Even if you did tombstone those files, it could only be updated as
  often as the pill
- And those updates would be sent over the network to people who didn't
  need them

This moves the snapshot out of the pill and refactors Azimuth's
initialization process.  On boot, when app/azimuth starts up, it first
downloads a snapshot from bootstrap.urbit.org and uses that to
initialize its state.  As before, updates after this initial snapshot
come from an Ethereum node directly and are verified locally.

Relevant commands are:

- `-azimuth-snap-state %filename` creates a snapshot file
- `-azimuth-load "url"` downloads and inits from a snapshot, with url
  defaulting to https://bootstrap.urbit.org/mainnet.azimuth-snapshot
- `:azimuth &azimuth-poke-data %load snap-state` takes a snap-state any
  way you have it

Note the snapshot is downloaded from the same place as the pill, so this
doesn't introduce additional trust beyond what was already required.
When remote scry is released, we should consider allowing downloading
the snapshot in that way.
This commit is contained in:
Philip Monk 2022-06-30 22:55:57 -07:00
parent 370c6dcbc8
commit 89b9977ac8
9 changed files with 107 additions and 80 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
bin/solid.pill
View File

@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c45166ff0f8ab8dc1552bcef519c77c0afa6ca52f8ed1ba31ed632012667d619
size 8674763
oid sha256:fc6ca562b15bec33467dd9ca8bc24d7cde4108d03f3695abb417781dd6769453
size 22891931

135
pkg/arvo/app/azimuth.hoon
View File

@ -6,22 +6,16 @@
default-agent,
verb,
dbug
:: Generally don't update the snapshot until we have clay tombstoning.
::
/* snap %azimuth-snapshot /app/azimuth/version-0/azimuth-snapshot
:: To update, run from dojo:
:: -azimuth-snap-state %default 'version-0'
::
:: To recreate from a full list of logs (at /app/azimuth/logs/eth-logs):
:: -azimuth-snap-logs %default 'version-0'
::
=/ snap=snap-state snap
=/ last-snap=@ number.id.snap
::
=, jael
|%
+$ app-state
$: %6
$: %7
url=@ta
=net
refresh=_~m5
@ -30,10 +24,14 @@
own=owners
spo=sponsors
logs=(list =event-log:rpc:ethereum)
sap=snap-state
==
::
+$ poke-data
$% :: %listen
$% :: %load: load snapshot
::
[%load snap=snap-state]
:: %listen
::
[%listen whos=(list ship) =source:jael]
:: %watch: configure node url and network
@ -75,14 +73,18 @@
==
::
++ init-timer
|= =bowl:gall
|= at=@da
^- card
[%pass /init %arvo %b %wait now.bowl]
[%pass /init %arvo %b %wait at]
::
++ start-log-retrieval
|= [=ship args=vase]
^- card
[%pass /wa %agent [ship %eth-watcher] %poke %eth-watcher-poke args]
::
++ start-azimuth-load
^- card
[%pass /al %arvo %k %fard %base %azimuth-load %noun !>(~)]
--
::
=<
@ -92,22 +94,10 @@
def ~(. (default-agent this %|) bowl)
::
++ on-init
^- (quip card _this)
=/ points=@ud ~(wyt by points.nas.snap)
%- %- slog
[leaf+"ship: loading azimuth snapshot ({<points>} points)"]~
::
=: net.state %default
nas.state nas.snap
own.state owners.snap
spo.state sponsors.snap
url.state 'http://eth-mainnet.urbit.org:8545'
==
:_ this
?: .^(? %j /(scot %p our.bowl)/fake/(scot %da now.bowl))
~
~[(nuke-azimuth-tracker bowl) (init-timer bowl)]
::
~[(init-timer now.bowl)]
++ on-save !>(state)
++ on-load
|= old=vase
@ -122,7 +112,7 @@
`old-state
%- %- slog :_ ~
leaf+"ship: loading snapshot with {<(lent logs.old-state)>} events"
=. +.state +:(state-5-to-6 old-state)
=. +.state +:(state-6-to-7 (state-5-to-6 old-state))
=^ cards state
(%*(run-logs do nas.state *^state:naive) logs.state)
[(jael-update:do (to-udiffs:do cards)) state]
@ -130,12 +120,12 @@
?. ?=(%2 -.old-state)
`old-state
~& > '%azimuth: updating to state 3'
=. +.state +:(state-5-to-6 old-state)
=. +.state +:(state-6-to-7 (state-5-to-6 old-state))
:: replace naive state and indices with snapshot
::
=: nas.state nas.snap
own.state owners.snap
spo.state sponsors.snap
=: nas.state nas.sap.state
own.state owners.sap.state
spo.state sponsors.sap.state
logs.state ~
:: TODO: shouldn't be needed but have seen eth-watcher
:: threads use a url='' if this is not used
@ -144,8 +134,8 @@
==
=/ points=@ud ~(wyt by points.nas.state)
%- %- slog :_ ~
leaf+"ship: processing azimuth snapshot ({<points>} points)"
=/ snap-cards=udiffs:point (run-state:do id.snap points.nas.state)
leaf+"ship: processing azimuth snapshot (~{<points>} points)"
=/ snap-cards=udiffs:point (run-state:do id.sap.state points.nas.state)
:_ [%3 url net whos nas own spo logs]:state
%+ weld
(jael-update:do snap-cards)
@ -160,22 +150,30 @@
=^ cards-4 old-state
?. ?=(%4 -.old-state) [cards-3 old-state]
=^ cards this
%- %*(. on-poke +.state.this +:(state-5-to-6 old-state))
%- %*(. on-poke +.state.this +:(state-6-to-7 (state-5-to-6 old-state)))
[%azimuth-poke !>([%watch [url net]:old-state])]
~& > '%azimuth: updating to state 5'
[cards [%5 url net whos nas own spo logs]:state.this]
=? old-state ?=(%5 -.old-state)
(state-5-to-6 old-state)
?> ?=(%6 -.old-state)
=? old-state ?=(%6 -.old-state)
(state-6-to-7 old-state)
?> ?=(%7 -.old-state)
[cards-4 this(state old-state)]
::
++ app-states $%(state-0 state-1-2-3-4-5 app-state)
::
++ state-5-to-6
|= state-1-2-3-4-5
^- app-state
[%6 url net ~m5 whos nas own spo logs]
++ app-states $%(state-0 state-1-2-3-4-5 state-6 app-state)
::
+$ state-6
$: %6
url=@ta
=net
refresh=_~m5
whos=(set ship)
nas=^state:naive
own=owners
spo=sponsors
logs=(list =event-log:rpc:ethereum)
==
+$ state-1-2-3-4-5
$: ?(%1 %2 %3 %4 %5)
url=@ta
@ -196,6 +194,15 @@
own=owners
logs=(list =event-log:rpc:ethereum)
==
++ state-5-to-6
|= state-1-2-3-4-5
^- state-6
[%6 url net ~m5 whos nas own spo logs]
::
++ state-6-to-7
|= state-6
^- app-state
[%7 url net refresh whos nas own spo logs *snap-state]
--
::
++ on-poke
@ -219,9 +226,9 @@
[(subscribe-to-eth-watcher bowl)]~
::
%resnap
=: nas.state nas.snap
own.state owners.snap
spo.state sponsors.snap
=: nas.state nas.sap.state
own.state owners.sap.state
spo.state sponsors.sap.state
==
`this
==
@ -229,7 +236,22 @@
?. ?=(%azimuth-poke mark)
(on-poke:def mark vase)
=+ !<(poke=poke-data vase)
|-
?- -.poke
%load
=/ points=@ud ~(wyt by points.nas.snap.poke)
%- %- slog
[leaf+"ship: loading azimuth snapshot ({<points>} points)"]~
::
=: net.state %default
nas.state nas.snap.poke
own.state owners.snap.poke
spo.state sponsors.snap.poke
url.state 'http://eth-mainnet.urbit.org:8545'
sap.state snap.poke
==
$(poke [%kick ~])
::
%listen
[[(listen-to-azimuth (silt whos.poke) source.poke)]~ this]
::
@ -237,8 +259,8 @@
=/ last-block=@
?^ logs.state
number:(last-block-id:dice logs.state)
~& >> %no-logs-in-azimuth-state
last-snap
:: ~& >> %no-logs-in-azimuth-state
number.id.sap.state
=+ [our=(scot %p our.bowl) now=(scot %da now.bowl)]
=+ .^(dudes=(set [dude:gall ?]) %ge our %base now /)
=/ running=? (~(has in dudes) [%eth-watcher &])
@ -251,7 +273,7 @@
(listen-to-azimuth ~ [%| dap.bowl])
:: we poke eth-watcher to retrieve logs from the latest we have
::
%*(start do last-snap last-block)
%*(start do number.id.sap.state last-block)
=? cards !running
:: restart %eth-watcher
::
@ -271,18 +293,18 @@
::
?: (~(has by wex.bowl) [/eth-watcher our.bowl %eth-watcher])
cards
~& >> %resubscribing-to-eth-watcher
:: ~& >> %resubscribing-to-eth-watcher
[(subscribe-to-eth-watcher bowl) cards]
::
%watch
=: nas.state ?:(?=(%default net.poke) nas.snap *^state:naive)
own.state ?:(?=(%default net.poke) owners.snap ~)
spo.state ?:(?=(%default net.poke) sponsors.snap ~)
=: nas.state ?:(?=(%default net.poke) nas.sap.state *^state:naive)
own.state ?:(?=(%default net.poke) owners.sap.state ~)
spo.state ?:(?=(%default net.poke) sponsors.sap.state ~)
net.state net.poke
url.state url.poke
logs.state ~
==
[start:do this]
`this
==
::
++ on-watch
@ -309,7 +331,7 @@
:- %leaf
"ship: processing azimuth snapshot ({<points>} points)"
=/ snap-cards=udiffs:point
(%*(run-state do logs.state ~) id.snap points.nas.state)
(%*(run-state do logs.state ~) id.sap.state points.nas.state)
[(weld (jael-update:do snap-cards) start:do) this]
::
++ on-leave on-leave:def
@ -369,19 +391,24 @@
::
++ on-arvo
|= [=wire =sign-arvo]
?: &(=(/al wire) ?=(%arow +<.sign-arvo))
?- -.p.sign-arvo
%& `this
%|
%- (slog 'loading azimuth snapshot failed! still trying' p.p.sign-arvo)
[~[(init-timer (add ~s10 now.bowl))] this]
==
?. &(=(/init wire) ?=(%wake +<.sign-arvo))
(on-arvo:def wire sign-arvo)
?^ error.sign-arvo
%- (slog 'azimuth: failed to initialize!' ~)
`this
:_ this
~[(subscribe-to-eth-watcher bowl) (listen-to-azimuth ~ [%| dap.bowl])]
~[start-azimuth-load]
::
++ on-fail on-fail:def
--
|_ =bowl:gall
:: TODO: maybe flop the endianness here so metamask signs it in normal
:: order?
::
++ verifier
^- ^verifier:naive
@ -514,7 +541,7 @@
:+ %watch /[dap.bowl]
^- config:eth-watcher
:* url.state =(%czar (clan:title our.bowl)) refresh.state ~h30
(max launch.net ?:(=(net.state %default) +(last-snap) 0))
(max launch.net ?:(=(net.state %default) +(number.id.sap.state) 0))
~
~[azimuth.net]
~[naive.net]

BIN
pkg/arvo/app/azimuth/version-0.azimuth-snapshot
View File

Binary file not shown.

4
pkg/arvo/app/eth-watcher.hoon
View File

@ -469,6 +469,10 @@
[~ this(dogs.state (~(put by dogs.state) path u.dog(running ~)))]
::
%thread-done
:: if empty, that means we cancelled this thread
::
?: =(*vase q.cage.sign)
`this
=+ !<([vows=disavows pup=watchpup] q.cage.sign)
=. u.dog
%_ u.dog

1
pkg/arvo/sur/dice.hoon
View File

@ -7,6 +7,7 @@
+$ owners (jug owner ship)
+$ sponsors (map ship [residents=(set ship) requests=(set ship)])
+$ history (map address:ethereum (tree hist-tx))
+$ events (list event-log:rpc:ethereum)
+$ net ?(%mainnet %ropsten %local %default)
+$ snap-state [%0 =id:block:jael nas=^state:naive =owners =sponsors]
::

3
pkg/arvo/sys/lull.hoon
View File

@ -2202,6 +2202,9 @@
:: %eyre: cancel request
::
[%cancel-request ~]
:: %khan: fire thread
::
$>(%fyrd task:khan)
:: %dill: reset terminal configuration
::
$>(%hail task:dill)

24
pkg/arvo/sys/vane/jael.hoon
View File

@ -306,28 +306,6 @@
%^ poke-watch hen %azimuth
%+ fall node.tac
(need (de-purl:html 'http://eth-mainnet.urbit.org:8545'))
=. +>.$
:: get everything from /app/azimuth because jael subscriptions
:: seem to be flaky for now
::
?: &
%- curd =< abet
(sources:~(feel su hen now pki etn) ~ [%| %azimuth])
::
?- (clan:title our)
%czar
%- curd =< abet
(sources:~(feel su hen now pki etn) ~ [%| %azimuth])
::
*
=. +>.$
%- curd =< abet
%+ sources:~(feel su hen now pki etn)
(silt (turn spon-points head))
[%| %azimuth]
%- curd =< abet
(sources:~(feel su hen now pki etn) ~ [%& (need spon-ship)])
==
::
=. moz
%+ weld moz
@ -391,7 +369,7 @@
:: [%listen whos=(set ship) =source]
::
%listen
~& [%jael-listen whos source]:tac
:: %- (slog leaf+"jael: listen {<whos.tac>} {<source.tac>}" ~)
%- curd =< abet
(sources:~(feel su hen now pki etn) [whos source]:tac)
::

15
pkg/arvo/ted/azimuth/load.hoon Normal file
View File

@ -0,0 +1,15 @@
/- spider, *dice
/+ strand, strandio, naive, dice
=, strand=strand:spider
^- thread:spider
|= arg=vase
=/ m (strand ,vase)
^- form:m
=/ url=tape
?~ lur=!<((unit tape) arg)
"https://bootstrap.urbit.org/mainnet.azimuth-snapshot"
u.lur
;< =cord bind:m (fetch-cord:strandio url)
=+ ;;(snap=snap-state (cue cord))
;< ~ bind:m (poke-our:strandio %azimuth %azimuth-poke !>([%load snap]))
(pure:m !>(~))

1
pkg/base-dev/lib/strandio.hoon
View File

@ -433,7 +433,6 @@
%- pure:m
?~ full-file.client-response ''
q.data.u.full-file.client-response
::
++ fetch-cord
|= url=tape