(1) In `+thread-fail`, `+thread-clean` is called before `+thread-http-fail` and `+cancel-scry` meaning neither of the latter two actually. `serving` has been replaced with `(~(del by serving.state) tid)` in the state and therefore we cannot `(~(get by serving.state) tid)` in `+thread-http-fail` or `+cancel-scry`.
(2) In `+thread-http-fail` return an informative tang as json instead of an empty 500 http response when the failure results from an internal crash.
Include and render identities associated with requests, channels, and
login sessions. Provide the ability to kick identities and their
sessions, logging them out.
It is no longer guaranteed that the src.bowl for incoming HTTP-related
events is equal to our.bowl. Instead, it will reflect the identity
associated with the request, our or otherwise.
When serving publicly-accessible endpoints, the assertion never made
much sense, but with recent changes actively prevents guests from
accessing the endpoints. Here, we correct all such cases.
Prevously we were tracking remote scry requests using a map, assuming
that every thread would do just one remote scry request. This is not
right. A thread that did multiple +keen:strandio was treated as
if just the last call existed, overwritten previous entries in the map.
Now we track remote scries using a jug that accounts for multiple %keen
tasks per thread.
The logic for sending %yawns to %ames has been updated for the following
scenarios:
- +thread-fail will always send a %yawn task
- +thread-done doesn't send %yawn tasks
- unless a running thread is stopped
- if %spider is reloaded:
- %yawn tasks will be sent for any running or starting thread
/lib/strandio also removes +take-tune from +keen, decoupling
sending %tasks and receiving %signs. This allows for clients
to request multiple paths at future cases, without blocking.
%spider will send a %yawn task to ames if a thread fails
or stops. if the thread is done, it will delete the scry
from its state without notifying %ames
Dill's new %logs endpoint can be used to receive system output as $told
nouns. Dill no longer prints system output itself, leaving the display
of it up to terminal handlers (or the runtime). For now, to maintain the
status quo, drum subscribes to dill %logs, and prints them inline in the
default session.
Dojo had also been incorrectly updated, assuming the type of
%lens-command changed to match. Since lens should only ever be used for
running commands on the local ship (and even that being contentious (; ),
we simply auto-fill the sole session id with the local ship name.
PR #5840 mostly fixed#1559, but introduced a new bug. before, you could safely `=dir` into a desk without a case, and it would use the nonexistent case `ud+0` as the beam for dojo state, and switch that out for da+now whenever it tries to resolve the current path. but this check causes it to fail, because `ud+0` is a nonexistent case. this uses he-beam to transform the beam in the conditional to see if the case is 0, and if it is, changes the case to da+now before it scries
This allows you to pass a thread directly into khan, instead of passing
a filename. This has several implications:
- The friction for using threads from an app is significantly lower.
Consider:
=/ shed
=/ m (strand ,vase)
;< ~ bind:m (poke:strandio [our %hood] %helm-hi !>('hi'))
;< ~ bind:m (poke:strandio [our %hood] %helm-hi !>('there'))
(pure:m !>('product'))
[%pass /wire %arvo %k %lard %base shed]
- These threads close over their subject, so you don't need to parse
arguments out from a vase -- you can just refer to them. The produced
value must still be a vase.
++ hi-ship
|= [=ship msg1=@t msg2=@t]
=/ shed
=/ m (strand ,vase)
;< ~ bind:m (poke:strandio [ship %hood] %helm-hi !>(msg1))
;< ~ bind:m (poke:strandio [ship %hood] %helm-hi !>(msg2))
(pure:m !>('product'))
[%pass /wire %arvo %k %lard %base shed]
- Inline threads can be added to the dojo, though this PR does not add
any sugar for this.
=strandio -build-file %/lib/strandio/hoon
=sh |= message=@t
=/ m (strand:rand ,vase)
;< ~ bind:m (poke:strandio [our %hood] %helm-hi !>('hi'))
;< ~ bind:m (poke:strandio [our %hood] %helm-hi !>(message))
(pure:m !>('product'))
|pass [%k %lard %base (sh 'the message')]
Implementation notes:
- Review the commits separately: the first is small and implements the
real feature. The second moves the strand types into lull so khan can
refer to them.
- In lull, I wanted to put +rand inside +khan, but this fails to that
issue that puts the compiler in a loop. +rand depends on +gall, which
depends on +sign-arvo, which depends on +khan. If +rand is in +khan,
this spins the compiler. The usual solution is to either move
everything into the same battery (very ugly here) or break the
recursion (which we do here).
Before this, the %watch to eth-watcher was happening before the %poke,
and so eth-watcher was responding with its entire history immediately.
This is bad because it takes a lot of memory to process that many logs,
and also because those logs are stale.
Now, the %poke happens first, which clears the history.
%kick is supposed to start back from the snapshot and move forward.
Without this, we would only fetch logs that we hadn't already fetched.
Thus, if you were up-to-date when you kicked, you would miss anything
that happened between the time the snapshot was taken and the present,
though you would see things after the present.
Also reverted lull change to make this a safer upgrade.
Previously, the initial Azimuth snapshot was stored in Clay and shipped
in the pill. This causes several problems:
- It bloats the pill
- Updating the snapshot added large blobs to Clay's state. Even now
that tombstoning is possible, you don't want to have to do that
regularly.
- As a result, the snapshot was never updated.
- Even if you did tombstone those files, it could only be updated as
often as the pill
- And those updates would be sent over the network to people who didn't
need them
This moves the snapshot out of the pill and refactors Azimuth's
initialization process. On boot, when app/azimuth starts up, it first
downloads a snapshot from bootstrap.urbit.org and uses that to
initialize its state. As before, updates after this initial snapshot
come from an Ethereum node directly and are verified locally.
Relevant commands are:
- `-azimuth-snap-state %filename` creates a snapshot file
- `-azimuth-load "url"` downloads and inits from a snapshot, with url
defaulting to https://bootstrap.urbit.org/mainnet.azimuth-snapshot
- `:azimuth &azimuth-poke-data %load snap-state` takes a snap-state any
way you have it
Note the snapshot is downloaded from the same place as the pill, so this
doesn't introduce additional trust beyond what was already required.
When remote scry is released, we should consider allowing downloading
the snapshot in that way.
using col as a seperate made it look like a bunch of =<, which doesn't
make sense for e.g. pritning chapters. paths aren't quite right either,
so we don't use +stap, and just want identifiers separated by fas
without any leading fas
Includes patched versions of ames' and clay's +load arms.
In clay, we do a dumb ;; hack to get the state to adapt properly. This
shouldn't be needed ($case had an extra... case added to it, old ones
should still nest), and so we should revisit the logic there to make it
cleaner/better before release.
It's too easy to get trapped in the dojo %dy-edit-busy escape room. Just
type something like:
-build-file /=base/gen/ls/hoon
This modifies the dojo output to tell the user how to get out.
Fixes#1462.
Incorporates @Fang- suggested changes (thanks!).
Drops the path serialization as it will print on two separate lines,
and it is already displayed in dojo immediately above the error message:
> =dir /=base=/ge
dojo: dir does not exist
If there were pending-logs in an existing watchdog that was not fully
restarted, and the number of the starting block is newer than the first
in pending, when starting a new thread, those logs will be carried over
to the new thread, which will then be re-downloaded and will fail to be
verified in /lib/naive
Only "shortmoons" though, due to some ames lane size limitation which
makes encoding longer ships difficult.
Notably the -ph-moon-az test does not pass, the moon cannot talk to a
non-sponsor galaxy.
We had trie operations independently implemented in +de in arvo,
+an:cloy in zuse, +zu in clay, lib/trie, and app/spider. This unifies
them all into +de in arvo, aggregating the used operations.
And inject their latest keys as soon as we pull them from cache. This
way, we avoid having to do the whole boot sequence again just for a
modified dawn event.
addresses #5442 by adding %thread-done and %thread-fail marks. also
fixes await-thread:strandio and removes some blank lines from
app/spider.hoon
%thread-done loses the type of the result, so you'll need to use ;; to
get it back. the real way to fix this is to have threads produce cages
instead of vases
Small touch-ups to simulation behavior and ph tests. Most of them pass
now, even if they're still really slow at times.
The breach ones don't pass, but also complain of dangling bone, so might
work once the fix for that is in.
If a batch gets bigger than a max size defined by the ethereum node
the raw transaction is sent to, the /ted/roller/send thread will crash and
the batch will be blocked, stopping any subsequent batches to be sent.
This detects when the current batch reaches a certain threshold and only
includes transactions up to that point, moving the ones that are not sent
back to the pending queue, adjusting their history and finding status.
This gives us much higher cycle times when retrying tests over and over,
since we can just fast-forward past the boot sequence, which never
changes unless we change the pill.
Too often when dealing with big types the compiler traces and other such
outputs become hard to read. Wrapping a type as $+(shorthand big-type)
will now print #shorthand in place of the type.
Too often when dealing with big types the compiler traces and other such
outputs become hard to read. Wrapping a type as $+(shorthand big-type)
will now print #shorthand in place of the type.
If a generator has a named argument named "drum-session", and the caller
doesn't specify a value for it, dojo auto-fills it with the current drum
session identifier.
We update the sole protocol to more cleanly support multiple sessions.
Primarily, the "sole id" is updated to be a [@p @ta] instead of a @ta,
and it is now generated based off the connected dill session, rather
than statically.
This change ripples out to applications that support the sole protocol:
the subscription path becomes /sole/[ship]/[session] (as opposed to
/sole/[per-ship-constant]), and %sole-action pokes include the new id as
well.
For shoe agents, this means (at the very least) updating the function
signatures of the shoe arms.
/lib/sole has been updated to include helper functions for parsing a
sole-id from a subscription path, and turning a sole-id into its
corresponding path. It also has a function to aid in migrating old
sole-ids.
Existing sole agents are made to kick any known open sessions, forcing a
resubscribe by drum, so that they may use exclusively the new format
going forward. Third-party agents are recommended to do the same.
Note that some functionality, such as |link, still operates exclusively
on the default session. Improvements in this area to follow soon.
Resolves a good number of conflicts. Most notably, re-propagates removal
of gall's %onto, confirms new /app/herm behavior, coerces hood/drum
state adapters back into place, and updates webterm to use the latest
api.
The previous value—used for testing—didn't consider
block reorgs, which meant that if we zoom to the latest
block that has no transactions, but that gets later replaced
by a 1-block reorg that does have a transaction, we'll miss it,
making our Azimuth state incomplete.
To fix it, we rewind the Azimuth state to the contents of the snapshot,
and then start retrieving logs from the latest one we have.
* master:
[nix] track nixos-21.11 branch whenever niv updates, bump to include qemu-in-virtualization fix
webterm: v1.0.0
herm: permission checks
herm: avoid trailing empty path segments
pmnsh: update secp256k1 configure flags
secp256k1: use nixpkgs provided secp256k1 and add to sources-pmnsh
webterm: remove border, let term live in page
theme: cleaning up a few mismatches
webterm: handle old-style blits and belts
build: correct lmdb static builds
build: explicitly override h2o build platforms to support darwin
ci: upgrade cachix/install-nix-action from v13 -> v16
build: remove haskell related nix code and haskell.nix dependency
webterm: update imports
@urbit/api: move term types
webterm: fix broken imports
webterm: update package name for lerna
webterm: commit missing api files
webterm: backport
This adds support for handling cases where the send-batch thread failed,
mainly among them, a thread crash. One of the events that causes this
behavior is a ver low gas price for this L1 transaction.
Here we add support for manually bumping the price for such transaction,
and for increasing the default fallback gas-price, together with discarding
any malformed batch from the sending queue.
Also kick the call to +mule out of the loop. By uncommenting the
diagnostics in u3m_fall, I measured that running through the 290k events
the azimuth snapshot required this much memory:
Head recursive, +mule in: 1.1GB
Head recursive, +mule out: 780MB
Tail recursive, +mule in: 700MB
Tail recursive, +mule out: 70MB
So this commit chooses the last one. The most delicate part is making
sure the effects are the right order; this uses the usual idiom.
Kicking +mule out of the loop is okay because lib/naive should never
fail, and if it does then azimuth shouldn't advance until an out-of-band
solution is decided.
Addresses #5431
