This is sufficient to bring the normal (non-prerelease-bugged) cases
into the new world.
For the prerelease ships that ran a buggier version of the new gall
subscription logic, we note that the conditional may trigger for the
nonce=1 case where it had already triggered for their
(shouldn't-be-possible) nonce=0 case. This results in a %leave on a wire
that wasn't in use. This no-ops on the publisher side though, and the
flow gets corked right away, so this is considered harmless.
In response to clog notification from remote ames, we were sending a
%cork to clean up the flow. However, the wire we were using had the /sys
prefix already stripped off. Here, we put it back in.
Start by killing subscription nonce 0, then work our way up instead of
down. We enhance the printf with a "total nonces" indicator so we can
still easily see the progress being made.
Previous +ap-doff kicked the agent repeatedly. We needed to kick
it only once. Now publisher agents clear their incoming subscription
state without the subscriber making lots of new subscriptions because
of repeated kicking.
+on-plea gets called in two very different ways:
1) handling request from local vane to send %plea to peer
2) handling %cork request from another ship, which our local ames has %pass'ed
to ourselves
In the second case, we shouldn't print misleadingly, or bind a duct in the ossuary.
+ap-nuke was not including the nonce, but should.
+ap-handle-peers was potentially including a zero nonce.
(The latter shouldn't have been possible, but there's a bug in +load
where sub-nonce.yoke gets initialized as 0 instead of 1.)
Gall tells ames to %cork flows for subscriptions it has closed.
Receiving a kick also closes a subscription, but gall wasn't issuing a
%cork in that case. We correct that here.
Inlines +mo-handle-ames-response's logic at its only callsite.
seems that this structure has been unused since
e75ab631a4 and confuses
newbies trying to figure out exactly what the commit
structure is (which is how I came across this)
Without this, a ship would send a cork on a max of one flow per
recork timer, which could take years to clear for some ships.
This starts a hot loop of trying the next cork once one gets
positively acked.
The previous recork timer queued up %cork messages without sending them.
It also relied on making sure pump timers didn't get set for recork bones.
This was fragile.
The new design enqueues up to one new %cork message per ship during each
recork timer, based on the state of the flow. If the flow is closing but
there are no outstanding messages in it, then it needs to be recorked.
Flows will be recorked in ascending numerical order by bone.
The condition got butchered during refactor: instead of avoiding the creation
of pump timers during recork wake, it was setting them _exclusively_ during
recork wake.
This test started failing presumably somewhere during #5886. Testing
with a comet on the network, the test seems inaccurate: the comet can
communicate and be communicated to just fine.
this refactors the parser for %brcn and %brpt to separate the optional
argument(s) from the required argument(s).
also adds +blab, which allows for a minor refactor of a couple other
arms as well as being used for %brcn and %brpt
Problem:
by-channel has its own copy of server-state from line 2182. discard-channel returns an altered state, with one channel removed from the state of by-channel.
but the state of by-channel isn't changing with each iteration, so |trim is only removing one channel per invocation.
Solution:
update by-channel on each iteration.
this commit replaces the previous intermediate parsing structure, $whit,
with a new one better suited for batch comments and taking into account
that {# %label} syntax is no longer being used anywhere. basically,
this makes it so that all doccords are batch comments, where if they are
preceded by a (list link) then they will try to attach to the given
link (only utilizes first link for now), and a blank link means it will
try to attach to the following hoon or spec
This is a temporary fix, and first part of the gall-request-queue-fix
release in two stages. This gives a publisher ship the ability to
understand a %cork and handle it properly, but no subscriber will
be sending %corks at this stage when leaving a subscription.
We still add a nonce to all subscription wires but it doesn't
increment it when resubscribing, allowing flows to be reused.
Tested locally with toy pub/sub agents and Group join/leaving
Previously, the initial Azimuth snapshot was stored in Clay and shipped
in the pill. This causes several problems:
- It bloats the pill
- Updating the snapshot added large blobs to Clay's state. Even now
that tombstoning is possible, you don't want to have to do that
regularly.
- As a result, the snapshot was never updated.
- Even if you did tombstone those files, it could only be updated as
often as the pill
- And those updates would be sent over the network to people who didn't
need them
This moves the snapshot out of the pill and refactors Azimuth's
initialization process. On boot, when app/azimuth starts up, it first
downloads a snapshot from bootstrap.urbit.org and uses that to
initialize its state. As before, updates after this initial snapshot
come from an Ethereum node directly and are verified locally.
Relevant commands are:
- `-azimuth-snap-state %filename` creates a snapshot file
- `-azimuth-load "url"` downloads and inits from a snapshot, with url
defaulting to https://bootstrap.urbit.org/mainnet.azimuth-snapshot
- `:azimuth &azimuth-poke-data %load snap-state` takes a snap-state any
way you have it
Note the snapshot is downloaded from the same place as the pill, so this
doesn't introduce additional trust beyond what was already required.
When remote scry is released, we should consider allowing downloading
the snapshot in that way.
using col as a seperate made it look like a bunch of =<, which doesn't
make sense for e.g. pritning chapters. paths aren't quite right either,
so we don't use +stap, and just want identifiers separated by fas
without any leading fas
Includes patched versions of ames' and clay's +load arms.
In clay, we do a dumb ;; hack to get the state to adapt properly. This
shouldn't be needed ($case had an extra... case added to it, old ones
should still nest), and so we should revisit the logic there to make it
cleaner/better before release.
changes $whit to have a (unit link) instead of (unit term). this holds
the identifier for where a comment is supposed to go. changes to parsers
in docs:vast to accomodate this.
this only allows for batch comments written for arms within a given
core. someday, the feature should allow you to write comments
virtually anywhere. the (unit link) in $whit should become a (unit (list
link)) to accommodate this
Because the publisher will send the cork plea back to the subscriber on
the next bone, we are not able to know the bone for the original cork.
To handle it, we add the cork bone to the plea path
still wip: it keeps resending the cork plea faster than its ~h1 timer
Removes the !: at the top of gall, so that it no longer gets included in traces about agent builds or crashes.
We also refine intentional crashes with ~_s, so that we still see a crash reason even if we don't get a full trace.
Lastly, flops the trace for +on-load crashes, which were getting printed bottom-first.
It's too easy to get trapped in the dojo %dy-edit-busy escape room. Just
type something like:
-build-file /=base/gen/ls/hoon
This modifies the dojo output to tell the user how to get out.
Fixes#1462.
Incorporates @Fang- suggested changes (thanks!).
Drops the path serialization as it will print on two separate lines,
and it is already displayed in dojo immediately above the error message:
> =dir /=base=/ge
dojo: dir does not exist
this is almost a revert of the commit that added in the %funk tag, but
not quite, so its a new commit. i apparently forgot that product docs
are actually built by +wrap, not +boog, so it was treating postfix
arm-docs as if they were product docs
i thought this would be a neat feature but ted called it too clever and
probably not what you really want. this code is atrocious though and
needs some serious cleanup
not actually sure this is correct yet, but it fixed the issue where
there would be a crash when looking for docs in an arm like +bar in
|%
++ foo 'foo'
++ bar foo
--
Regularized arm names to +etch-* and +sift-* to match Vere. Renamed
$packet to $shot. Used $meow, $purr, and $keen to match Vere's naming.
Reorganized packet decoding arms and moved some to Lull for later use in
Aqua.
If a desk already existed, we would crash the generator, embedding a
message in the resulting stack trace.
To improve legibility, we turn this into an %ask generator, prompting
only if we are about to overwrite an existing desk.
We also update the =force argument to be =hard instead, to match |nuke.
Stale lanes may cause forwarding loops. Imagine the following:
1) Planet A is live. Galaxy B, its indirect sponsor, learns of its route.
2) A goes offline. Another ship, C, is started in its place, at the same route.
3) B receives a packet for A, forwards it to the known route.
4) C received the packet, forwards it to B.
5) Repeat from 3.
Here, we update the forward lane(s) scry used by the runtime to not produce a
peer's lane if they haven't communicated with us in the last hour. Everyone's
supposed to ping their sponsorship chain every 30 seconds. If those aren't
going through, you shouldn't expect to be reachable anyway.
We may or may not want to update +send-blob to match.
Previously, if the pointer for a syntax error pointed to the end of the file
(and the file ended in a newline) the code snippet rendering would try to
display a line _beyond_ the end of the file, causing a crash.
Here, we detect that case, and display `<<end of file>>` instead.
Changes the test command to check if args contains a single path and
wraps it in a list. Now a test thread can be started without providing
a list:
-test %/tests/lib
And passing a list still works:
test %/tests/lib ~
One nit in this change is the lark expression to reach into args: if
args is ~ then instead of getting a useful need/have error, the test
ends in a "false-start". Perhaps this could be fixed by detecting ~
and setting it to %/tests.
Instead of having a separate "busy" flag and pending scry request state,
we now have a single "busy" unit that, if there is a pending request,
contains details about the pending request. In the ames case, this is
simply %ames. But in the scry case, it contains all the details we need
to cancel the request, timer, or both when needed.
Additionally, we now make sure to always cancel the scry timeout timer
whenever we get a scry response.
Previously, when the scry timeout fired and we retried the request with
ames, we would always start over from the very first request (ie,
fetching the ""manifest"" containing the files we need to download).
This behavior is not correct in the case where we had already received
that initial response, and were now fetching individual files. So here,
if that's the case, we simply call into +work after marking this peer as
having known-broken scry, which will resume work as appropriate.
(The actual bug here that broke the fallback behavior entirely was the
busy flag not getting unset. We now clean it up properly.)
We also move setting of the sad timer into +retry-with-ames, instead of
doing it at each individual callsite. (In fact, one of the callsites was
missing this behavior.)
This accounts for a possible race condition where ames expects a
response, but regresses into the larval state. Upon receiving the
$sign on +take, we would remain stuck as a larva. Now we check
that we have enough information to re-evolve and then start a
/larval timer to begin draining the queue.
When +ke-abet-gone gets called, we're going to remove this keen
entirely. +ke-set-wake does a whole song-and-dance and may even set a
new timer. So instead, we simply call +ke-rest, if we have a known
outstanding timer.
Previously we stored the nonce in $boat, which changed the $bowl of each
agent. This compiles and all agents reload, but more testing is needed.
It also renames inbound/outbound watches to $bitt/$boat.
Similar to how %x on the empty desk was already treated specially. We
continue supporting %d on non-empty desks, but do add a print marking it
as deprecated.
The previous commit already started treating the 0v0 commit as
equivalent to the 0 aeon (that is, never-real root), so we should give
the 0v0 commit for all 0 aeons.
If there were pending-logs in an existing watchdog that was not fully
restarted, and the number of the starting block is newer than the first
in pending, when starting a new thread, those logs will be carried over
to the new thread, which will then be re-downloaded and will fail to be
verified in /lib/naive
Instead of scrying for revisions of files at paths in desks, we now scry
for data corresponding to a given lobe. This removes concerns of aeons,
paths, and other such frivolities, and lets us ask for the specific data
we need regardless of where it may live.
How we're supposed to answer permissioning questions around data this
way remains to be seen.
Previously we were dropping events that used old
wires that lacked a rift in them. This seems a
bad behavior because we don't want to destroy a
flow that has not been processed by both ends.
Note: pending a fix to test-old-ames-wire
This converts the blob store from having deltas, directs, and
tombstones, to just having direct pages. This simplifies a lot of code,
since we don't have to constantly ensure that deltas always have their
parent available.
This removes the hardcoded text diff logic from clay, which was
previously required for bootstrapping.
Over the wire, we handle both old and new requests and responses
transparently, so communication is normal in both directions across
ships which do or do not have this change.
the logic for getting the docs on the default arm of a core is now more
similar to how other arm docs are gotten, rather than having its own
system. there is still more room for improvement
this was wiping out some comments buried that weren't written at the top
of an arm. not sure if this is used in another fashion that will create
issues, so I just commented it out to help remember that it used to be
there in case it needs further changes, like if I should actually skip
%know and %made notes but not %help notes
Only "shortmoons" though, due to some ames lane size limitation which
makes encoding longer ships difficult.
Notably the -ph-moon-az test does not pass, the moon cannot talk to a
non-sponsor galaxy.
We had trie operations independently implemented in +de in arvo,
+an:cloy in zuse, +zu in clay, lib/trie, and app/spider. This unifies
them all into +de in arvo, aggregating the used operations.
before this, it was grabbing the initial arm-doc from the AST rather
than the type. now %arm items have all 3 types of docs available. the
interface has been degraded somewhat though, as %arm items no longer
have a single docs field. more refactoring will be needed to figure out
the best way to do this.
rewrites select-arm-docs so that it checks for nested hint types and
sees if the outermost help hint has a %funk link with the name of the
arm in order to tell that its an arm-doc
docs written above an arm are now distinguishable in the AST from
docs written above the product of the arm, by tagging docs written
above the arm with a %funk link
Having the origin at the top left instead of the bottom left is more
conventional and ergonomic. The only thing this complicates is
prompt-specific logic, where we care about the coordinates of the
bottom-most line on the screen.
For that reason, the bulk of the changes here are in vere, where we
treat the bottom-most line specially, drawing the spinner onto it.
Webterm is likewise updated to account for the new coordinate system.
Drum now opts to accept clicks anywhere on the screen, and does its best
to move the cursor as close to the clicked location as possible (within
the confines of the prompt).
initial commit for library for finding and printing doccords. has some
basic functionality for looking through a type and finding the docs
within it and printing them, but is mostly unfinished
And inject their latest keys as soon as we pull them from cache. This
way, we avoid having to do the whole boot sequence again just for a
modified dawn event.
This crashed at runtime when `a` is null because it tried to instantiate
`meg`, and that means bunting its argument, and that calls `node`, where
the assert failed.
Wet gates use the bunt of their formal argument, so we use that.
addresses #5442 by adding %thread-done and %thread-fail marks. also
fixes await-thread:strandio and removes some blank lines from
app/spider.hoon
%thread-done loses the type of the result, so you'll need to use ;; to
get it back. the real way to fix this is to have threads produce cages
instead of vases
%rez has always used "width & height". Certainly, "x & y" is more
standard than "row & column". As such, we settle on making %hop and %hit
respect the more natural ordering. This change is safe because these
interfaces haven't made it to livenet yet.
Intended use is for transitory moons to be able to breach themselves on
startup.
If you run a moon without persistence, then every time the program is
restarted, it must be breached. This lets the moon breach itself
instead of requiring direct interaction with the planet. The moon
should reserve the first bone for this purpose, and then every time it
starts up, it should send [%helm-moon-breach ~moon-name] to hood on the
planet.
Small touch-ups to simulation behavior and ph tests. Most of them pass
now, even if they're still really slow at times.
The breach ones don't pass, but also complain of dangling bone, so might
work once the fix for that is in.
This fixes a space leak where the entire ford/clay core would be
included in the ford cache. Heavily reduces memory usage by clay,
reducing total usage from 100-200MB to around 10MB.
Instead of reporting a single memory size for built files, marks and
conversions, we now report memory size per path, mark name and mark
pair, respectively.
If a batch gets bigger than a max size defined by the ethereum node
the raw transaction is sent to, the /ted/roller/send thread will crash and
the batch will be blocked, stopping any subsequent batches to be sent.
This detects when the current batch reaches a certain threshold and only
includes transactions up to that point, moving the ones that are not sent
back to the pending queue, adjusting their history and finding status.
This gives us much higher cycle times when retrying tests over and over,
since we can just fast-forward past the boot sequence, which never
changes unless we change the pill.
scries %azimuth for logs.state, filters the L2 events, and then gets
their timestamps. ought to use ted/eth/get-timestamps but doesn't due to
issues with spawning child threads, but there's probably a way to do it
without child threads
This changes the parser for +tall so that it looks before and after a
hoon for doccords, and then extracts a label for %brcn if it exists.
+wrap will be used to annotating most hoons, but this commit only covers
%brcn
Too often when dealing with big types the compiler traces and other such
outputs become hard to read. Wrapping a type as $+(shorthand big-type)
will now print #shorthand in place of the type.
Too often when dealing with big types the compiler traces and other such
outputs become hard to read. Wrapping a type as $+(shorthand big-type)
will now print #shorthand in place of the type.
+balk defines a datastructure for a parsed remote scry path format in
addition to conversion gates to and from paths.
The new scry path format is /~ship/rift/life/vane/care/spur. Note that
desk no longer appears in the scry path format and should instead be
encoded as the first item of the spur.
instead make it a rolling 128-bit integer. 128 bits is the same size as
the +sham space, so is one natural choice for "big enough to never have
to think about this." 64, 32, 16, even 8 bits would probably be fine.
Threads should eventually take and produce $cage instead of $vase. Since
%khan is likely to be used by third parties, we write to the eventual
intended API. We ignore the mark on the input $cage (it is safe to
always specify %noun), and we always use %noun as the output mark.
%fyrd now makes more sense. It was previously discarding the type of the
output %arow and re-encoding the raw noun as a vase of the output mark;
it is now performing mark conversion from the mark of the output $cage
to the originally requested output mark.
nara: swedish for near, provides a trace from the start of the nearest virtualization
to the statement wrapped by the nara hint.
hela: swedish for whole, provides a trace from the start of the event
to the statement wrapped by the hela hint.
The eventual goal is to present these traces in a way that is most salient & useful to developers
and in a way that is aesthetically pleasing as the bout hint is useful and nice looking.
For this pull request the goal is to provide the actual functionality in code that we can feel
safe and comfortable adding to the master branch. To this end, I've added a trivial set of tests for
nara and hela in pkg/arvo/tests/run/hints.hoon - they only prove that invoking these hints will not
crash the runtime, but I'm open to ideas for how I can test these hints further.
If a generator has a named argument named "drum-session", and the caller
doesn't specify a value for it, dojo auto-fills it with the current drum
session identifier.
Use the correct wires when cancelling the scry request and/or its
timers.
Note that we may produce more %rests and %yawns than strictly necessary,
but these no-op cleanly in those cases.
We update the sole protocol to more cleanly support multiple sessions.
Primarily, the "sole id" is updated to be a [@p @ta] instead of a @ta,
and it is now generated based off the connected dill session, rather
than statically.
This change ripples out to applications that support the sole protocol:
the subscription path becomes /sole/[ship]/[session] (as opposed to
/sole/[per-ship-constant]), and %sole-action pokes include the new id as
well.
For shoe agents, this means (at the very least) updating the function
signatures of the shoe arms.
/lib/sole has been updated to include helper functions for parsing a
sole-id from a subscription path, and turning a sole-id into its
corresponding path. It also has a function to aid in migrating old
sole-ids.
Existing sole agents are made to kick any known open sessions, forcing a
resubscribe by drum, so that they may use exclusively the new format
going forward. Third-party agents are recommended to do the same.
Note that some functionality, such as |link, still operates exclusively
on the default session. Improvements in this area to follow soon.
tid was accidentally getting set to the name of the output mark. As we
don't currently support cancelling threads, there is no reason to
maintain the originally-intended correspondence between tid and conn
request-id.
Take the opportunity to clean up indentation somewhat.
This can occur after upgrade, or when receiving a %delta blob. We do not
know the path or commit timestamp of a file that resolves to the blob,
so we simply fall back to old ames-style blob-by-lobe fetching.
In +work, we simply want to fetch file data, regardless of what the
overarching request is. (In fact, %sing would likely never hit this path
anyway.)
In order to be able to make the scry request, we track the timestamp of
the commit that contains a blob we're missing, and scry for that %da
revision.
There's two edge cases where we cannot immediately know the timestamp
that are currently assumed broken. Fix soon.
%v requests are already handled specially for the ames case. We should
continue to respect that, only doing those kinds of requests over ames.
Also cancels the scry timer after receiving a response, instead of not
doing that.
Ancient version upgraders beware! If you're coming from a home-based
arvo, you must first upgrade to the version prior to this, or else
you'll be in trouble!
When clay wants to download blobs from a foreign ship, it attempts doing
this using the new remote scry protocol.
If it doesn't receive a response within ~m1, it falls back to using the
old ames-based syncing instead. We remember this "prefer ames" state for
the specific ship for an hour, after which we'll begin trying the scry
flow again.
Compiles, but untested. Some TODOs and REVIEWs remaining herein.
We intentionally leave the dist-upgraded flag in state to avoid
cluttering the diff here. The next commit will remove it.
Also strips out `$` from khan top-level comment.
There are arguments for keeping $crag in lull, and on the other side for
moving $cast to arvo. This seemed like the most reasonable approach.
%fyrd is now implemented in terms of %fard, and likewise %avow in terms
of %arow. State is tracked via wire rather than in a global map.
Unit tests adjusted to match.
These take and produce vases, and assign random tids (rather than
deducing them from the input duct.)
Since %fard does not require mark conversion, we make the mark/beak on
$thread-state optional (and use this to decide whether to send %avow or
%arow.) Provide a state adapter since it's possible that people have
been experimenting with this vane.
This makes the negative case of %avow/%arow kind of clunky, since there
is no content difference, but the following does not seem possible
within the Hoon type system:
=/ gif
?~ p.tad
%arow %avow
[hen %give gif %| p.cag tang]~
- use desk parameter instead of %base everywhere
- formatting clean up
- make |story-remove take a case instead of an aeon
- make desk param optional for story-set and story-log
+sign:schnorr crashes on `=(0 sk)`, so the bounds checking code is not
exercised for sk=0. It also crashes on `(gte sk n.domain.c)`, which is
redundant with the size check on sk, so we remove that.
Since we cannot scry the key out of jael during load, and the
alternative is defering this logic through the larval core in some way,
we simply include the type for the old acru interface, letting us reuse
the old core to initialize the new one.
Instead of exporting keys so that caller can do this themselves, we
expose arms for signing and authenticating that produce and operate on
just the signature, without mangling it into the message.
;;(vase ...) does a nest-check of the type of the kernel. This is
undesirable, so we instead run everything through +slum and cast the
result to +tang.
