Previously, if the pointer for a syntax error pointed to the end of the file
(and the file ended in a newline) the code snippet rendering would try to
display a line _beyond_ the end of the file, causing a crash.
Here, we detect that case, and display `<<end of file>>` instead.
Instead of having a separate "busy" flag and pending scry request state,
we now have a single "busy" unit that, if there is a pending request,
contains details about the pending request. In the ames case, this is
simply %ames. But in the scry case, it contains all the details we need
to cancel the request, timer, or both when needed.
Additionally, we now make sure to always cancel the scry timeout timer
whenever we get a scry response.
Previously, when the scry timeout fired and we retried the request with
ames, we would always start over from the very first request (ie,
fetching the ""manifest"" containing the files we need to download).
This behavior is not correct in the case where we had already received
that initial response, and were now fetching individual files. So here,
if that's the case, we simply call into +work after marking this peer as
having known-broken scry, which will resume work as appropriate.
(The actual bug here that broke the fallback behavior entirely was the
busy flag not getting unset. We now clean it up properly.)
We also move setting of the sad timer into +retry-with-ames, instead of
doing it at each individual callsite. (In fact, one of the callsites was
missing this behavior.)
This accounts for a possible race condition where ames expects a
response, but regresses into the larval state. Upon receiving the
$sign on +take, we would remain stuck as a larva. Now we check
that we have enough information to re-evolve and then start a
/larval timer to begin draining the queue.
When +ke-abet-gone gets called, we're going to remove this keen
entirely. +ke-set-wake does a whole song-and-dance and may even set a
new timer. So instead, we simply call +ke-rest, if we have a known
outstanding timer.
Previously we stored the nonce in $boat, which changed the $bowl of each
agent. This compiles and all agents reload, but more testing is needed.
It also renames inbound/outbound watches to $bitt/$boat.
Similar to how %x on the empty desk was already treated specially. We
continue supporting %d on non-empty desks, but do add a print marking it
as deprecated.
The previous commit already started treating the 0v0 commit as
equivalent to the 0 aeon (that is, never-real root), so we should give
the 0v0 commit for all 0 aeons.
Instead of scrying for revisions of files at paths in desks, we now scry
for data corresponding to a given lobe. This removes concerns of aeons,
paths, and other such frivolities, and lets us ask for the specific data
we need regardless of where it may live.
How we're supposed to answer permissioning questions around data this
way remains to be seen.
Previously we were dropping events that used old
wires that lacked a rift in them. This seems a
bad behavior because we don't want to destroy a
flow that has not been processed by both ends.
Note: pending a fix to test-old-ames-wire
This converts the blob store from having deltas, directs, and
tombstones, to just having direct pages. This simplifies a lot of code,
since we don't have to constantly ensure that deltas always have their
parent available.
This removes the hardcoded text diff logic from clay, which was
previously required for bootstrapping.
Over the wire, we handle both old and new requests and responses
transparently, so communication is normal in both directions across
ships which do or do not have this change.
We had trie operations independently implemented in +de in arvo,
+an:cloy in zuse, +zu in clay, lib/trie, and app/spider. This unifies
them all into +de in arvo, aggregating the used operations.
This fixes a space leak where the entire ford/clay core would be
included in the ford cache. Heavily reduces memory usage by clay,
reducing total usage from 100-200MB to around 10MB.
Instead of reporting a single memory size for built files, marks and
conversions, we now report memory size per path, mark name and mark
pair, respectively.
instead make it a rolling 128-bit integer. 128 bits is the same size as
the +sham space, so is one natural choice for "big enough to never have
to think about this." 64, 32, 16, even 8 bits would probably be fine.
Threads should eventually take and produce $cage instead of $vase. Since
%khan is likely to be used by third parties, we write to the eventual
intended API. We ignore the mark on the input $cage (it is safe to
always specify %noun), and we always use %noun as the output mark.
%fyrd now makes more sense. It was previously discarding the type of the
output %arow and re-encoding the raw noun as a vase of the output mark;
it is now performing mark conversion from the mark of the output $cage
to the originally requested output mark.
Use the correct wires when cancelling the scry request and/or its
timers.
Note that we may produce more %rests and %yawns than strictly necessary,
but these no-op cleanly in those cases.
tid was accidentally getting set to the name of the output mark. As we
don't currently support cancelling threads, there is no reason to
maintain the originally-intended correspondence between tid and conn
request-id.
Take the opportunity to clean up indentation somewhat.
This can occur after upgrade, or when receiving a %delta blob. We do not
know the path or commit timestamp of a file that resolves to the blob,
so we simply fall back to old ames-style blob-by-lobe fetching.
In +work, we simply want to fetch file data, regardless of what the
overarching request is. (In fact, %sing would likely never hit this path
anyway.)
In order to be able to make the scry request, we track the timestamp of
the commit that contains a blob we're missing, and scry for that %da
revision.
There's two edge cases where we cannot immediately know the timestamp
that are currently assumed broken. Fix soon.
%v requests are already handled specially for the ames case. We should
continue to respect that, only doing those kinds of requests over ames.
Also cancels the scry timer after receiving a response, instead of not
doing that.
Ancient version upgraders beware! If you're coming from a home-based
arvo, you must first upgrade to the version prior to this, or else
you'll be in trouble!
When clay wants to download blobs from a foreign ship, it attempts doing
this using the new remote scry protocol.
If it doesn't receive a response within ~m1, it falls back to using the
old ames-based syncing instead. We remember this "prefer ames" state for
the specific ship for an hour, after which we'll begin trying the scry
flow again.
Compiles, but untested. Some TODOs and REVIEWs remaining herein.
We intentionally leave the dist-upgraded flag in state to avoid
cluttering the diff here. The next commit will remove it.
Also strips out `$` from khan top-level comment.
There are arguments for keeping $crag in lull, and on the other side for
moving $cast to arvo. This seemed like the most reasonable approach.
%fyrd is now implemented in terms of %fard, and likewise %avow in terms
of %arow. State is tracked via wire rather than in a global map.
Unit tests adjusted to match.
These take and produce vases, and assign random tids (rather than
deducing them from the input duct.)
Since %fard does not require mark conversion, we make the mark/beak on
$thread-state optional (and use this to decide whether to send %avow or
%arow.) Provide a state adapter since it's possible that people have
been experimenting with this vane.
This makes the negative case of %avow/%arow kind of clunky, since there
is no content difference, but the following does not seem possible
within the Hoon type system:
=/ gif
?~ p.tad
%arow %avow
[hen %give gif %| p.cag tang]~
- use desk parameter instead of %base everywhere
- formatting clean up
- make |story-remove take a case instead of an aeon
- make desk param optional for story-set and story-log
Since we cannot scry the key out of jael during load, and the
alternative is defering this logic through the larval core in some way,
we simply include the type for the old acru interface, letting us reuse
the old core to initialize the new one.
Instead of exporting keys so that caller can do this themselves, we
expose arms for signing and authenticating that produce and operate on
just the signature, without mangling it into the message.
- only store metadata in the persistent map. just enough to support
(eventual) thread cancellation and output mark lookup.
- try to delete thread state at other failure points not covered by
%kick.
- reflect back the passed output mark rather than form.dais. not sure
about this one yet.
Resolves a good number of conflicts. Most notably, re-propagates removal
of gall's %onto, confirms new /app/herm behavior, coerces hood/drum
state adapters back into place, and updates webterm to use the latest
api.
By moving this behavior for packets with ames-style headers into lull,
other vanes that want to do protocols in similar style (like the upcoming
%fine vane) can reuse the logic here.
Note that we parameterize the main en- and decoding functions, so that
we may pass in the "is ames" flag and protocol versions as appropriate
to the context.
This change greatly improves the ergonomics of working with channel JSON
in statically typed languages, as the polymorphism is moved out of the
actual diff and into the event framing.
+wake had accumulated several layers of abstractions which were later
rendered unnecessary. This removes those abstractions and should have
no semantic effect.
This adds support for tombstoned files to clay. It does not include any
way to actually tombstone them; that is left for later.
This allows tombstoning at the level of a file. Precisely, this expands
+blob:clay by adding a %dead case:
+$ blob :: fs blob
$% [%delta p=lobe q=[p=mark q=lobe] r=page] :: delta on q
[%direct p=lobe q=page] :: immediate
[%dead p=lobe ~] :: tombstone
== ::
Thus, we maintain the invariant that every lobe corresponds to a blob,
but now a blob may be an explicit tombstone.
Details:
- This has not been tested at all, except that it compiles and boots.
- This does not have a state adapter from master. The only state change
is the definition of +cach.
- Additionally, out-of-date ships may unexpectedly receive a %dead blob
from a foreign clay which would interfere with their ability to download
that desk. No code changes necessary, but sponsors should avoid
tombstoning files in %base for a while so their children can get the
update.
- A merge will only fail if the tombstoned file conflicts with another
change. Note that as written, merging from a past desk *can* bring a
tombstoned file to the head of a desk. Possibly this shouldn't be
allowed.
This also includes a couple refactors that were made possible by ford
fusion (since everything is synchronous now) but never got done. In
both cases we get to remove a monad, which simplifies the code
considerably.
- refactor +merge's error handling to use !!/mule instead of threading
through errors
- refactor all +read-* functions and related parts of +try-fill-sub to
eagerly convert lobes to cages.
We also add support reading %a/b/c/e/f/r/x from past and foreign desks,
when possible. Apologies that all of these are in one commit, it was
all a single chunk of work.
This is a draft until we have a way to tombstone. I suspect we'll want
to have a mechanism of keeping track of gc roots and trace to remove,
but this PR doesn't suggest any particular strategy.
Jael needs to be reconfigured to listen to the new aagent for azimuth
events, and the old app needs to be shut down. We do this in
/app/azimuth's +on-init.
Additionally, we make sure that jael doesn't crash when it (as expected)
loses its subscription to the old agent.
When you loaded an app with an error, then fixed the error, it would
create the main gall %mult subscription at a time in the past. Then,
clay would never fill the subscription since it couldn't get the old %a
entries for the apps.
This fixes the issue in two ways: first, don't subscribe in the past.
Second, if clay can't get the old versions, just fire the subscription
anyway.
Previously, if trying to bind to an endpoint that was already bound to,
eyre would reject it. This doesn't play very nicely in a softdist world
where uninstalled apps might not get a chance to clean up, and apps
might re-bind simply for being re-installed.
Here we change eyre to overwrite an existing binding if it conflicts
with the new one to be added.
As SSE are unidirectional, the client always realises that the
connection has failed faster than the server does. Hence, resuming a
subscription is useless, because channels can only be bound to one duct
at a time. Now, instead of failing a request for a channel
that is already bound to a duct, we replace the duct and continue
normally.
Start with |start %desk %app-name
Everywhere in the kernel that we deal with marks, we infer the app it's
connected to and use the marks from that desk.
Also some light renaming in gall, especially path->wire and
current-agent->yoke.
Subsequent tasks:
- Dojo needs a syntax to run generators and threads from other desks
- The home desk should be split into at least a minimal base desk and
big "userspace" desk. Dill's initialization logic should be updated
to handle
- |show-package, |install, and |uninstall should to be written
- Clay should have smarter handling of system versions instead of just
ignoring what's on each desk. It's not clear that this will work
correctly when sys updates right now.
Improves the multikeyfile format by taking a single ship and a list of
life+key pairs, instead of a list of full seeds.
Also decouples these changes from the dawn event, once again putting a
single seed into it. In the multikeyfile case, keys are injected as
%rekey events to jael near the end of the boot sequence.
Haskell-side changes may or may not be incomplete, boot presently fails
at some unknown point with what looks like a noun conversion error.
Allows booting with a keyfile containing multiple keys, as long as one
of them matches current PKI pubkeys for the specified ship.
All relevant keys are loaded into jael and will automatically be put to
use when they match PKI state.
Notably includes some changes to webterm's app.tsx that are required to
keep it functioning correctly. As of yet unclear why exactly this is
necessary, presumably hook shenanigans triggered by recent-ish changes.
Four changes:
- implement +validate-u to allow %u requests over the network
- make +validate-x use our local marks to make %x requests generally
work over the network
- in +start-request, if a foreign ship is making a request that we
shouldn't send over the network, ignore it. This closes a DOS vector.
- in +duce, if we're about to make a request to a foreign ship which
they won't be able to answer, crash the event.
Combined, these fix many of the common cases of weirdness around foreign
clay requests. Notably absent is a fix for reading `%a` across the
network, which I still maintain should happen against the foreign
hoon/zuse.
fixes#4834
see also #4307
Also switches everything to ropsten by default, including ivory pill.
Batches work on ropsten now.
Also adds +tx as a hacky development tool to create text for metamask to
sign and then turn that into a batch. A useful reference for bridge and
aggregator work.
By factoring their shared logic out into +build-dependency, which gets
passed the relevant details about how to track the file being built in
the dependency stack.
Hoon files may want to import nouns from all files in a given directory.
/~ lets you do so, importing as a (map @ta *) (but with typed values).
Note the description as "directories" here, instead of "path prefix".
The behavior, as implemented, will not include /path/hoon for /~ /path,
instead only including /path/more/hoon and more deeply nested files.
This seems to be, generally, the behavior you want, for example when
importing from /app/myapp/* for /app/myapp/hoon.
Actually using the resulting map requires some manual casting, which is
not ideal. Some code style improvement work remains to be done as well.
Implements tasks for creating and deleting new sessions, and allows
terminal handler agents to distinguish between sessions.
Includes bits of preparation in drum to more fully support multiple
distinct sessions, but doesn't get it all the way there just yet.
Gall would send %onto gifts to notify about app updates and update
failures. This would end up in dill, which printed some appropriate
text.
Here, we make gall responsible for doing this printing itself (by
having it explicitly ask dill to print some tape/tank), instead of
relying on the receiving end of some bespoke notification protocol.
Avoid allocating hundreds of thousands of cells when giving large
requests. This took the footprint of this function on initial landscape
load from 1 second to 100 ms.
The previous version allowed for redundant values (both [%bac ~] and
[%key ~ %bac ~] for example), had an odd constraint in @cF, and relied
unnecessarily on $<.
Also rewords some of the belt and blit descriptions.
Previously, we relied on foolish hacks, like [%met %bac], to send
"special" keystrokes with modifiers.
This updates the belt type to have %key, which represents a single
keystroke, with any combination of modifier keys.
Note that this has overlap with %txt to some extent. [%key ~ 'a'] should
be considered equivalent and preferred to [%txt 'a' ~], but updating
existing usage is left to a later commit.
This simplifies the behavior of individual blits, making their
implementation simpler and giving arvo more control.
This lets us write on top of existing content, instead of completely
replacing the affected row. Additionally, lets us draw starting at the
cursor position, instead of the leftmost column.
To retain the previous behavior, preface with [%hop 0] to move the
cursor to the start of the line, [%wyp ~] to clear the existing content,
and finally your %lin to render it.
Some of the remainder are still _presently_ unused, but point at
functionality we want to support again in the near future. The ones
removed here are either redundant or have no clear purpose.
As per the note a couple lines up, +fore depends on drum semantics being
active. We can only guarantee those being present for the default
session, not for any others. So, we print a warning when appropriate.
Before recent dill changes, this wouldn't always be visible, since it
would get drawn in place of (and subsequently get overwritten by) the
prompt. Now that it displays consistently again, it should look a bit
better than just a noun dump.
This is somewhat redundant with gall's own "reloading agent" printfs,
but you know what they say: printfs don't real!
This prepares us for actually making use of multiple session in a sane
way.
Notable implicit change is that we no longer crash on an "unrecognized
duct", instead always handling it as destined for the default session.
As follow-up to 3fdef1468, we now no longer store the prompt or cursor
in state. These were still used for view initialization, but it's more
appropriate to %hey the underlying console application in those cases.
The scry endpoints we remove, expecting clients that depended on them
to send a %hail instead.
No longer inserts newlines or redraws the prompt post-print, pushing
this responsibility down to drum where it belongs.
Additionally, separates the flow for dill's own output, from that of the
console application. This lets us keep the desired behavior for now, and
will ease reworking in the future.
Last-printed-line and cursor position are still kept around in dill
state, in order to respond to the relevant scry endpoints. These should
either be refactored to scry into the underlying console app, or be
removed entirely in favor of %hey.
Instead of confining you to just the bottom row. 0,0 is bottom left.
Doesn't behave exactly as expected for non-zero column coordinates yet,
but all in due time.
Likewise for belt. This necessitates renaming the %mor blit for newlines
to %nel, making this require a new runtime version. That's fine, more
breaking changes are to follow.
