This removes the logic from cleaning up stale subscriptions in %gall,
leaving +ap-rake as it was, and moves it to the +on-kroc arm in %ames.
Failed subscriptions from nacking a %watch plea that were
not properly corked (fixed in https://github.com/urbit/urbit/pull/6102)
are a subset of the more general "stale re-subscription" issue, so
we take care of all stale flows at the same time, by focusing on the
current subscription—leaving all others to be corked automatically—and
checking if it received a nack, to subsequently cork it.
This modifies the %rake task in %gall, to select what kind of
subscriptions we try to close:
=mode %o: kill old pre-nonce subscriptions
=mode %z: kill old pre-nonce subscriptions, including sub-nonce = 0
=mode %r: kills all stale resubscription flows
It also adds a dry-run option to both tasks (%kroc in ames, %rake in gall)
Address PR6136 comments to improve the interface to this scry.
Now it looks like .^((set ship) %cs /=landscape=/subs)
instead of .^((set ship) %cs %/subs/landscape)
During pill and install prop generation.
In autopill, we stop tracking a copy of the blob store, instead just
having the pill lib scry it out on-demand.
Implements a /cx/[our]//[now]/cult/[desk] endpoint, for getting a set of
pending requests for any given desk. We don't give the $cult for the
desk as-is, but instead slim the $roves back down into $raves, remove
clay protocol version metadata, and make sure to put our @p in place of
empty "for" fields.
This flow is not supported, and it was causing issues releasing
416. This change just drops the responses to avoid crashing, but at
some point we should either support this flow or reject the request in
the first place.
As of version %5, dill uses a new wire format for its userspace
subscriptions. Its existing subscriptions (read: the one subscription
into %hood for the default session) was never updated to use this new
style.
We observed a bug on one ship, where it had both old-style and new-style
subscriptions into hood, resulting in output being rendered twice. How
exactly this happened remains as of yet unclear.
Here, we forcefully clean up the old-style subscription, and
(re)establish the equivalent new-style subscription. This will prevent
issues like this from reoccurring.
This reverts commit 31bb93846c, reversing
changes made to 7940dd442b.
Reverting because we can't upgrade jetted code without ensuring the jets
change in lockstep.
See #6052. This is completely different from the +* used at the top
of doors, and has almost entirely been replaced by |$. The exception is
the use of the `%made` spec, not present in `|$`. I do not see an
obvious way to change `|$` to use `%made` since this `+*` parser uses
the name of the arm in the `%made` structure, unless we change the
AST of |$.
|* foo bar is sugar for =+ foo |@ ++ $ bar --, and newbies find
the old style confusing. this switches out the |@ pattern for the |*
one, at least in layer <=4. the only ones remaining are +toad, +rune,
and +runo, which are already tweaked in #5873 so we omit them here.
Adds .snub to ames-state, a global blocklist for ships. If a packet is
received from a ship that is in the .snub set, it is immediately
dropped. Adds %snub to ames' $task, to allow manipulating this list
all this did was set .nut. while it could be used with doccords, it is
currently unused, and none of the other values in the sample of _ax are
set this way (bug, def, cox, hay, dom). i experimented a little bit with
trying to make use of this but it made things overall more unreadable,
and it wouldn't make sense to do it without doing the same for other
values of the sample. im guessing this is just an old style.
Previously, fake breaches triggered by a %ruin task would only get sent to
subscribers watching for the affected ship specifically. Now, we send them to
both those subscribers, and the ones watching for pubkey changes on all ships.
when +apse sees a link, it presume that the following a batch comment,
and stops parsing so that it can be picked up by apex:docs next
this required a change to +leap, which has been rewritten to pretty much
look like +gap but stop parsing when encountering doccords.
previously we just threw them out and wasn't sure whether it was the
right answer. this violates the principle of least surprise - even
though it hard to see the value of attaching multiple empty $cuff notes
to an arm, we shouldn't stop the programmer from doing it without any
indication or explanation as to why. its the behaviour you'd expect
given how doccords is structured.
it is desirable for both apex:docs and apse:docs to parse into an
intermediate representation that never ends up in an AST so that it is
clear that these parsed representations may be altered in the future
without worrying about old types nesting with new types. this was
already the case for $whit, but apse:docs parsed directly as a $help,
which is used in ASTs. so apse:docs now parses as a $whiz, which is
simply a cord. in the future, if postfix comments are used for something
like invariants, or allow $links, we may want to change this.
this also changes $whit to remove .use, which was unused. similarly,
+glom is removed since its not used anywhere.
this might actually be undesirable, don't want to leave this as a trap
for somebody in the future thinking we knew it was definitely the right
answer. having batch comments follow the chapter declaration does make a
certain amount of sense, stylistically
future-proofing %gist specs by putting a %help tag on the $help. this
looks pointless at first glance, but it allows the opportunity for %gist
specs to have a $% in the future in a way such that the old type nests
with the new one, eliding the need for a typo->type migration
Most of the memory stays in gall anyway, and this means you need to
recompile everything the next time anything changes, which could be
counterproductive. It's important that %trim not make things worse.
The functionality is moved to the debug %stir task.
These were originally added because they reduced memory usage, primarily
by clearing the memoization cache. Now that the memoization cache is
no longer used, we use less memory without them. On ~wicdev-wisryt with
~30 apps, updating Clay now takes ~320MB.
(addendum to +team change)
address feedback from ~rovnys-ricfer, ~master-morzod,
~ritpub-sipsyl, ~tacryt-socryp, ~wicdev-wisryt, and others.
the original functionality of +team has been split out
between +team:title and +moon:title.
also:
fixes "middle core" and "surface core" comments in title
this is to handle potential future cases where doccords might be kinds
of notes other than %help notes. example: #6085 to document invariants
in clay.
$whit (used for apex:docs/batch comments) also ought to be changed but
im still thinking about what that should look like.
partial revert of 3d3ea61d53, which introduced core names by completing
an unimplemented feature that was already present in hoon.hoon. we've
decided to remove this for the initial launch since it violates the
principle of least surprise for the name of a core to end up in its
$garb and yet only be used for doccords, as opposed to something like a
wing resolution. it was also confusing that this only worked for |% and
|@.
this breaks two of the tests for the dprint library, which have been
commented out. these tests ought to be restored once dprint is rewritten
in order to implement a different way to refer to cores not built by arms
this constitutes a pretty major rework of how whitespace is handled in
hoon in order to change the doccords syntax from :> and :< to ::.
in summary: throughout the hoon parser (+vast) many instances of +gap
have been replaced by +jump, which first tries to remove whitespace (+leap)
until it arrives at something that can be parsed as a prefix
doccord (+apex:docs:vast). if it does not encounter a doccord, it
instead uses +gap as normal.
if you follow along with the parser, you will notice that every time
jump is called, it then tries to call +apex:docs via +scye or +seam. if
apex:docs succeeds, it will end up consuming a newline at the end,
hiding the fact that there was valid whitespace from the parser. thus
+apex:docs then inserts a newline after successfully parsing a prefix
doccord, which will then be consumed by a subsequent invocation of +gap,
ensuring that there was proper whitespace if the doccord would have been
consumed by +gap instead of +leap.
there are a few other changes:
+hint in the compiler throws out doccords attached to %noun types. this
was already the behavior before doccords, and the change was made before
i understood what i was doing.
similarly for commenting out the %note case in +open:ap - this was an
earlier mistake
postfix comments for chapters are now enabled.
+expx was unused and removed in order to be rid of the
convention-defying +exp1. other unused +ex(p/q)* were commented out.
arms that handle batch comments (+glow and +whap) were refactored
+toad, which was used to change between tall and wide form, tries to
+jump before +gap. since +jump is ;~(pose leap:docs gap), i would have
thought that just using +jump there would be fine, but it fails for some
reason.
some parsers built with |@ were rewritten to use |*
|$ was made so that any doccords put on the spec are converted into hoon
doccords on the $ arm. it wouldn't compile otherwise. there's probably a
more principled way to do this but it works fine for now.
- fix `fragment-num` and `num-fragments` having duplicate faces
- fix faces being wrapped around wrong things in various places
- fix `bone` not being printed in "hear last in-progess" message
- make pretty tape interpolation style more uniform
in the past, +team meant "our / our moon", but
it has been primarly used to represent "our"
moons as having the full permissions of their
parents doesn't make a lot of sense anymore
this looks like the more elegant solution
instead of changing each instance of +team
I've combed through the uses of +team throughout
urbit/urbit and I'm quite sure that each instance
is better off as just "our"
The +on-cork handler asserts that the peer is known to us. This is the
incorrect behaviour, because it will crash when corking a flow to a peer
that is still an %alien. This can happen, for instance, when making a
gall subscription for the first time and then corking it before the
alien naturalises.
if a cert is configured and a secure port is live it will set the
redirect flag in http-config.state.
When it gets a ++request it will return a 301 redirect to
https://[host]/[path] if:
1. not already secure
2. redirect flag set
3. secure port live
4. is not requesting /.well-known/acme-challenge/...
5. the host is in domains.state
It will not happen if forwarded-secured, localhost, local loopback, ip
addresses or domains not in domains.state.
in ++load it checks the secure port is live and a cert is set and
enables it if so (for people who already use in-urbit letencrypt)
%rule %cert tasks also toggle it (only turning it on if secure port
live)
%live tasks also toggle it (only turning it on if cert set)
Have tested with a couple of ships and seems to work fine.
This is useful in combination with pyry's auto arvo.network dns config
system - can finally get rid of reverse proxies entirely.
Eyre always gets passed request headers in lowercase, so we should search for
the lowercased version of the header.
Arguably `+get-header` should lowercase keys before comparing them, but that's
a more serious behavioral change.
This allows you to pass a thread directly into khan, instead of passing
a filename. This has several implications:
- The friction for using threads from an app is significantly lower.
Consider:
=/ shed
=/ m (strand ,vase)
;< ~ bind:m (poke:strandio [our %hood] %helm-hi !>('hi'))
;< ~ bind:m (poke:strandio [our %hood] %helm-hi !>('there'))
(pure:m !>('product'))
[%pass /wire %arvo %k %lard %base shed]
- These threads close over their subject, so you don't need to parse
arguments out from a vase -- you can just refer to them. The produced
value must still be a vase.
++ hi-ship
|= [=ship msg1=@t msg2=@t]
=/ shed
=/ m (strand ,vase)
;< ~ bind:m (poke:strandio [ship %hood] %helm-hi !>(msg1))
;< ~ bind:m (poke:strandio [ship %hood] %helm-hi !>(msg2))
(pure:m !>('product'))
[%pass /wire %arvo %k %lard %base shed]
- Inline threads can be added to the dojo, though this PR does not add
any sugar for this.
=strandio -build-file %/lib/strandio/hoon
=sh |= message=@t
=/ m (strand:rand ,vase)
;< ~ bind:m (poke:strandio [our %hood] %helm-hi !>('hi'))
;< ~ bind:m (poke:strandio [our %hood] %helm-hi !>(message))
(pure:m !>('product'))
|pass [%k %lard %base (sh 'the message')]
Implementation notes:
- Review the commits separately: the first is small and implements the
real feature. The second moves the strand types into lull so khan can
refer to them.
- In lull, I wanted to put +rand inside +khan, but this fails to that
issue that puts the compiler in a loop. +rand depends on +gall, which
depends on +sign-arvo, which depends on +khan. If +rand is in +khan,
this spins the compiler. The usual solution is to either move
everything into the same battery (very ugly here) or break the
recursion (which we do here).
Whenever a session gets created or removed, send the set of valid auth
tokens to the runtime, so that it may use them in determining whether
incoming requests are authenticated or not.
%kick is supposed to start back from the snapshot and move forward.
Without this, we would only fetch logs that we hadn't already fetched.
Thus, if you were up-to-date when you kicked, you would miss anything
that happened between the time the snapshot was taken and the present,
though you would see things after the present.
Also reverted lull change to make this a safer upgrade.
Previously, when the larva got to processing enqueued events, it was
doing so without loading state into the adult beforehand, resulting in
incorrect processing of events.
Here, we make the larva call +molt more eagerly, ensuring that the adult
always has its state available when we use it.
Yes, there is a global timer for closing flows, but all that does is
enqueue a cork message. +on-stir needs to set _pump_ timers for all
flows that might still have messages to send, which includes closing
flows.
When ames notifies us that our subscription has been kicked, we enqueue
a cork to clean up the flow. Unlike the %leave case, however, we were
not registering the cork in the queue of outstanding comms. We would
eventually get an ack, but not know what for, and erroneously inject
%poke-acks and %watch-acks.
Here we simply add a %cork entry to the queue before sending it.
This is sufficient to bring the normal (non-prerelease-bugged) cases
into the new world.
For the prerelease ships that ran a buggier version of the new gall
subscription logic, we note that the conditional may trigger for the
nonce=1 case where it had already triggered for their
(shouldn't-be-possible) nonce=0 case. This results in a %leave on a wire
that wasn't in use. This no-ops on the publisher side though, and the
flow gets corked right away, so this is considered harmless.
In response to clog notification from remote ames, we were sending a
%cork to clean up the flow. However, the wire we were using had the /sys
prefix already stripped off. Here, we put it back in.
Start by killing subscription nonce 0, then work our way up instead of
down. We enhance the printf with a "total nonces" indicator so we can
still easily see the progress being made.
Previous +ap-doff kicked the agent repeatedly. We needed to kick
it only once. Now publisher agents clear their incoming subscription
state without the subscriber making lots of new subscriptions because
of repeated kicking.
+on-plea gets called in two very different ways:
1) handling request from local vane to send %plea to peer
2) handling %cork request from another ship, which our local ames has %pass'ed
to ourselves
In the second case, we shouldn't print misleadingly, or bind a duct in the ossuary.
+ap-nuke was not including the nonce, but should.
+ap-handle-peers was potentially including a zero nonce.
(The latter shouldn't have been possible, but there's a bug in +load
where sub-nonce.yoke gets initialized as 0 instead of 1.)
Gall tells ames to %cork flows for subscriptions it has closed.
Receiving a kick also closes a subscription, but gall wasn't issuing a
%cork in that case. We correct that here.
Inlines +mo-handle-ames-response's logic at its only callsite.
seems that this structure has been unused since
e75ab631a4 and confuses
newbies trying to figure out exactly what the commit
structure is (which is how I came across this)
Without this, a ship would send a cork on a max of one flow per
recork timer, which could take years to clear for some ships.
This starts a hot loop of trying the next cork once one gets
positively acked.
The previous recork timer queued up %cork messages without sending them.
It also relied on making sure pump timers didn't get set for recork bones.
This was fragile.
The new design enqueues up to one new %cork message per ship during each
recork timer, based on the state of the flow. If the flow is closing but
there are no outstanding messages in it, then it needs to be recorked.
Flows will be recorked in ascending numerical order by bone.
The condition got butchered during refactor: instead of avoiding the creation
of pump timers during recork wake, it was setting them _exclusively_ during
recork wake.
this refactors the parser for %brcn and %brpt to separate the optional
argument(s) from the required argument(s).
also adds +blab, which allows for a minor refactor of a couple other
arms as well as being used for %brcn and %brpt
