This fixes a space leak where the entire ford/clay core would be
included in the ford cache. Heavily reduces memory usage by clay,
reducing total usage from 100-200MB to around 10MB.
Instead of reporting a single memory size for built files, marks and
conversions, we now report memory size per path, mark name and mark
pair, respectively.
If a batch gets bigger than a max size defined by the ethereum node
the raw transaction is sent to, the /ted/roller/send thread will crash and
the batch will be blocked, stopping any subsequent batches to be sent.
This detects when the current batch reaches a certain threshold and only
includes transactions up to that point, moving the ones that are not sent
back to the pending queue, adjusting their history and finding status.
This gives us much higher cycle times when retrying tests over and over,
since we can just fast-forward past the boot sequence, which never
changes unless we change the pill.
scries %azimuth for logs.state, filters the L2 events, and then gets
their timestamps. ought to use ted/eth/get-timestamps but doesn't due to
issues with spawning child threads, but there's probably a way to do it
without child threads
This changes the parser for +tall so that it looks before and after a
hoon for doccords, and then extracts a label for %brcn if it exists.
+wrap will be used to annotating most hoons, but this commit only covers
%brcn
Too often when dealing with big types the compiler traces and other such
outputs become hard to read. Wrapping a type as $+(shorthand big-type)
will now print #shorthand in place of the type.
Also I took the liberty of making `build-signing-participants` use a `%lyfe` scry and remove unknowns from the set rather than crashing. This is a choice - happy to change it back the other way.
Too often when dealing with big types the compiler traces and other such
outputs become hard to read. Wrapping a type as $+(shorthand big-type)
will now print #shorthand in place of the type.
Disables automatic dismissal of push notifications in an attempt to
tighten the debugging space. This should prevent unintentional 404s. To
prevent a space leak we clear notifications once a week on a timer.
verified using vim regex search (and corresponding regexes for the 3, 2,
and 1 letter motes):
/c3__\(.\)\(.\)\(.\)\(.\) *c3_s4\(('\1','\2','\3','\4')\)\@!
instead make it a rolling 128-bit integer. 128 bits is the same size as
the +sham space, so is one natural choice for "big enough to never have
to think about this." 64, 32, 16, even 8 bits would probably be fine.
Threads should eventually take and produce $cage instead of $vase. Since
%khan is likely to be used by third parties, we write to the eventual
intended API. We ignore the mark on the input $cage (it is safe to
always specify %noun), and we always use %noun as the output mark.
%fyrd now makes more sense. It was previously discarding the type of the
output %arow and re-encoding the raw noun as a vase of the output mark;
it is now performing mark conversion from the mark of the output $cage
to the originally requested output mark.
* master: (33 commits)
groups: updating glob and version
interface: use single sig in NotificationText
interface: fix subscription reconnect issues
landscape: fixing bad glob
landscape: updating glob and version
interface: adds `theme-color` meta tag, removes outdated safari web app meta tag
zuse: add missing assertions
landscape: cache marks again
zuse: comment clarifying sk bounds check
bounds-check against sk=1
zuse: style cleanup, use +rep/+end
pill: solid, brass
interface: refine joining error cases
group-view: fix errored rollback
helm: cleanup +poke-rekey to match #5522
helm: fix |rekey to work with multikey files
test: schnorr bounds checking
zuse: boundary assertions for schnorr
zuse: schnorr test cases
zuse: schnorr address
...
nara: swedish for near, provides a trace from the start of the nearest virtualization
to the statement wrapped by the nara hint.
hela: swedish for whole, provides a trace from the start of the event
to the statement wrapped by the hela hint.
The eventual goal is to present these traces in a way that is most salient & useful to developers
and in a way that is aesthetically pleasing as the bout hint is useful and nice looking.
For this pull request the goal is to provide the actual functionality in code that we can feel
safe and comfortable adding to the master branch. To this end, I've added a trivial set of tests for
nara and hela in pkg/arvo/tests/run/hints.hoon - they only prove that invoking these hints will not
crash the runtime, but I'm open to ideas for how I can test these hints further.
The `ProfileOverlay` component expects a `ship` arg as a patp without the sig. This deSigs the ship value used in `NotificationText`.
This closesurbit/landscape#1376
the full type output by vars is now:
(list [cord (list [cord (list [cord @])])])
it's a mouthful, but a consistent mouthful. the first layer of the list
is the IO driver name, nam_m. the second layer is the instance name,
either %all or some driver-relevant identifier (e.g. http instance.) the
third layer is the list of labels and values.
This actually raises difficult questions about the schema for the /vars
peel. One level of nesting makes sense to aggregate per IO driver, but
multiple levels is confusing.
The current output is extremely unprincipled: you just have to know that
there are multiple http servers and handle the output accordingly.
Options would be:
1. collapse the ambiguity to the top level, i.e. 'http-0i8080',
'http-global', etc.
2. collapse the ambiguity to the inner level, i.e. '0i8080-connections'.
3. create a proper recursive data type that e.g. uses an $each.
3b. send some kind of schema.
5. recognize that we have entered a terrifying hall of mirrors and back
out the entire approach of nested metrics in favor of just a bigass
flat list of labels with values like borgmon does.
tid was accidentally getting set to the name of the output mark. As we
don't currently support cancelling threads, there is no reason to
maintain the originally-intended correspondence between tid and conn
request-id.
Take the opportunity to clean up indentation somewhat.
Also strips out `$` from khan top-level comment.
There are arguments for keeping $crag in lull, and on the other side for
moving $cast to arvo. This seemed like the most reasonable approach.
%fyrd is now implemented in terms of %fard, and likewise %avow in terms
of %arow. State is tracked via wire rather than in a global map.
Unit tests adjusted to match.
support a `%depth-first` scry constructed as follows `.^(* %gx /=graph-store=/graph/<ship of resource>/<name of resource>/depth-first/<count>/<start>/graph-update-3)`
These take and produce vases, and assign random tids (rather than
deducing them from the input duct.)
Since %fard does not require mark conversion, we make the mark/beak on
$thread-state optional (and use this to decide whether to send %avow or
%arow.) Provide a state adapter since it's possible that people have
been experimenting with this vane.
This makes the negative case of %avow/%arow kind of clunky, since there
is no content difference, but the following does not seem possible
within the Hoon type system:
=/ gif
?~ p.tad
%arow %avow
[hen %give gif %| p.cag tang]~
- use desk parameter instead of %base everywhere
- formatting clean up
- make |story-remove take a case instead of an aeon
- make desk param optional for story-set and story-log
+sign:schnorr crashes on `=(0 sk)`, so the bounds checking code is not
exercised for sk=0. It also crashes on `(gte sk n.domain.c)`, which is
redundant with the size check on sk, so we remove that.
'~.' was a pun with the @ta encoding, which could cause people to get
confused about how the mapping actually worked.
I had previusly wanted the escape sequence to be a valid @ta, since I
had wanted to check that path components were (sane %ta) prior to doing
$knot conversion. However @joemfb mentioned that it was desirable to
have it not be a valid @ta so that if someone messed up the encoding in
the future, the paths would be detectably wrong.
This necessitates calling _unix_knot_to_string on unsanitized input,
which means we can no longer assume that it won't contain non-(sane %ta)
characters, which means we can no longer assert that '\\' is not in the
string on Linux.
Also added a seemingly forgotten null byte to _unix_knot_to_string.
- Don't test for '\\' in u3_unix_safe. Doing otherwise was crashing vere
when unmounting a mountpoint that had come to contain a file with '\\'
in its path. This might mean you can do bad things on Windows if other
checks fail.
- Ignore any files whose names do not pass `(sane %ta)` when scanning
directories. (This reimplements `(sane %ta)` in C. Perhaps it should
instead call `(sane %ta)`.)
- Use '~.' rather than '~' for the escape. We ignore files that end in
'~', probably for vim backup-file reasons.
- Add a _unix_string_to_knot missed in the prior conversion.
unix cannot represent the file with empty name, and it has special
mappings for '.' and '..'. as these three are all valid arvo `+knot`s,
we need to escape them if we come across them.
the method we use to escape is: if we encounter any of those three
`+knot`s, or any `+knot` starting with '~', we prepend its filename with
a '~'. and when going from filename to `+knot`, we do the reverse; i.e.
we ignore a '~' if it is the first character of a filename.
the current implementation just crashes if it encounters a `+knot`
containing '/' or '\\', neither of which are valid under the current
implementation of `@ta` (which only accepts numbers, lowercase, '-',
'~', '.', and '_'.)
it also crashes if it encounters a file containing '\\'. something else
should happen here; most likely vere should just ignore the file.
;;(vase ...) does a nest-check of the type of the kernel. This is
undesirable, so we instead run everything through +slum and cast the
result to +tang.
- only store metadata in the persistent map. just enough to support
(eventual) thread cancellation and output mark lookup.
- try to delete thread state at other failure points not covered by
%kick.
- reflect back the passed output mark rather than form.dais. not sure
about this one yet.
Fallback to the default happens in dawn.c, which correctly points to
roller.urbit.org, an endpoint that matches its request/response logic.
Continuing to use an Ethereum endpoint instead of an L2 one will just result
in 400s, since they don't speak the same language.
This seems to be what nix settled on.
(As of now, I can build urbit on M1 Mac with stock nix. nix-build -A
urbit hangs for some reason, but nix-shell ./configure && make works.)
The previous value—used for testing—didn't consider
block reorgs, which meant that if we zoom to the latest
block that has no transactions, but that gets later replaced
by a 1-block reorg that does have a transaction, we'll miss it,
making our Azimuth state incomplete.
To fix it, we rewind the Azimuth state to the contents of the snapshot,
and then start retrieving logs from the latest one we have.
Get some space between the IO driver and vane, since the driver is
mostly self-contained now and only depends on %khan for thread running.
"Conn" is a nautical term; it is the status of being in control of a
ship's movements, or the act of controlling a ship.
No mark files exist for any of the drum marks, so trying to poke remote drums
would fail anyway, but relying on the mark system in that way seems a bit
fragile, so we add an explicit permission check.
No mark files exist for any of the helm marks (except `%helm-hi`), so trying to
poke remote helms would fail anyway, but relying on the mark system in that way
seems a bit fragile, so we add an explicit permission check.
Before this, u3v_lily would erroneously accept atoms bigger than 64 bits
that, when truncated to 64 bits, were 31-bit numbers.
Decided to drop _cv_mole altogether. Another option would be to write a
u3r_chub_fit, write _cv_mole in terms of that, and check width in
u3v_lily as it currently does.
I tried to add a test case, but it seems that tests don't have access to
an ivory pill for +scot / +slaw. This would be the test case, more or
less:
{
c3_l lit_l;
c3_w big_w[] = {0, 0, 1};
u3_atom big = u3i_words(3, big_w);
u3_noun cod = u3dc("scot", c3__ux, big);
if ( c3y == u3v_lily(c3__ux, cod, &lit_l) ) {
printf("fail\n");
}
}
(The refcounting was also messed up, possibly from my refactor to use
+slaw instead of +slay, but this seems to have been unrelated.)
- Caught a couple potential leaks.
- Reworded error messages to be more @tas-compliant.
- Used unique error codes in bal_f calls.
- Moved u3z calls onto same line as returns.
Not sure we actually want to guarantee exactly-once response semantics,
so you can't rely on request ordering to correlate requests even if you
never send simultaneous requests.
Also do a worked example of a %urth scry, which hopefully raises the
question of whether %urth should try to support views.
It was if 0'd out in noun/vortex.c. That seems like a more reasonable
place to put it then a whole other self-contained file.
The commented version used slay instead of slaw and had an extra paren;
I opted to replace it with the impelementation from reck.c from http.c.
Inlines the server initialization in _khan_io_talk. Adds a van_o loobean
that differentiates between having the vane and missing the vane; van_o
is c3n if the %born event failed. In that case, we don't bother
forwarding %fyrd requests at all, but %peek requests still go through.
It was a mistake to call it except from _khan_close_chan, so just inline
it at the one call site. Also lets us stop doing our own pointer
juggling in _khan_io_exit.
While we're at it, flip argument order in _khan_close_chan;
superordinate first seems more natural.
- Have _khan_read_wire take a tag rather than produce it; if the tag is
not the one passed, it assumes it knows nothing of the rest of the
wire structure and simply returns c3n.
- rename _khan_search_chan to _khan_find_chan, the latter being more
flwy.
- Reformat %peek responses to start with request-id.
- Send %fyrd errors back over the channel that started the %fyrd.
- Factor out _khan_read_wire, reused by both _khan_poke_bail and
_khan_io_kick.
- Move _khan_search_chan up above _khan_poke_bail to use it there.
liv_o was always c3n without the vane. The pier was waiting for all
drivers to report liv_o == c3y. Tragedy ensued.
Maybe now we want to open the socket for peeks and moves even if the
vane is not present, so this may need to be rethought even further.
This is just the simple resolution for the culprit unearthed by
git-bisect.
Keeping a request list allows us to detect the case where a scry result
has returned on a connection that has already been closed, so we can
noop in that case.
Flipping the order of tag and request-id seems more natural for the
%fyrd case; otherwise there's no clear thing to send to the vane.
(Sending the whole jar sends request-id twice; once in the wire, once in
the body. Sending only the command itself loses context and introduces
potential namespace overlaps.)
As of now, scries are working.
And the format the IO driver takes is:
$: request-id=@ud
$% [%fyrd fyrd-args=*]
[%peek peek-args=*]
[%move move-args=*]
==
Boot was broken, fixing the hark-note file mark and re-adding the
hark-store library fixed it.
This lets us push a new pill, which is necessary for the fix in #5434 to
actually work.
Fixes#5501Fixes#5502
* master:
[nix] track nixos-21.11 branch whenever niv updates, bump to include qemu-in-virtualization fix
webterm: v1.0.0
herm: permission checks
herm: avoid trailing empty path segments
pmnsh: update secp256k1 configure flags
secp256k1: use nixpkgs provided secp256k1 and add to sources-pmnsh
webterm: remove border, let term live in page
theme: cleaning up a few mismatches
webterm: handle old-style blits and belts
build: correct lmdb static builds
build: explicitly override h2o build platforms to support darwin
ci: upgrade cachix/install-nix-action from v13 -> v16
build: remove haskell related nix code and haskell.nix dependency
webterm: update imports
@urbit/api: move term types
webterm: fix broken imports
webterm: update package name for lerna
webterm: commit missing api files
webterm: backport
* master:
[nix] track nixos-21.11 branch whenever niv updates, bump to include qemu-in-virtualization fix
webterm: v1.0.0
herm: permission checks
herm: avoid trailing empty path segments
pmnsh: update secp256k1 configure flags
secp256k1: use nixpkgs provided secp256k1 and add to sources-pmnsh
webterm: remove border, let term live in page
theme: cleaning up a few mismatches
webterm: handle old-style blits and belts
build: correct lmdb static builds
build: explicitly override h2o build platforms to support darwin
ci: upgrade cachix/install-nix-action from v13 -> v16
build: remove haskell related nix code and haskell.nix dependency
webterm: update imports
@urbit/api: move term types
webterm: fix broken imports
webterm: update package name for lerna
webterm: commit missing api files
webterm: backport
