We were decoding and then re-encoding the query string in URLs.
This round-trip caused us to lose information about plus ('+')
ASCII characters encoded as "%2B".
A change was made prior to percent encode plus signs in order to fix an
issue with the Google cookie consent page.
Unforunately, this was treating a symptom of a problem and not the root
cause and is incorrect behavior.
When we want to use the find_first_index that base Vector provides, we
need to provide an element of the real contained type. That's impossible
for OwnPtr, however, and even with RefPtr there might be instances where
we have a raw reference to the object we want to find, but no smart
pointer. Therefore, overloading this function (with an identical body,
the magic is done by the find_index templatization) with `T const&` as a
parameter allows there use cases.
On oss-fuzz, the LibJS REPL is provided a file encoded with Windows-1252
with the following contents:
/ô¡°½/
The REPL assumes the input file is UTF-8. So in Windows-1252, the above
is represented as [0x2f 0xf4 0xa1 0xb0 0xbd 0x2f]. The inner 4 bytes are
actually a valid UTF-8 encoding if we only look at the most significant
bits to parse leading/continuation bytes. However, it decodes to the
code point U+121c3d, which is not a valid code point.
This commit adds additional validation to ensure the decoded code point
itself is also valid.
These functions are _very_ misleading, as `first()` and `last()` return
references, but `{first,last}_matching()` return copies of the values.
This commit makes it so that they now return Optional<T&>, eliminating
the copy and the confusion.
This implements Optional<T&> as a T*, whose presence has been missing
since the early days of Optional.
As a lot of find_foo() APIs return an Optional<T> which imposes a
pointless copy on the underlying value, and can sometimes be very
misleading, with this change, those APIs can return Optional<T&>.
This method exploits the fact that the values themselves hold the tree
pointers, and as a result this let's us skip the O(logn) traversal down
to the matching Node for a Key-Value pair.
Adds a new optional parameter 'reserved_chars' to
AK::URL::percent_encode. This new optional parameter allows the caller
to specify custom characters to be percent encoded. This is then used
to percent encode plus signs by HttpRequest::to_raw_request.
As seen on TV, HashTable can get "thrashed", i.e. it has a bunch of
deleted buckets that count towards the load factor. This means that hash
tables which are large enough for their contents need to be resized.
This was fixed in 9d8da16 with a workaround that shrinks the HashTable
back down in these cases, as after the resize and re-hash the load
factor is very low again. However, that's not a good solution. If you
insert and remove repeatedly around a size boundary, you might get
frequent resizes, which involve frequent re-allocations.
The new solution is an in-place rehashing algorithm that I came up with.
(Do complain to me, I'm at fault.) Basically, it iterates the buckets
and re-hashes the used buckets while marking the deleted slots empty.
The issue arises with collisions in the re-hash. For this reason, there
are two kinds of used buckets during the re-hashing: the normal "used"
buckets, which are old and are treated as free space, and the
"re-hashed" buckets, which are new and treated as used space, i.e. they
trigger probing. Therefore, the procedure for relocating a bucket's
contents is as follows:
- Locate the "real" bucket of the contents with the hash. That bucket is
the starting point for the target bucket, and the current (old) bucket
is the bucket we want to move.
- While we still need to move the bucket:
- If we're the target, something strange happened last iteration or we
just re-hashed to the same location. We're done.
- If the target is empty or deleted, just move the bucket. We're done.
- If the target is a re-hashed full bucket, we probe by double-hashing
our hash as usual. Henceforth, we move our target for the next
iteration.
- If the target is an old full bucket, we swap the target and to-move
buckets. Therefore, the bucket to move is a the correct location and the
former target, which still needs to find a new place, is now in the
bucket to move. So we can just continue with the loop; the target is
re-obtained from the bucket to move. This happens for each and every
bucket, though some buckets are "coincidentally" moved before their
point of iteration is reached. Either way, this guarantees full in-place
movement (even without stack storage) and therefore space complexity of
O(1). Time complexity is amortized O(2n) asssuming a good hashing
function.
This leads to a performance improvement of ~30% on the benchmark
introduced with the last commit.
Co-authored-by: Hendiadyoin1 <leon.a@serenityos.org>
The hash table buckets had three different state booleans that are in
fact exclusive. In preparation for further states, this commit
consolidates them into one enum. This has the added benefit on not
relying on the compiler's boolean packing anymore; we definitely now
only need one byte for the bucket state.
Currently this can parse XML and resolve external resources/references,
and read a DTD (but not apply or verify its rules).
That's good enough for _most_ XHTML documents as the HTML 5 spec
enforces its own rules about document well-formedness, and does not make
use of XML DTDs (aside from a list of predefined entities).
An accompanying `xml` utility is provided that can read and dump XML
documents, and can also run the XML conformance test suite.
This is an enum-like type that works with arbitrary sized storage > u64,
which is the limit for a regular enum class - which limits it to 64
members when needing bit field behavior.
Co-authored-by: Ali Mohammad Pur <mpfard@serenityos.org>
Previously, case-insensitively searching the haystack "Go Go Back" for
the needle "Go Back" would return false:
1. Match the first three characters. "Go ".
2. Notice that 'G' and 'B' don't match.
3. Skip ahead 3 characters, plus 1 for the outer for-loop.
4. Now, the haystack is effectively "o Back", so the match fails.
Reducing the skip by 1 fixes this issue. I'm not 100% convinced this
fixes all cases, but I haven't been able to find any cases where it
doesn't work now. :^)
Day and month name constants are defined in numerous places. This
pulls them together into a single place and eliminates the
duplication. It also ensures they are `constexpr`.
Even though the StringView(char*, size_t) constructor only runs its
overflow check when evaluated in a runtime context, the code generated
here could prevent the compiler from optimizing invocations from the
StringView user-defined literal (verified on Compiler Explorer).
This changes the user-defined literal declaration to be consteval to
ensure it is evaluated at compile time.
C++20 provides the `requires` clause which simplifies the ability to
limit overload resolution. Prefer it over `EnableIf`
With all uses of `EnableIf` being removed, also remove the
implementation so future devs are not tempted.
Since the allocated memory is going to be zeroed immediately anyway,
let's avoid redundantly scrubbing it with MALLOC_SCRUB_BYTE just before
that.
The latest versions of gcc and Clang can automatically do this malloc +
memset -> calloc optimization, but I've seen a couple of places where it
failed to be done.
This commit also adds a naive kcalloc function to the kernel that
doesn't (yet) eliminate the redundancy like the userland does.
Previously, if you forgot to set a key on a SourceGenerator, you would
get this less-than-helpful error message:
> Generate_CSS_MediaFeatureID_cpp:
/home/sam/serenity/Meta/Lagom/../../AK/Optional.h:174: T
AK::Optional<T>::release_value() [with T = AK::String]: Assertion
`m_has_value' failed.
Now, it instead looks like this:
> No key named `name:titlecase` set on SourceGenerator
Generate_CSS_MediaFeatureID_cpp:
/home/sam/serenity/Meta/Lagom/../../AK/SourceGenerator.h:44:
AK::String AK::SourceGenerator::get(AK::StringView) const: Assertion
`false' failed.
This is the IPv6 counter part to the IPv4Address class and implements
parsing strings into a in6_addr and formatting one as a string. It
supports the address compression scheme as well as IPv4 mapped
addresses.
If the utilization of a HashTable (size vs capacity) goes below 20%,
we'll now shrink the table down to capacity = (size * 2).
This fixes an issue where tables would grow infinitely when inserting
and removing keys repeatedly. Basically, we would accumulate deleted
buckets with nothing reclaiming them, and eventually deciding that we
needed to grow the table (because we grow if used+deleted > limit!)
I found this because HashTable iteration was taking a suspicious amount
of time in Core::EventLoop::get_next_timer_expiration(). Turns out the
timer table kept growing in capacity over time. That made iteration
slower and slower since HashTable iterators visit every bucket.
Just walk the table from start to finish, deleting buckets as we go.
This removes the need for remove() to return an iterator, which is
preventing me from implementing hash table auto-shrinking.
This will be caught by new test cases: when the initial chunk is empty,
a dereference before calling operator++ on the iterator will crash as
the initial chunk's size is never checked.
Previously, although we were taking a moved chunk, we still copied it
into our chunk list. This makes DisjointChunk compatible with containers
that don't have a copy constructor but a move constructor.
This adds a NOLINT directive to the definition of the TODO() macro.
clang-tidy wants the assert replaced with a static_assert, since the
macro simply resolves to assert(false). This is obviously nonsensical,
since we want the code to still compile even with TODO().
The same fix has already been implemented for VERIFY_NOT_REACHED().
While the code did already VERIFY that the ErrorOr holds a value, this
was done by Variant, so the error message was just that `has<T>()` is
false. This is less helpful than I would like, especially if backtraces
are not working and this is all you have to go on. Adding this extra
VERIFY means the assertion message (`!is_error()`) is easier to
understand.
Parse JSON floating point literals properly,
No longer throwing a SyntaxError when the decimal portion
of the number exceeds the capacity of u32.
Added tests to AK/TestJSON and LibJS/builtins/JSON/JSON.parse
We shouldn't let copy/move ctors or assignments be instantiated if the
assignee type does not have a copy/move constructor (even if they're not
used anywhere).
This matches the likes of the adopt_{own, ref}_if_nonnull family and
also frees up the name to allow us to eventually add OOM-fallible
versions of these functions.
Before this was incorrectly assuming that if the current node `n` was at
least the key and the left child of `n` was below the key that `n` was
always correct.
However, the right child(ren) of the left child of `n` could still be
at least the key.
Also added some tests which produced the wrong results before this.
Static variables consume memory and can be subject to less
optimization. This variable is only used in 1 place and can be moved
into the function and make it non-static.
This is not ASCII-betical because `_` comes after all the uppercase
characters. Treating `_` as a ` ` (space character), these lists are
now alphabetical.
This is being used by GUID partitions so the first three dash-delimited
fields of the GUID are stored in little endian order but the last two
fields are stored in big endian order, hence it's a representation which
is mixed.
The next commit will destroy overload detection otherwise, so let's add
this constructor. Currently, the same work is already done implicitly
through the implicit `String(StringView)` constructor.
The ArrayLike type concept focuses on array-like data accesses. This
means the ability to randomly index, obtain size information, as well as
being able to expose the storage pointer. The last two already have the
standard APIs `size` and `data`, respectively.
The ArrayLike concept should always be fulfilled by Vector, FixedArray
and Array as the three main array-like types. C-style arrays themselves
of course can't fulfil ArrayLike (which could be considered ironic), but
as we don't use them much anyways this isn't a problem.
Before this commit all consume_until overloads aside from the Predicate
one would consume (and ignore) the stop char/string, while the
Predicate overload would not, in order to keep behaviour consistent,
the other overloads no longer consume the stop char/string as well.
This was used in `HashMap::try_ensure_capacity`, but was missing from
`HashTable`s implementation. No one had used
`HashMap::try_ensure_capacity` before so it went unnoticed!
Apologies for the enormous commit, but I don't see a way to split this
up nicely. In the vast majority of cases it's a simple change. A few
extra places can use TRY instead of manual error checking though. :^)
Rather than casting the FixedPoint to double, format the FixedPoint
directly. This avoids using floating point instruction, which in
turn enables this to be used even in the kernel.
Because AK/Concepts.h includes AK/Forward.h and concepts cannot be
forward declared, slightly losen the FixedPoint template arguments
so that we can forward declare it in AK/Forward.h
In order for this method to be used within LibC itself, avoid using the
floor() method from LibM, which is not available from within LibC. This
header similarly avoids other standard headers as well.
vdbgln() was responsible for ~10% of samples on pv's flamegraph for
RequestServer (under request_did_finish) when loading github.com in
Browser and recording a whole-system profile. This makes that almost
completely disappear.
BigEndianInputBitStream is the Core::Stream API's bitwise input stream
for big endian input data. The functionality and bitwise read API is
almost unchanged from AK::BitStream, except that this bit stream only
supports big endian operations.
As the behavior for mixing big endian and little endian reads on
AK::BitStream is unknown (and untested), it was never done anyways. So
this was a good opportunity to split up big endian and little endian
reading.
Another API improvement from AK::BitStream is the ability to specify
the return type of the bit read function. Always needing to static_cast
the result of BitStream::read_bits_big_endian into the desired type is
adding a lot of avoidable noise to the users (primarily FlacLoader).
Regardless of the backing type that the number would otherwise parse to,
if it is zero and the sign was supposed to be negative then it needs to
be a floating point number to represent the correct value.
This implements an 8-bit front stencil buffer. Stencil operations are
SIMD optimized. LibGL changes include:
* New `glStencilMask` and `glStencilMaskSeparate` functions
* New context parameter `GL_STENCIL_CLEAR_VALUE`
In this particular case, auto is a footgun, because we are very
certain about the type that we want to return. const-correctness
could have been violated (as Vector did), because Iterator did not
enforce that the returned pointer was actually const if the Iterator
was an Iterator over a const container.
Methods marked as const should always only return Foo const&, never
Foo&. Previously, this only worked correctly with Foo, but not with
Foo&: If someone fetched a T const&, this would have been expanded
to Foo& const& which is just Foo&. Therefore, they were able to modify
the elements of the Vector, even though it was marked as const.
This commit fixes that by introducing VisibleType as an alias for
RemoveReference<T> and using it throughout Vector.
There are also other cases where we don't require a mutable reference
to the underlying type, only a const reference (for example in a find
operation). In these cases, we now also correctly expand the type
to Foo const&.
This is particularly useful in the Kernel, where the physical pages of
a VMObject are stored as a FixedArray but often passed around as a Span
from which a new FixedArray should be cloned.
This makes the following code behave as expected:
Variant<int, String> x { some_string() };
x.visit(
[](String const&) {}, // Expectation is for this to be called
[](auto&) {});
This is useful for writing new data at the end of a ByteBuffer. For
instance, with the Stream API:
auto pending_bytes = TRY(stream.pending_bytes());
auto receive_buffer = TRY(buffer.get_bytes_for_writing(
pending_bytes));
TRY(stream.read(receive_buffer));
Except for tangential accessors such as data(), there is no more feature
of FixedArray that is untested after this large expansion of its test
cases. These tests, with the help of the new NoAllocationGuard, also
test the allocation contract that was fixated in the last commit.
Hopefully this builds confidence in future Kernel uses of FixedArray
as well as its establishment in the real-time parts of the audio
subsystem. I'm excited :^)
FixedArray always *almost* had the following allocation guarantees:
There is (possibly) one allocation in the constructor and one (or more)
deallocation(s) in the destructor. No other operation allocates or
deallocates. With this removal of the public clear() method, which
nobody except the test used anyways, those guarantees are now completely
true and furthermore fixated with an explanatory comment.
Because we don't have our LibC on Lagom, the allocation guard global
flag doesn't exist and NoAllocationGuard fails to build on Lagom.
Whoops. For now, just disable NoAllocationGuard's functionality here.
StringView::for_each_split_view allows you to process the splits in a
StringView without needing to allocate a Vector<StringView> to store
each of the parts.
Since we migrated the implementation from the normal split_view path, we
can also re-implement split_view in terms of for_each_split_view.
This mechanism was unsafe to use in any multithreaded context, since
the hook function was invoked on a raw pointer *after* decrementing
the local ref count.
Since we don't use it for anything anymore, let's just get rid of it.
Currently, we define a CaseInsensitiveStringTraits structure for String.
Using this structure for StringView involves allocating a String from
that view, and a second string to convert that intermediate string to
lowercase.
This defines CaseInsensitiveStringViewTraits (and the underlying helper
case_insensitive_string_hash) to avoid allocations.
NoAllocationGuard is an RAII stack guard that prevents allocations
while it exists. This is done through a thread-local global flag which
causes malloc to crash on a VERIFY if it is false. The guard allows for
recursion.
The intended use case for this class is in real-time audio code. In such
code, allocations are really bad, and this is an easy way of dynamically
enforcing the no-allocations rule while giving the user good feedback if
it is violated. Before real-time audio code is executed, e.g. in LibDSP,
a NoAllocationGuard is instantiated. This is not done with this commit,
as currently some code in LibDSP may still incorrectly allocate in real-
time situations.
Other use cases for the Kernel have also been added, so this commit
builds on the previous to add the support both in Userland and in the
Kernel.
FixedArray now doesn't expose any infallible constructors anymore.
Rather, it exposes fallible methods. Therefore, it can be used for
OOM-safe code.
This commit also converts the rest of the system to use the new API.
However, as an example, VMObject can't take advantage of this yet,
as we would have to endow VMObject with a fallible static
construction method, which would require a very fundamental change
to VMObject's whole inheritance hierarchy.
The previous implementation had some pretty short cycles and two fixed
points (1711463637 and 2389024350). If two keys hashed to one of these
values insertions and lookups would loop forever.
This version is based on a standard xorshift PRNG with period 2**32-1.
The all-zero state is usually forbidden, so we insert it into the cycle
at an arbitrary location.
This is just to allow removing the 'clang-format off' directive. This
concept is only used within this header, so it doesn't need to be in the
global namespace.
We have whittled away at the usages of these AK::Vector APIs in the
Kernel. This change disables them from being visible when building
the Kernel to make sure no new usages get added.
If we didn't define our own `move` and `forward` and inject it into the
`std` namespace, then we would error just after trying to `using` those,
if no one has included it before. Now, we will include `utility` from
the STD if we aren't replacing the `std`.
This was a premature optimization from the early days of SerenityOS.
The eternal heap was a simple bump pointer allocator over a static
byte array. My original idea was to avoid heap fragmentation and improve
data locality, but both ideas were rooted in cargo culting, not data.
We would reserve 4 MiB at boot and only ended up using ~256 KiB, wasting
the rest.
This patch replaces all kmalloc_eternal() usage by regular kmalloc().
This is an interface to downcast(), which degrades errors into runtime
errors, and allows seemingly-correct-but-not-quite constructs like the
following to appear to compile, but fail at runtime:
Variant<NonnullRefPtr<T>, U> foo = ...;
Variant<RefPtr<T>, U> bar = foo;
The expectation here is that `foo` is converted to a RefPtr<T> if it
contains one, and remains a U otherwise, but in reality, the
NonnullRefPtr<T> variant is simply dropped on the floor, and the
resulting variant becomes invalid, failing the assertion in downcast().
This commit adds a Variant<Ts...>(Variant<NewTs...>) constructor that
ensures that no alternative can be left out at compiletime, for the
users that were using this interface for merely increasing the number of
alternatives (for instance, LibSQL's Value class).
This is a raffinement of 49cbd4dcca.
Previously, the container was scanned to compute the size in the unhappy
path. Now, using `all_of` happy and unhappy path should be fast.
In order to reduce our reliance on __builtin_{ffs, clz, ctz, popcount},
this commit removes all calls to these functions and replaces them with
the equivalent functions in AK/BuiltinWrappers.h.
While watching Andreas' most recent video, I noticed that this function
only worked with 32 bit values, but was a serious performance
bottleneck for the kernel. As such, I reworked it to use `size_t`, so
it now can switch to 64-bit sweeps on 64-bit platforms. This caused
test-js to go from 12.5 seconds hot to 11.5 seconds hot on my machine
when running on KVM x86_64.
The goal of this file is to enable C++ overloaded functions for
standard builtin functions that we use. It contains fallback
implementations for systems that do not have the builtins available.
Before, if we couldn't read enough data out of the buffer, we would re-
fill the buffer and recursively call read(), which in turn reads data
from the buffer into the resliced target span. This incurs very
intensive superflous memmove's when large chunks of data are read from
a buffered stream.
This commit changes the behavior so that when we exhaust the buffer, we
first read any necessary additional data directly into the target, then
fill up the buffer again. Effectively, this results in drastically
reduced overhead from Buffered when reading large contiguous chunks.
Of course, Buffered is designed to speed up data access patterns with
small frequent reads, but it's nice to be able to combine both access
patterns on one stream without penalties either way.
The final performance gain is about an additional 80% of abench decoding
speed.
This unbreaks the /var/run/utmp system which starts out as an empty
string, and is then turned into an object by the first update.
This isn't necessarily the best way for this to work, but it's how
it used to work, so this just fixes the regression for now.
This fixes at least half of our LibC includes in the kernel. The source
of truth for errno codes and their description strings now lives in
Kernel/API/POSIX/errno.h as an enumeration, which LibC includes.
This is the same strategy that LLVM's compiler-rt uses to make sure that
each UBSAN error is only reported once, when UBSAN is *not* deadly.
Otherwise, each time we head through a UB codepath, we will log the same
error over and over. That behavior just adds noise to the logs and makes
it nearly impossible to run binaires that have some common code path
with flagged UB in them.
compiler-rt goes the extra step to make sure the "clear" action is
atomic, but we don't really have that many multi-threaded apps gettting
tested with UBSAN yet, so we can add that later.
This will allow us to avoid some potentially expensive type conversion
during lookup, like form String to StringView, which would allocate
memory otherwise.
Instead of checking __linux__ macro directly, the code should check if
this macro is defined. This is already done correctly a couple of lines
above.
I ran into this when trying to build libjs-test262 on MacOS where I got
the following error message
error: "__linux__" is not defined, evaluates to 0 [-Werror=undef]
This is a convenience template that implements reference count
forwarding. This means that an object forwards ref() and unref() to
another object.
We can use this when two ref-counted objects need to keep each other
alive. This situation poses two problems:
- Using 2x RefPtr would cause a ref cycle and leak both objects.
- Using 2x WeakPtr would allow one of them to be destroyed early.
With RefCountForwarder, only one of the objects has a ref count. The
object with the ref count points to the forwarding object by using a
non-counting smart pointer (OwnPtr or NonnullOwnPtr). Thus, both objects
are kept alive by the same ref count, and they can safely point to each
other without worrying about disjoint lifetimes.
Note that the return type for the non-const method error() changed. This
is most likely an accident, hidden by the fact that ErrorType typically
is Error.
This makes it an error to not do something with a returned smart
pointer, which should help prevent mistakes. In cases where you do need
to ignore the value, casting to void will placate the compiler.
I did have to add comments to disable clang-format on a couple of lines,
where it wanted to format the code like this:
```c++
private : NonnullRefPtr() = delete;
```
Previously, Vector::extend for a moved vector would move the other
vector into this vector if this vector was empty, thereby throwing away
existing allocated capacity. Therefore, this commit allows the move to
only happen if this vector's capacity is too small to fit the other
vector. This will also alleviate bugs where callers relied on the
capacity to never shrink with calls to unchecked_append, extend and the
like.
This mirrors the existence of append() for data pointers and is very
useful when the program needs to have a guarantee of no allocations,
as is necessary for real-time audio.
During the build process on macOS, multiple versions of <unistd.h> were
being included (Apple's version and GCC's version). It appears that
all other places #include the version from GCC, but in Platform.h the
Apple header was being used. GCC's <unistd.h> is wrapped in
`extern "C"`, while Apple's is not. This causes a conflicting
declaration, so we need to wrap the #include with extern "C".
Issue has been observed on macOS Mojave.
See https://github.com/microsoft/vcpkg/issues/11320 for a similar issue.
This creates an error that contains the name of the syscall that failed.
This allows error handlers to print out the name of the call if they
want to. :^)
Avoid including a per-translation unit copy of all these functions.
Also, drive-by two clang-tidy fixes for readability-qualified-auto and
readability-implicit-bool-conversion.
Currently TimeManagement wont compile on AARCH64, so it is not included.
This creates a link error since format.cpp now relies on functionality
in TimeManagement.cpp to add timestamps to log lines.
This PR disables that functionality for AARCH64 builds until
TimeManagement will compile.
This is a naive implementation based on the symmetry with `asin`.
Before, I'm not really sure what we were doing, but it was returning
wildly incorrect results.
This isn't a complete conversion to ErrorOr<void>, but a good chunk.
The end goal here is to propagate buffer allocation failures to the
caller, and allow the use of TRY() with formatting functions.
Also add slightly richer parse errors now that we can include a string
literal with returned errors.
This will allow us to use TRY() when working with JSON data.
This feels like it was a refactor transition kind of conversion. The
places that were relying on it can easily be changed to explicitly ask
for the ptr() or a new vaddr() method on Userspace<T*>.
FlatPtr can still implicitly convert to Userspace<T> because the
constructor is not explicit, but there's quite a few more places that
are relying on that conversion.
