Lock each directory before entering it so when using -j, the same
dependency isn't built more than once at a time.
This doesn't get full -j parallelism though, since one make child
will be sitting idle waiting for flock to receive its lock and
continue making (which should then do nothing since it will have
been built already). Unfortunately there's not much that can be
done to fix that since it can't proceed until its dependency is
built by another make process.
Allow everything to be built from the top level directory with just
'make', cleaned with 'make clean', and installed with 'make
install'. Also support these in any particular subdirectory.
Specifying 'make VERBOSE=1' will print each ld/g++/etc. command as
it runs.
Kernel and early host tools (IPCCompiler, etc.) are built as
object.host.o so that they don't conflict with other things built
with the cross-compiler.
Using int was a mistake. This patch changes String, StringImpl,
StringView and StringBuilder to use size_t instead of int for lengths.
Obviously a lot of code needs to change as a result of this.
This patch reduces the O(n) tab completion to something like O(log(n)).
The cache is just a sorted vector of strings and we binary search it to
get a string matching our input, and then check the surrounding strings
to see if we need to remove any characters. Also we no longer stat each
file every time.
Also added an #include in BinarySearch since it was using size_t. Oops.
If `export` is called, we recache. Need to implement the `hash` builtin
for when an executable has been added to a directory in PATH.
This is a special specifier that does not output anything to the stream,
but saves the number of already output chars to the provided pointer.
This is apparently used by GNU Nano.
binary_search takes a haystack, a size, a needle and a compare function.
The compare function should return negative if a < b, positive if a > b
and 0 if a == b. The "sane default" compare function is integral_compare
which implements this with subtraction a - b.
binary_search returns a pointer to a matching element, NOT necessarily
the FIRST matching element. It returns a nullptr if the element was not
found.
This patch includes tests for binary_search.
It's missing query string parsing from new URLs, but you can set the
query string programmatically, and it will be part of the URL when
serialized through to_string().
We were forgetting to adopt the WeakLink, causing a reference leak.
This ended up costing us one allocation per exec(), with this stack:
kmalloc_impl()
Inode::set_vmo()
InodeVMObject::create_with_inode()
Process::do_exec()
Process::exec()
Process::sys$execve()
This was a pain to track down, in the end I caught it by dumping out
every live kmalloc pointer between runs and diffing the sets. Then it
was just a matter of matching the pointer to a call stack and looking
at what went wrong. :^)
I'll be reconstructing parts of the VisualBuilder application here and
then we can retire VisualBuilder entirely once all the functionality
is available in HackStudio.
Since NonnullRefPtr and NonnullOwnPtr cannot be null, it is pointless
to convert them to a bool, since it would always be true.
This patch makes it an error to null-check one of these pointers.
It's too dang frustrating that we actually crash whenever we hit some
unimplemented printf specifier. Let's just log the whole format string
and carry on as best we can.
Add dedicated internal types for Int64 and UnsignedInt64. This makes it
a bit more straightforward to work with 64-bit numbers (instead of just
implicitly storing them as doubles.)
This is just a wrapper around strstr() for now. There are many better
ways to search for a string within a string, but I'm just adding a nice
API at the moment. :^)
Previously we would not run destructors for items in a CircularQueue,
which would lead to memory leaks.
This patch fixes that, and also adds a basic unit test for the class.
ELFLoader::layout() had a "failed" variable that was never set. This
patch checks the return value of each hook (alloc/map section and tls)
and fails the load if they return null.
I also needed to patch Process so that the alloc_section_hook and
map_section_hook actually return nullptr when allocating a region fails.
Fixes#664 :)
This class inherits from CircularQueue and adds the ability dequeue
from the end of the queue using dequeue_end().
Note that I had to make some of CircularQueue's fields protected to
properly implement dequeue_end.
This kind of thing is a bit annoying. On Serenity, size_t is the same
size as u32, but not the same type. Because of "long" or whatever.
This patch makes String not complain about duplicate overloads.
The former allows you to inspect the string while it's being built.
It's an explicit method rather than `operator StringView()` because
you must remember you can only look at it in between modifications;
appending to the StringBuilder invalidates the StringView.
The latter lets you clear the state of a StringBuilder explicitly, to
start from an empty string again.
This simplifies the ownership model and makes Region easier to reason
about. Userspace Regions are now primarily kept by Process::m_regions.
Kernel Regions are kept in various OwnPtr<Regions>'s.
Regions now only ever get unmapped when they are destroyed.
`AK::String` can now be reversed via AK::String::reverse(). This makes
life a lot easier for functions like `itoa()`, where the output
ends up being backwards. Very much not like the normal STL
(which requires an `std::reverse` object) way of doing things.
A call to reverse returns a new `AK::String` so as to not upset any
of the possible references to the same `StringImpl` shared between
Strings.
The old implementation tried to move forward as long as the current
byte looks like a UTF-8 character continuation byte (has its two
most significant bits set to 10). This is correct as long as we assume
the string is actually valid UTF-8, which we do (we also have a separate
method that can check whether it is the case).
We can't, however, assume that the data after the end of our string
is also valid UTF-8 (in fact, we're not even allowed to look at data
outside out string, but it happens to a valid memory region most of
the time). If the byte after the end of our string also has its most
significant bits set to 10, we would move one byte forward, and then
fail the m_length > 0 assertion.
One way to fix this would be to add a length check inside the loop
condition. The other one, implemented in this commit, is to reimplement
the whole function in terms of decode_first_byte(), which gives us
the length as encoded in the first byte. This also brings it more
in line with the other functions around it that do UTF-8 decoding.
This patch adds support for TLS according to the x86 System V ABI.
Each thread gets a thread-specific memory region, and the GS segment
register always points _to a pointer_ to the thread-specific memory.
In other words, to access thread-local variables, userspace programs
start by dereferencing the pointer at [gs:0].
The Process keeps a master copy of the TLS segment that new threads
should use, and when a new thread is created, they get a copy of it.
It's basically whatever the PT_TLS program header in the ELF says.
This was a workaround to be able to build on case-insensitive file
systems where it might get confused about <string.h> vs <String.h>.
Let's just not support building that way, so String.h can have an
objectively nicer name. :^)
Passing these through the generic JsonValue path was causing us to
instantiate temporary JsonValues that incurred a heap allocation.
This avoids that by adding specialized overloads for string types.
Right now if we encounter an unknown character, printf (and its related
functions) fail in a really bad way, where they forget to pull things off
the stack. This usually leads to a crash somewhere else, which is hard to
debug.
This patch makes printf abort as soon as it encounters a formatting
character that it can't handle. This is not the optimal solution, but it
is an improvement for debugging.
Utf8View wraps a StringView and implements begin() and end() that
return a Utf8CodepointIterator, which parses UTF-8-encoded Unicode
codepoints and returns them as 32-bit integers.
This is the first step towards supporting emojis in Serenity ^)
https://github.com/SerenityOS/serenity/issues/490
When printing hex numbers, we were printing the wrong thing sometimes. This
was because we were dividing the digit to print by 15 instead of 16. Also,
dividing by 16 is the same as shifting four bits to the right, which is a
bit closer to our actual intention in this case, so let's use a shift
instead.
This way, primitive JsonValue serialization is still handled by
JsonValue::serialize(), but JsonArray and JsonObject serialization
always goes through serializer classes. This is no less efficient
if you have the whole JSON in memory already.
We use consumable annotations to catch bugs where you get the .value()
of an Optional before verifying that it's okay.
The bug here was that only has_value() would set the consumed state,
even though operator bool() does the same job.
Normally you want to access the T&, but sometimes you need to grab at
the NonnullPtr, for example when moving it out of the Vector before
a call to remove(). This is generally a weird pattern, but I don't have
a better solution at the moment.
Using the new get_process_name() syscall, we can automatically prefix
all userspace debug logging.
Hopefully this is more helpful than annoying. We'll find out! :^)
Add the concept of a PeekType to Traits<T>. This is the type we'll
return (wrapped in an Optional) from HashMap::get().
The PeekType for OwnPtr<T> and NonnullOwnPtr<T> is const T*,
which means that HashMap::get() will return an Optional<const T*> for
maps-of-those.
Okay, so, OwnPtr<T>::release_nonnull() returns a NonnullOwnPtr<T>.
It assumes that the OwnPtr is non-null to begin with.
Note that this removes the value from the OwnPtr, as there can only be
a single owner.
The printf formatting mini-language actually allows you
to pass a '*' character in place of the fill width specification,
in which case it eats one of the passed in arguments and uses it
as width, so implement that.
This replaces Optional<T>(U&&) which clang-tidy complained may hide the
regular copy and move constructors. That's a good point, clang-tidy,
and I appreciate you pointing that out!
This allows us to take advantage of the now-optimized (to do memmove())
Vector::append(const T*, int count) for collecting these strings.
This is a ~15% speedup on the load_4chan_catalog benchmark.
This can definitely be improved with better trivial type detection and
by using the TypedTransfer template in more places.
It's a bit annoying that we can't get <type_traits> in Vector.h since
it's included in the toolchain compilation before we have libstdc++.
This has several significant changes to the networking stack.
* Significant refactoring of the TCP state machine. Right now it's
probably more fragile than it used to be, but handles quite a lot
more of the handshake process.
* `TCPSocket` holds a `NetworkAdapter*`, assigned during `connect()` or
`bind()`, whichever comes first.
* `listen()` is now virtual in `Socket` and intended to be implemented
in its child classes
* `listen()` no longer works without `bind()` - this is a bit of a
regression, but listening sockets didn't work at all before, so it's
not possible to observe the regression.
* A file is exposed at `/proc/net_tcp`, which is a JSON document listing
the current TCP sockets with a bit of metadata.
* There's an `ETHERNET_VERY_DEBUG` flag for dumping packet's content out
to `kprintf`. It is, indeed, _very debug_.
Keep a 256-entry string cache during parse to avoid creating some new
strings when possible. This cache is far from perfect but very cheap.
Since none of the strings are transient, this only costs us a couple of
pointers and a bit of ref-count manipulation.
The cache hit rate on 4chan_catalog.json is ~33% and the speedup on
the load_4chan_catalog benchmark is ~7%.
I was able to get parsing time down to about 1/3 of the original time
by using callgrind+kcachegrind. There's definitely more improvements
that can be made here, but I'm gonna be happy with this for now. :^)
- Return more specific types from parse_array() and parse_object().
- Don't create a throwaway String in extract_while().
- Use a StringView in parse_number() to avoid a throwaway String.
This is a shameless copy-paste of String::to_int(). We should find some
way to share this code between String and StringView instead of having
two duplicate copies like this.
Use AK::exchange() to switch out the internal storage. Also mark these
functions with [[nodiscard]] to provoke an compile-time error if they
are called without using the return value.
This gives us much better error messages when you try to use them.
Without this change, it would complain about the absence of functions
named ref() and deref() on RefPtr itself. With it, we instead get a
"hey, this function is deleted" error.
Change operator=(T&) to operator=T(const T&) also, to keep assigning
a const T& to a NonnullRefPtr working.
Instead of aborting the program when we hit an assertion, just print a
message and keep going.
This allows us to write tests that provoke assertions on purpose.
There was a bug in the "prepend_vector_object" test but it was masked
by us not printing failures. (The bug was that we were adding three
elements to the "objects" vector and then checking that another
vector called "more_objects" indeed had three elements. Oops!)
This is a complement to append() that works by constructing the new
element in-place via placement new and forwarded constructor arguments.
The STL calls this emplace_back() which looks ugly, so I'm inventing
a nice word for it instead. :^)
It doesn't seem sane to try to iterate over a HashTable while it's in
the middle of being cleared. Since this might cause strange problems,
this patch adds an assertion if an iterator is constructed during
clear() or rehash() of a HashTable.
An operation often has two pieces of underlying information:
* the data returned as a result from that operation
* an error that occurred while retrieving that data
Merely returning the data is not good enough. Result<> allows exposing
both the data, and the underlying error, and forces (via clang's
consumable attribute) you to check for the error before you try to
access the data.
Put simply, Error<> is a way of forcing error handling onto an API user.
Given a function like:
bool might_work();
The following code might have been written previously:
might_work(); // but what if it didn't?
The easy way to work around this is of course to [[nodiscard]] might_work.
But this doesn't work for more complex cases like, for instance, a
hypothetical read() function which might return one of _many_ errors
(typically signalled with an int, let's say).
int might_read();
In such a case, the result is often _read_, but not properly handled. Like:
return buffer.substr(0, might_read()); // but what if might_read returned an error?
This is where Error<> comes in:
typedef Error<int, 0> ReadError;
ReadError might_read();
auto res = might_read();
if (might_read.failed()) {
switch (res.value()) {
case EBADF:
...
}
}
Error<> uses clang's consumable attributes to force failed() to be
checked on an Error instance. If it's not checked, then you get smacked.
We had some kernel-specific gizmos in AK that should really just be in the
Kernel subdirectory instead. The only thing remaining after moving those
was mmx_memcpy() which I moved to the ARCH(i386)-specific section of
LibC/string.cpp.
So we already have ByteBuffer::wrap() which is like a StringView for random
data. This might not be the best abstraction actually, but this will be
immediately useful so let's add it.
Clang loses the typestate when passing NonnullRefPtr's via lambda captures.
This is unfortunate, but not much we can do about it. Allowing ptr() makes
it possible to use captured NonnullRefPtrs as you'd expect.
Add an "ElementType" typedef to NonnullOwnPtr and NonnullRefPtr to allow
clients to easily find the pointee type. Then use this to remove a template
argument from NonnullPtrVector. :^)
It's not possible to grow one of these vectors beyond what's already in them
since it's not possible to default-construct Nonnull{Own,Ref}Ptr.
Add Vector::shrink() which can be used when you want to shrink the Vector
and delete resize() from the specialized Vectors.
This works just like NonnullRefPtr, except for NonnullOwnPtr's instead.
NonnullOwnPtrVector<T> inherits from Vector<NonnullOwnPtr<T>>, and adds some
comforts on top, like making accessors return T& so we can chase dots (.)
instead of arrows (->) :^)
This is just like OwnPtr (also single-owner), except it cannot be null.
NonnullOwnPtr is perfect as the return type of functions that never need to
return nullptr.
It's also useful as an argument type to encode the fact that the argument
must not be nullptr.
The make<Foo>() helper is changed to return NonnullOwnPtr<Foo>.
Note: You can move() out of a NonnullOwnPtr, and after that the object is
in an invalid state. Internally it will be a nullptr at this point, so we'll
still catch misuse, but the only thing that should be done in this state
is running the destructor. I've used consumable annotations to generate some
warnings when using a NonnullOwnPtr after moving from it, but these only
work when compiling with clang, so be aware of that.
Restructure the makefile a little so it only builds objects once, and
then run them on make clean.
This is a little slower (since we're relinking tests each makeall), but
it also ensures that it will work.
And use it in the scheduler.
IntrusiveList is similar to InlineLinkedList, except that rather than
making assertions about the type (and requiring inheritance), it
provides an IntrusiveListNode type that can be used to put an instance
into many different lists at once.
As a proof of concept, port the scheduler over to use it. The only
downside here is that the "list" global needs to know the position of
the IntrusiveListNode member, so we have to position things a little
awkwardly to make that happen. We also move the runnable lists to
Thread, to avoid having to publicize the node.
The to_foo() functions are for converting when you might not be sure of the
underlying value type. The as_foo() family assumes that you know exactly
what the underlying value type is.
Meet TStyle. It allows you to write things like this:
dbg() << TStyle(TStyle::Red, TStyle::Bold) << "Hello, friends!";
Any style used will be reset along with the newline emitted when the dbg()
temporary goes out of scope. :^)
This can definitely be improved, but I think it's a decent place to start.
This is the same as calling FileSystemPath(foo).string(). The majority of
clients only care about canonicalizing a path, so let's have an easy way
to express that.
We shouldn't allow constructing e.g an OwnPtr from a RefPtr, and similar
conversions. Instead just delete those functions so the compiler whines
loudly if you try to use them.
This patch also deletes constructing OwnPtr from a WeakPtr, even though
that *may* be a valid thing to do, it's sufficiently weird that we can
make the client jump through some hoops if he really wants it. :^)
This patch removes copy_ref() from RefPtr and NonnullRefPtr. This means that
it's now okay to simply copy these smart pointers instead:
- RefPtr = RefPtr // Okay!
- RefPtr = NonnullRefPtr // Okay!
- NonnullRefPtr = NonnullRefPtr // Okay!
- NonnullRefPtr = RefPtr // Not okay, since RefPtr can be null.
I had a silly ambition that we would avoid unnecessary ref count churn by
forcing explicit use of "copy_ref()" wherever a copy was actually needed.
This was making RefPtr a bit clunky to work with, for no real benefit.
This patch adds the missing copy construction/assignment stuff to RefPtr.
You can currently use this to detect the CPU architecture like so:
#if ARCH(I386)
...
#elif ARCH(X86_64)
...
#else
...
#endif
This will be helpful for separating out architecture-specific code blocks.
Instead of computing the path length inside the syscall handler, let the
caller do that work. This allows us to implement to new variants of open()
and creat(), called open_with_path_length() and creat_with_path_length().
These are suitable for use with e.g StringView.
This makes me wonder if the open() syscall should take characters+length
and we'd compute the length at the LibC layer instead. That way we could
also provide an optional non-POSIX open() that takes the length directly..
This allows you to do things like:
vector.insert_before_matching(value, [](auto& entry) {
return value < entry;
});
Basically it scans until it finds an element that matches the condition
callback and then inserts the new value before the matching element.
The first implementation class is DebugLogStream, which can be used like so:
dbg() << "Hello friends, I am " << m_years << " years old!";
Note that it will automatically print a newline when the object created by
dbg() goes out of scope.
This API will grow and evolve, so let's see what we end up with :^)
Instead of manually doing String::format("%d"/"%u") everywhere, let's have
a String API for this. It's just a wrapper around format() for now, but it
could be made more efficient in the future.
Solve this by adding find() overloads to HashTable and SinglyLinkedList
that take a templated functor for comparing the values.
This allows HashMap to call HashTable::find() without having to create
a temporary Entry for use as the table key. :^)
This is prep work for supporting HashMap with NonnullRefPtr<T> as values.
It's currently not possible because many HashTable functions require being
able to default-construct the value type.
Update ProcessManager, top and WSCPUMonitor to handle the new format.
Since the kernel is not allowed to use floating-point math, we now compile
the JSON classes in AK without JsonValue::Type::Double support.
To accomodate large unsigned ints, I added a JsonValue::Type::UnsignedInt.
The LibC build is a bit complicated, since the toolchain depends on it.
During the toolchain bootstrap, after we've built parts of GCC, we have
to stop and build Serenity's LibC, so that the rest of GCC can use it.
This means that during that specific LibC build, we don't yet have access
to things like std::initializer_list.
For now we solve this by defining SERENITY_LIBC_BUILD during the LibC
build and excluding the Vector/initializer_list support inside LibC.
Get rid of the ConstIterator classes for these containers and use templated
FooIterator<T, ...> and FooIterator<const T, ...> helpers.
This makes the HashTable class a lot easier to read.
This means you can now do this:
void harmonize(NonnullRefPtrVector<Voice>& voices)
{
for (auto& voice : voices) {
voice.sing(); // Look, no "->"!
}
}
Pretty dang cool :^)
This is a slot-in convenience replacement for Vector<NonnullRefPtr<T>> that
makes accessors return T& instead of NonnullRefPtr<T>&.
Since NonnullRefPtr guarantees non-nullness, this allows you to access these
vector elements using dot (.) rather than arrow (->). :^)
This avoids putting pressure on kmalloc() during backtrace symbolication.
Since we dump backtrace for every process that exits, this is actually a
decent performance improvement for things like GCC that chain a lot of
processes together.
This parser assumes that the JSON is well-formed and will choke horribly
on invalid input.
Since we're primarily interested in parsing our own output right now, this
is less of a problem. Longer-term we're gonna need something better. :^)
- Delete the default constructor instead of just making it private.
It's never valid to create an empty NonnullRefPtr.
- Add copy assignment operators. I originally omitted these to force use
of .copy_ref() at call sites, but the hassle/gain ratio is minuscule.
- Allow calling all the assignment operators in all consumable states.
This codifies that it's okay to overwrite a moved-from NonnullRefPtr.
It's kinda funny how I can make a mistake like this in Serenity and then
get so used to it by spending lots of time using this API that I start to
believe that this is how printf() always worked..
There's no need for a member char* m_characters if we always store them
in the inline buffer. So with this patch, we now do.
After that, rearrange the members a bit for ideal packing. :^)
We'll now try to detect crashes that were due to dereferencing nullptr,
uninitialized malloc() memory, or recently free()'d memory.
It's not perfect but I think it's pretty good. :^)
Also added some color to the most important parts of the crash log,
and added some more modes to /bin/crash for exercising this code.
Fixes#243.
