The amount of aliases in the likely-subtags dataset is quite large, so
this also needed to change the way the data is generated. Otherwise, the
compiler would complain about the size of the generated code.
Previously, a static method was generated that would effectively parse
the dataset into a HashMap of Unicode::LanguageID at runtime. We now
perform that parsing at generation-time, and instead generate an Array
of a structure similar to Unicode::LanguageID (we cannot use the same
structure because it contains String and Optional, which cannot be used
at compile-time).
This option is similar to the qgrub option, but instead of starting a
QEMU PIIX4 machine, it starts a QEMU Q35 machine, booting a grub image
disk within it.
The DOM specification says that the primary use case for these is to
give Promises abort semantics. It is also a prerequisite for Fetch,
as it is used to make Fetch abortable.
a
CLDR contains a set of likely subtag data where, given a locale, you can
resolve what is the most likely language, script, or territory of that
locale. This data is needed for resolving territory aliases. These
aliases might contain multiple territories, and we need to resolve which
of those territories is most likely correct for a locale.
Note that the likely subtag data is quite huge (a few thousand entries).
As an optimization encouraged by the spec, we only generate the smallest
subset of this data that we actually need (about 150 entries).
Most alias substitutions are "simple", meaning that alias matching is
done by examining a single locale subtag. However, there are a handful
of "complex" aliases where matching is done by examining multiple
subtags. For example, the variant subtag "lojban" causes the locale
"art-lojban" to be canonicalized to "jbo", but only when the language
subtag is "art" (i.e. this should not occur for the locale "en-lojban").
This generates a method to perform complex alias matching.
CLDR contains a set of aliases for languages, territories, etc. that no
longer are meant to be used (e.g. due to deprecation). For example, the
language "aam" is deprecated and should be canonicalized as "aas".
This is needed so all headers and files exist on disk, so that
the sonar cloud analyzer can find them when executing the compilation
commands contained in compile_commands.json, without actually building.
Co-authored-by: Andrew Kaster <akaster@serenityos.org>
This should help prevent deadlocks where a thread blocks on a Mutex
while interrupts are disabled, and makes it impossible for the holder of
the Mutex to make forward progress because it cannot be scheduled in.
Hide it behind a new debug macro LOCK_IN_CRITICAL_DEBUG for now, because
Ext2FS takes a series of Mutexes from the page fault handler, which
executes with interrupts disabled.
This environment variable is already widely used across our build
scripts and tooling, so serenity.sh should respect it as well.
It still uses i686 as the fallback.
This allows us to remove all the add_subdirectory calls from the top
level CMakeLists.txt that referred to targets linking LagomCore.
Segregating the host tools and Serenity targets helps us get to a place
where the main Serenity build can simply use a CMake toolchain file
rather than swapping all the compiler/sysroot variables after building
host libraries and tools.
Moving this helper CMake file to the centralized Meta/CMake folder helps
to get a better grasp on what extra files are required for the build,
and what files are generated.
While we're at it, don't use add_compile_definitions for
ENABLE_UNICODE_DATA, which only needs to be seen by LibUnicode sources.
By using SerenityOS_SOURCE_DIR we can make custom targets and commands
agnostic to the actual location of the root CMakeLists directory.
All we care about is the root of the SerenityOS project.
compile_gml, compile_ipc, and generate_state_machine all use host
tools to generate sources for the target build. As part of trying to
organize host tools into a common area, let's move these helper rules to
a common file that we can add other host tools to later. And, keep the
host tool helpers separate from the CMake target helpers for apps and
libraries.
It's hard to follow how all the functions in the utils.cmake helper file
flow together, so let's move the pieces that are related to each other
into specialized helpers. First up: all the ConfigureComponents related
properties and functions.
On my machine (c), /mnt/c/Windows/System32 is not on the PATH by
default. This causes reg.exe to fail, which is responsible for detecting
the presence of QEMU. By putting this path on the PATH on WSL, it will
always work regardless of the specific PATH configuration, and QEMU is
always detected.
There is a bit of a race here between the Fuzzer and non-Fuzzer Lagom
builds. If the Unicode caches are empty, and the Fuzzer build completes
first, then the UCD and CLDR directories will be empty or won't exist.
Skip handling the Unicode caches for this build.
The IRC Client application made some sense while our main communication
hub was an IRC channel. Now that we've moved on, IRC is just a random
protocol with no particular relevance to this project.
This also has the benefit of removing one major client of the single-
process Web::InProcessWebView class.
This command copies the project's source tree to
/home/anon/Source/serenity in the built disk image.
This will be useful for working on serenity inside serenity :^)
The on-target pipelines have a timeout of 6 hours to allow time for a
clean toolchain + Serenity build. Tests should time out much sooner than
that though.
Caches on Azure are immutable - so if a cache changes, but its key does
not, then the cache is not updated. Include a timestamp in the ccache
key so that we always push an updated cache from the master branch. Then
use a subkey without the timestamp to pull the cache.
We use a similar trick on GitHub Actions.
Previously debug-kernel.sh wouldn't detect source if you launched the
script from the wrong path. By explicitly setting the name, source will
be loaded by gdb in all situations.
We want to use use the 'du' option '--apparent-size' which is a
GNU coreutils extension. GNU coreutils is a build dependency so
we know it is available. With this commit we first try to pick up
du as 'gdu', and if that fails, try 'du' instead.
There was previously a case where the build-image-qemu.sh script
decided to mount an existing disk image, but without creating the
memory disk device and recording its /dev file name.
After this commit, We create the memory disk device just before
it is used to mount the disk image.
We were over-hashing for the GNU build on GitHub Actions by including
the LLVM patch as well. The GNU Toolchain doesn't care about our LLVM
patches.
For Azure, fix the inversion of the condition for which jobs check which
Build*.sh script, and add the Toolchain patch files to the cache
hash calculation.
The top-level CMakeLists.txt already automatically detects ccache, but
CI will invoke CMake with Lagom's CMakeLists.txt. Add an option to Lagom
to do the same detection.
The function arguments almost always optimized away, so you never get
much value out of showing these in the default back trace view, it just
adds a bunch of extra stuff that you need to visual wade through.
So lets disable showing them.
Similarly to the LibCpp parser regression tests, these tests run the
preprocessor on the .cpp test files under
Userland/LibCpp/Tests/preprocessor, and compare the output with existing
.txt ground truth files.
This script will instantiate a HackStudio template into a project on the
host. It currently supports all templates used by HackStudio.
To avoid having to maintain compatibility between other shells and the
Serenity shell in the postcreate scripts, we build the Serenity shell
with Lagom and use that to run the script.
Having lots of small files in Base/ may require more inodes in the
ext2 filesystem than the format utility sets aside by default. Let's
make a more educated guess since we have a rough idea of how many
inodes we need by counting files and directories.
This contains all the bits and pieces necessary to build a Clang binary
that will correctly compile SerenityOS.
I had some trouble with getting LLVM building with a single command, so
for now, I decided to build each LLVM component in a separate command
invocation. In the future, we can also make the main llvm build step
architecture-independent, but that would come with extra work to make
library and include paths work.
The binutils build invocation and related boilerplate is duplicated
because we only use `objdump` from GNU binutils in the Clang toolchain,
so most features can be disabled.
This seemed like a good idea at the time to avoid an unnecessary
dependency on qemu-system-i386. However this makes debugging the
kernel with GDB more difficult because GDB assumes that the QEMU
architectures matches the kernel architecture.
Before Libraries was moved to Userland/Libraries syslog.h had a bunch
of manually aligned defines and array initializations.
Andreas seems to have formatted the file with clang-format as part of
that file move. Since syslog.h is now properly formatted, we don't
need to exclude it from the linter list.
This commit implements the ISO 9660 filesystem as specified in ECMA 119.
Currently, it only supports the base specification and Joliet or Rock
Ridge support is not present. The filesystem will normalize all
filenames to be lowercase (same as Linux).
The filesystem can be mounted directly from a file. Loop devices are
currently not supported by SerenityOS.
Special thanks to Lubrsi for testing on real hardware and providing
profiling help.
Co-Authored-By: Luke <luke.wilde@live.co.uk>
As this is a test machine I use personally to test "modern" hardware
setups, it feels quite comfortable to not care too much about VGA with
this type of machine.
Also, we don't actively use the IDE controller on this machine type, so
let's just remove it :^)
This allows one to set their desired parameters for run.sh without the
need to set them in every terminal session or add it to the user account
shell files. If a run-local.sh file exists at the repository root and is
executable, it will be sourced. The file can contain any variables that
are expected to be set in run.sh.
This allows running QEMU inside WSL2 for hosts which have nested KVM
and WSLg support (e.g. Windows 11).
Running QEMU inside the WSL2 VM is slightly slower than running QEMU
on Windows, probably because of how WSLg handles screen updates.
Although it is nice to test the system without too many devices, in
reality bare metal hardware is far more complex than the default skeleon
that QEMU provides. As a preparation of supporting more devices, we
need to ensure we are capable of at least booting on complex hardware
setups without easily-observable problems. Later on, this can be the
foundations of testing new drivers :^)
This gets rid of the following warning message from QEMU on startup:
qemu-system-i386: warning: '-soundhw pcspk' is deprecated, please set a
backend using '-machine pcspk-audiodev=<name>' instead
Fixes#4093.
For users who use a custom kernel with WSL our previous method of
detecting WSL doesn't work. This new check instead detects WSL by
checking if the wslpath utility is available.
This standard CMake option controls whether add_library() calls will
use STATIC or SHARED by default. The flag is set to on by default
since that's what we want for normal CI jobs and local builds and the
test262 runner, but disabled for oss-fuzz builds.
This should finally fix the oss-fuzz build after it was broken in #9017
oss-fuzz un-breakage was verified by running the following commands in
the oss-fuzz repo:
python infra/helper.py build_image serenity
python infra/helper.py build_fuzzers --sanitizer address --engine afl \
--architecture x86_64 serenity /path/to/local/checkout/Meta/Lagom
python infra/helper.py check_build --sanitizer address --engine afl \
--architecture x86_64 serenity
Otherwise we're getting this warning:
WARNING: Image format was not specified for '_disk_image' and probing
guessed raw. Automatically detecting the format is dangerous
for raw images, write operations on block 0 will be restricted.
Specify the 'raw' format explicitly to remove the restrictions.
Previously we'd fall back to using cp if rsync wasn't available. Not
only is this considerably slower it also breaks when some of the files
in the target directory are symlinks because cp tries to dereference
them.
Fixes#8672.
This supports some binary property matching. It does not support any
properties not yet parsed by LibUnicode, nor does it support value
matching (such as Script_Extensions=Latin).
Split the Lagom build into shared libraries to match the Serenity build.
This reduces the cognitive load when trying to edit the Lagom CMakeLists
significantly. It also reduces the amount of source files that must be
compiled to run each test or host program significantly.
Also re-organize all the build rules into sections. And reorganize the
CMakeLists file in general.
By using the power of object libraries and $<TARGET_OBJECTS> we can make
sure to only build TestMain.cpp and JavaScriptTestRunnerMain.cpp once.
Previously we built these cpp files into object files once for every
single test executable. This change reduces the number of total compile
jobs in a Serenity target build by around 100.
LibTTF has a concrete dependency on LibGfx for things like Gfx::Bitmap,
and LibGfx has a concrete dependency in the TTF::Font class in
Gfx::FontDatabase. This circular dependency works fine for Serenity and
Lagom Linux builds of the two libraries. It also works fine for static
library builds on Lagom macOS builds.
However, future changes will make Lagom use shared libraries, and
circular library dependencies are not tolerated in macOS.
You can now see the outline of GUI widgets when hovering them.
For example:
$ export GUI_HOVER_DEBUG=1
$ FileManager
Then move the mouse around in the file manager. :^)
I have set up a commit.verbose variable in my git config,
which shows the patch diff on bottom of the commit message.
Unfortunately the character limit was also applied to the diff,
which meant that I got a false-positive lint error almost every time.
Unfortunately seems like QEMU tries to use SPICE for audio so we need to
explicitly set -audiodev to use either SDL or coreaudio (depending on
platorm)
This enables further work on implementing KASLR by adding relocation
support to the pre-kernel and updating the kernel to be less dependent
on specific virtual memory layouts.
This is primarily to allow using LibUnicode within LibJS and its REPL.
Note: this seems to be the first time that a Lagom dependency requires
generated source files. For this to work, some of Lagom's CMakeLists.txt
commands needed to be re-organized to include the CMake files that fetch
and parse UnicodeData.txt. The paths required to invoke the generator
also differ depending on what is currently building (SerenityOS vs.
Lagom as part of the Serenity build vs. a standalone Lagom build).
This implements a simple bootloader that is capable of loading ELF64
kernel images. It does this by using QEMU/GRUB to load the kernel image
from disk and pass it to our bootloader as a Multiboot module.
The bootloader then parses the ELF image and sets it up appropriately.
The kernel's entry point is a C++ function with architecture-native
code.
Co-authored-by: Liav A <liavalb@gmail.com>
If QEMU has the qemu_vdagent chardev (should be present in QEMU 6.1)
then we use that as the SPICE client.
If qemu_vdagent is not present, no SPICE client will be launched by
default because it makes the display a bit choppy.
Set SERENITY_SPICE to override the default behavior and use your default
SPICE client.
Let's remove the qcmd and q35_cmd options and instead have a simple
"q35" run option. Specifiying the kernel command line was a neat trick
I personally used for many debug sessions, but it seems better to stick
to setting it internally in the kernel or modifying the shell
SERENITY_KERNEL_CMDLINE environment variable to do this.
With this change, we use 6 PCIe root ports in the Q35 machine, and plug
the bochs-display device into one of those PCIe ports.
We plug the bochs-display as function 0 of that device, because
otherwise SeaBIOS and also the kernel will not detect its presence.
Unfortunately we can't just use --chown everywhere because only rsync
version 3.1 and newer support it and the default rsync on macOS is
2.6.9.
Fixes#8711.
SDL brings with it an annoying issue whereby trying to resize the window
before Serenity starts up prevents it from automatically resizing to fit
the screen.
This patch makes the previous behavior (i.e using the GTK backend) the
default unless SERENITY_SCREENS is greater than 1.
This patch introduces the SQLServer system server. This service is
supposed to be the only process/application talking to database storage.
This makes things like locking and caching more reliable, easier to
implement, and more efficient.
In LibSQL we added a client component that does the ugly IPC nitty-
gritty for you. All that's needed is setting a number of event handler
lambdas and you can connect to databases and execute statements on them.
Applications that wish to use this SQLClient class obviously need to
link LibSQL and LibIPC.
This adds a utility program which is essentially a command generator for
CMake. It reads the 'components.ini' file generated by CMake in the
build directory, prompts the user to select a build type and optionally
customize it, generates and runs a CMake command as well as 'ninja
clean' and 'rm -rf Root', which are needed to properly remove system
components.
The program uses whiptail(1) for user interaction.
These are usually incorrect, and people sometimes forget to add the
correct values as a result of them being optional, so they should just
be specified explicitly.
This removes all usages of the non-standard define_property helper
method and replaces all it's usages with the specification required
alternative or with define_direct_property where appropriate.
I didn't add any debug logging to the object rewrite, so this is now
unused. It's much more correct though, so we can get away with adding
ad-hoc logging, should that ever be necessary :^)
Side note: this should have a prefix, i.e. JS_OBJECT_DEBUG. The previous
name is too generic.
All GUI applications currently load all TTF fonts on startup
(to populate the Gfx::FontDatabase. This could probably be smarter.)
Before this patch, everyone would open the files and read them into
heap-allocated storage. Now we simply mmap() them instead. :^)
This reverts commit a808fff2ec.
I highly suspect the reason it was not working before was fixed in
017c5fc7d9, so the SB16 DMA can work even
if we initialize another E1000 network adapter.
QEMU appears to always relay absolute mouse coordinates relative to the
screen that the mouse is pointed to, without any way for us to know
what screen it was. So, when dealing with multiple displays force using
relative coordinates only.
This allows specifying how many screens we should use. This also then
only enables virtio-gpu if more than one display is requested.
This also adds an environment variable SERENITY_QEMU_DISPLAY_BACKEND
which allows overriding the default qemu display backend, as it may
not be available.
Multiboot only supports ELF32 executables. This changes the build
process to build an ELF32 executable which has a 32-bit entry point,
but consists of mostly 64-bit code.
`wasm-as` will do some semantic analysis on the modules, which is not
something we're looking for here.
Instead, use `wat2wasm` to generate the exact module.
Components are a group of build targets that can be built and installed
separately. Whether a component should be built can be configured with
CMake arguments: -DBUILD_<NAME>=ON|OFF, where <NAME> is the name of the
component (in all caps).
Components can be marked as REQUIRED if they're necessary for a
minimally functional base system or they can be marked as RECOMMENDED
if they're not strictly necessary but are useful for most users.
A component can have an optional description which isn't used by the
build system but may be useful for a configuration UI.
Components specify the TARGETS which should be built when the component
is enabled. They can also specify other components which they depend on
(with DEPENDS).
This also adds the BUILD_EVERYTHING CMake variable which lets the user
build all optional components. For now this defaults to ON to make the
transition to the components-based build system easier.
The list of components is exported as an INI file in the build directory
(e.g. Build/i686/components.ini).
Fixes#8048.
This commit adds a bunch of passes, the most interesting of which is a
pass that merges blocks together, and a pass that places blocks that
flow into each other next to each other, and a very simply pass that
removes duplicate basic blocks.
Note that this does not remove the jump at the end of each block in that
pass to avoid scope creep in the passes.
These are the actual structures that allow USB to work (i.e the ones
actually defined in the specification). This should provide us enough
of a baseline implementation that we can build on to support
different types of USB device.
These are pretty common on older LGA1366 & LGA1150 motherboards.
NOTE: Since the registers datasheets for all versions of the chip
besides versions 1 - 3 are still under NDAs i had to collect
several "magical vendor constants" from the *BSD driver and the
linux driver that i was not able to name verbosely, and as such
these are labeled with the comment "vendor magic values".
We call it E1000E, because the layout for these cards is somewhat not
the same like E1000 supported cards.
Also, this card supports advanced features that are not supported on
8254x cards.
This commit initializes the LibVideo library and implements parsing
basic Matroska container files. Currently, it will only parse audio
and video tracks.
Previously, AK::Function would accept _any_ callable type, and try to
call it when called, first with the given set of arguments, then with
zero arguments, and if all of those failed, it would simply not call the
function and **return a value-constructed Out type**.
This lead to many, many, many hard to debug situations when someone
forgot a `const` in their lambda argument types, and many cases of
people taking zero arguments in their lambdas to ignore them.
This commit reworks the Function interface to not include any such
surprising behaviour, if your function instance is not callable with
the declared argument set of the Function, it can simply not be
assigned to that Function instance, end of story.
This adds a new URL parser, which aims to be compliant with the URL
specification (https://url.spec.whatwg.org/). It also contains a
rudimentary data URL parser.
Since I introduced this functionality there has been a steady stream of
people building with `ALL_THE_DEBUG_MACROS` and trying to boot the
system, and immediately hitting this assert. I have no idea why people
try to build with all the debugging enabled, but I'm tired of seeing the
bug reports about asserts we know are going to happen at this point.
So I'm hiding this value under the new ENABLE_ALL_DEBUG_FACILITIES flag
instead. This is only set by CI, and hopefully no-one will try to build
with this thing (It's documented as not recommended).
Fixes: #7527
Previously ByteBuffer::grow() behaved like Vector<T>::resize().
However the function name was somewhat ambiguous - and so this patch
updates ByteBuffer to behave more like Vector<T> by replacing grow()
with resize() and adding an ensure_capacity() method.
This also lets the user change the buffer's capacity without affecting
the size which was not previously possible.
Additionally this patch makes the capacity() method public (again).
Since this program is setuid-root, it should be as simple as possible.
To that end, remove `/etc/plsusers` and use filesystem permissions to
achieve the same thing. `/bin/pls` is now only executable by `root` or
members of the `wheel` group.
Also remove all the logic that went to great lengths to `unveil()` a
minimal set of filesystem paths that may be used for the command.
The complexity-to-benefit ratio did not seem justified, and I think
we're better off keeping this simple.
Finally, remove pledge promises the moment they are no longer needed.
These dbgln's caused excessive load in the WebServer process,
accounting for ~67% of the processing time when serving a webpage
with a bunch of resources like serenityos.org/happy/2nd/.
With the increased volume of PRs being opened and merged lately,
multiple people have complained that the IRC is absolutely flooded with
SerenityBot posts. Remove the IRC notifications from the CI scripts, and
the Meta script that handles parsing the github actions context into
an IRC message.
It seems like overly-specific classes were written for no good reason.
Instead of making each adapter to have its own unique FramebufferDevice
class, let's generalize everything to keep implementation more
consistent.
This reverts commit 5018b3b4b7.
Let's help our future selves by making sure we don't introduce a bug
because of duplicating a network interface name or related bug to this
topic. Therefore, we can have multiple e1000 devices in the system now.
Some folks on discord said adding another e1000 network adapter made it
so we don't have networking on the system anymore.
To fix this, we will use other unsupported PCI device instead.
This allows a developer to easily dump a InlineLinkedList in the
debugger without having to manually running the list.
I needed this for a recent bug investigation in the Kernel.
This also optionally generates a test suite from the WebAssembly
testsuite, which can be enabled via passing `INCLUDE_WASM_SPEC_TESTS`
to cmake, which will generate test-wasm-compatible tests and the
required fixtures.
The generated directories are excluded from git since there's no point
in committing them.
This only tests "can it be parsed", but the goal of this commit is to
provide a test framework that can be built upon :)
The conformance tests are downloaded, compiled* and installed only if
the INCLUDE_WASM_SPEC_TESTS cmake option is enabled.
(*) Since we do not yet have a wast parser, the compilation is delegated
to an external tool from binaryen, `wasm-as`, which is required for the
test suite download/install to succeed.
This *does* run the tests in CI, but it currently does not include the
spec conformance tests.
For each .cpp file in the test suite data, there is a .ast file that
represents the "known good" baseline of the parser result.
Each .cpp file goes through the parser, and the result of
invoking `ASTNode::dump()` on the root node is compared to the
baseline to find regressions.
We also check that there were no parser errors when parsing the .cpp
files.
This is similar to the LibJS test data that resides in
/home/anon/js-tests.
It's more convenient than storing the test programs as raw strings
in the code.
As the parser now flattens out the instructions and inserts synthetic
nesting/structured instructions where needed, we can treat the whole
thing as a simple parsed bytecode stream.
This currently knows how to execute the following instructions:
- unreachable
- nop
- local.get
- local.set
- {i,f}{32,64}.const
- block
- loop
- if/else
- branch / branch_if
- i32_add
- i32_and/or/xor
- i32_ne
This also extends the 'wasm' utility to optionally execute the first
function in the module with optionally user-supplied arguments.
This had very bad interactions with ccache, often leading to rebuilds
with 100% cache misses, etc. Ali says it wasn't that big of a speedup
in the end anyway, so let's not bother with it.
We can always bring it back in the future if it seems like a good idea.
As we removed the support of VBE modesetting that was done by GRUB early
on boot, we need to determine if we can modeset the resolution with our
drivers, and if not, we should enable text mode and ensure that
SystemServer knows about it too.
Also, SystemServer should first check if there's a framebuffer device
node, which is an indication that text mode was not even if it was
requested. Then, if it doesn't find it, it should check what boot_mode
argument the user specified (in case it's self-test). This way if we
try to use bochs-display device (which is not VGA compatible) and
request a text mode, it will not honor the request and will continue
with graphical mode.
Also try to print critical messages with mininum memory allocations
possible.
In LibVT, We make the implementation flexible for kernel-specific
methods that are implemented in ConsoleImpl class.
This commit replaces the former, hand-written parser with a new one that
can be generated automatically according to a state change diagram.
The new `EscapeSequenceParser` class provides a more ergonomic interface
to dealing with escape sequences. This interface has been inspired by
Alacritty's [vte library](https://github.com/alacritty/vte/).
I tried to avoid changing the application logic inside the `Terminal`
class. While this code has not been thoroughly tested, I can't find
regressions in the basic command line utilities or `vttest`.
`Terminal` now displays nicer debug messages when it encounters an
unknown escape sequence. Defensive programming and bounds checks have
been added where we access parameters, and as a result, we can now
endure 4-5 seconds of `cat /dev/urandom`. :D
We generate EscapeSequenceStateMachine.h when building the in-kernel
LibVT, and we assume that the file is already in place when the userland
library is being built. This will probably cause problems later on, but
I can't find a way to do it nicely.
As of ~April 2021 the Meta/build-image-grub.sh script no longer works
for me (on Fedora 34) and fails with the following error:
/usr/sbin/grub2-install: warning: ../grub-core/partmap/msdos.c:403:
your core.img is unusually large. It won't fit in the embedding
area.
/usr/sbin/grub2-install: warning: Embedding is not possible. GRUB
can only be installed in this setup by using blocklists. However,
blocklists are UNRELIABLE and their use is discouraged..
/usr/sbin/grub2-install: error: will not proceed with blocklists.
Changing the size of the boot partition from 32 kiB to 1 MiB (2048
sectors) fixes the issue. This is also described in the following Ubuntu
grub2 bug (as well as 40 duplicates!) from 2012, which suggests the same
fix: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1059827
The Arch Linux wiki also uses 1 MiB in their BIOS/MBR examples for
parted: https://wiki.archlinux.org/title/Parted
I'm not sure why this suddenly stopped working, however I was able to
boot with an image created with this change applied.
Ideally we would never allocate under a spinlock, as it has many
performance and potentially functionality (deadlock) pitfalls.
We violate that rule in many places today, but we need a tool to track
them all down and fix them. This change introduces a new macro option
named `KMALLOC_VERIFY_NO_SPINLOCK_HELD` which can catch these
situations at runtime via an assert.
Previously the CMake options for -fsanitize=address, thread and
undefined were gated behind clang, which was unecessary. Only
-fsanitize=fuzzer is clang-only.
Since the operations are already complicated and will become even more
so soon, let's split them into their own files. We can also integrate
the NumberTheory operations that would better fit there into this class
as well.
This commit doesn't change behaviors, but moves the allocation of some
variables into caller classes.
Make messages which should be fatal, actually fail the build.
- FATAL is not a valid mode keyword. The full list is available in the
docs: https://cmake.org/cmake/help/v3.19/command/message.html
- SEND_ERROR doesn't immediately stop processing, FATAL_ERROR does.
We should immediately stop if the Toolchain is not present.
- The app icon size validation was just a WARNING that is easy to
overlook. We should promote it to a FATAL_ERROR so that people will
not overlook the issue when adding a new application. We can only make
the small icon message FATAL_ERROR, as there is currently one
violation of the medium app icon validation.
This commit introduces the ability to parse the document catalog dict,
as well as the page tree and individual pages. Pages obviously aren't
fully parsed, as we won't care about most of the fields until we
start actually rendering PDFs.
One of the primary benefits of the PDF format is laziness. PDFs are
not meant to be parsed all at once, and the same is true for pages.
When a Document is constructed, it builds a map of page number to
object index, but it does not fetch and parse any of the pages. A page
is only parsed when a caller requests that particular page (and is
cached going forwards).
Additionally, this commit also adds an object_cast function which
logs bad casts if DEBUG_PDF is set. Additionally, utility functions
were added to ArrayObject and DictObject to get all types of objects
from the collections to avoid having to manually cast.
This can currently parse a really simple module.
Note that it cannot parse the DataCount section, and it's still missing
almost all of the instructions.
This commit also adds a 'wasm' test utility that tries to parse a given
webassembly binary file.
It currently does nothing but exit when the parse fails, but it's a
start :^)
This currently (obviously) doesn't support any actual 3D hardware,
hence all calls are done via software rendering.
Note that any modern constructs such as shaders are unsupported,
as this driver only implements Fixed Function Pipeline functionality.
The library is split into a base GLContext interface and a software
based renderer implementation of said interface. The global glXXX
functions serve as an OpenGL compatible c-style interface to the
currently bound context instance.
Co-authored-by: Stephan Unverwerth <s.unverwerth@gmx.de>
Index page:
- Change links from "Man 1" to "Section 1"
Section index pages:
- Change title from "1" to "Section 1 - SerenityOS man pages"
- Change links from "foo" to "foo(1)"
Man pages:
- Change title from "foo" to "foo(1) - SerenityOS man pages"
The data structures here were changed from sets to dicts at some point,
with the sets now having different names - this would crash trying to
subtract two dicts.
Previously the directions omitted that you have to specify
`-CMAKE_CXX_COMPILER` when building the Fuzzers. This
would cause all kinds of weird problems at compilation and
link time. You can't specify one or the other, they must
both be pointing at clang in order for things to work as
experted. Fix this by updating the documentation to specify
that the user should specify both the C and CXX compiler explicitly
to be safe, as well as forcing the cmake clang argument handling
to modify the CXX compiler variable instead of the C version.
This should help with getting commit messages tidy before they pass
through CI's commit linter :^)
For this hook to work pre-commit has to be explicitly installed via:
`pre-commit install --hook-type commit-msg`
Just 'github.actor' may or may not be the actual PR author, but
especially since action runs have to be approved for first-time
contributors, it's been giving us a lot of incorrect results.
Most of the existing lint-ing shell scripts have the ability
to only run on the files which have actually changed.
The new port lint-ing script doesn't have this functionality
unfortunately. This forces us to lint ALL the ports on every
single change to any other file in the system if you have
the pre-commit hook setup for your git clone locally.
Instead we can use pre-commit's feature to only run a hook
if certain files have changed to reduce the situations in
which we would run the Meta/lint-ports.py script.
Clang's default constexpr-steps limit is 1048576, which is not enough
for LibGfx's generation of the unicode bidirectional class lookup table
while GCC doesn't have any limit at all, so this patch increases the
limit to an arbitrarily larger value.
As many macros as possible are moved to Macros.h, while the
macros to create a test case are moved to TestCase.h. TestCase is now
the only user-facing header for creating a test case. TestSuite and its
helpers have moved into a .cpp file. Instead of requiring a TEST_MAIN
macro to be instantiated into the test file, a TestMain.cpp file is
provided instead that will be linked against each test. This has the
side effect that, if we wanted to have test cases split across multiple
files, it's as simple as adding them all to the same executable.
The test main should be portable to kernel mode as well, so if
there's a set of tests that should be run in self-test mode in kernel
space, we can accomodate that.
A new serenity_test CMake function streamlines adding a new test with
arguments for the test source file, subdirectory under /usr/Tests to
install the test application and an optional list of libraries to link
against the test application. To accomodate future test where the
provided TestMain.cpp is not suitable (e.g. test-js), a CUSTOM_MAIN
parameter can be passed to the function to not link against the
boilerplate main function.
Using rsync is significantly faster when we're using an existing
image because we only have to copy files which are new or have
been updated. This cuts down the image creation time to about
a second or two assuming an old image exists.
This adds support for re-using and re-sizing existing disk images.
Disk images are checked with e2fsck prior to re-use and a new disk
image is automatically created when that check fails.
This adds support for detecting incorrect version numbers and links
in the ports list.
Also, unlike before it doesn't parse the package.sh script but executes
it instead which allows us to detect syntax errors.
SPDX License Identifiers are a more compact / standardized
way of representing file license information.
See: https://spdx.dev/resources/use/#identifiers
This was done with the `ambr` search and replace tool.
ambr --no-parent-ignore --key-from-file --rep-from-file key.txt rep.txt *
This patch adds a basic Zsh completion script for the commands and
targets provided by Meta/serenity.sh. There's some room for improvement
here, e.g. we could provide completion for available CMake targets -
currently completion stops after serenity.sh <command> <target>.
You can enable it by adding this to your .zshrc before completions are
loaded:
fpath=($SERENITY_SOURCE_DIR/Meta/ShellCompletions/zsh $fpath)
Until we get the goodness that C++ modules are supposed to be, let's try
to shave off some parse time using precompiled headers.
This commit only adds some very common AK headers, only to binaries,
libraries and the kernel (tests are not covered due to incompatibility
with AK/TestSuite.h).
This option is on by default, but can be disabled by passing
`-DPRECOMPILE_COMMON_HEADERS=OFF` to cmake, which will disable all
header precompilations.
This makes the build about 30 seconds faster on my machine (about 7%).
Move LibCompress unit tests to LibCompress/Tests directory and register
them with CMake's add_test. This allows us to run these tests with
ninja test instead of running a separate executable.
Also split the existing tests in 3 test files that better follow the
source code structure (inspired by AK tests).
This was brought up as something that would be useful by
`RealKC` on the discord, and I happened to agree that it
would be useful. Especially given the abundance of Toolchain
changes recently.
The only icons we are currently warning about are designed
and rendered as small icons intentionally, as their only use
is in desktop applets, and thus are exempt to this rule.
This reduces build spam back down to a minimum.
I should have just done this in the first place, back in #4729
This is a very basic implementation that only requests 4096 bytes
of entropy from the host once, but its still high quality entropy
so it should be a good fix for #4490 (boot-time entropy starvation)
for virtualized environments.
Co-authored-by: Sahan <sahan.h.fernando@gmail.com>
This commit includes a lot of small changes and additions needed to
finalize the base implementation of VirtIOQueues and VirtDevices:
* The device specific driver implementation now has to handle setting
up the queues it needs before letting the base device class know it
finised initialization
* Supplying buffers to VirtQueues is now done via ScatterGatherLists
instead of arbitary buffer pointers - this ensures the pointers are
physical and allows us to follow the specification in regards to the
requirement that individual descriptors must point to physically
contiguous buffers. This can be further improved in the future by
implementating support for the Indirect-Descriptors feature (as
defined by the specification) to reduce descriptor usage for very
fragmented buffers.
* When supplying buffers to a VirtQueue the driver must supply a
(temporarily-)unique token (usually the supplied buffer's virtual
address) to ensure the driver can discern which buffer has finished
processing by the device in the case in which the device does not
offer the F_IN_ORDER feature.
* Device drivers now handle queue updates (supplied buffers being
returned from the device) by implementing a single pure virtual
method instead of setting a seperate callback for each queue
* Two new VirtQueue methods were added to allow the device driver
to either discard or get used/returned buffers from the device by
cleanly removing them off the descriptor chain (This also allows
the VirtQueue implementation to reuse those freed descriptors)
This also includes the necessary changes to the VirtIOConsole
implementation to match these interface changes.
Co-authored-by: Sahan <sahan.h.fernando@gmail.com>
This patch allocates a physical page for each of the virtqueues and
memcpys to it when receiving a buffer to get a physical, aligned
contiguous buffer as required by the virtio specification.
Co-authored-by: Sahan <sahan.h.fernando@gmail.com>
Based on pull #3236 by tomuta, this adds helper methods for generic
device initialization, and partily-broken virtqueue helper methods
Co-authored-by: Tom <tomut@yahoo.com>
Co-authored-by: Sahan <sahan.h.fernando@gmail.com>
This change fixes two bugs:
1) If you run `serenity.sh` outside of your serenity git clone, the
`get_top_dir()` function won't be able to auto detect the serenity
root dir and we'll error out. By allowing the script to use an
existing $SERENITY_ROOT vlaue if it exists, we can solve
this problem.
2) If tried to run a command which ends up calling `build_toolchain()`
and you were outside of the root of the project, it should just
fail. Fix this by utilizing `$SERENITY_ROOT`.
I have this symlinked into ~/bin, when looking at the help/usage
it would previously print the fully qualified path to the script
making the help very difficult to read.
install-ports copys the necessary files from Ports/ to /usr/Ports. Also
refactor the compiler and destiation variables from .port_include.sh
into .hosted_defs.sh. .hosted_defs.sh does not exists when ports are
built in serenity
The end goal of this commit is to allow to boot on bare metal with no
PS/2 device connected to the system. It turned out that the original
code relied on the existence of the PS/2 keyboard, so VirtualConsole
called it even though ACPI indicated the there's no i8042 controller on
my real machine because I didn't plug any PS/2 device.
The code is much more flexible, so adding HID support for other type of
hardware (e.g. USB HID) could be much simpler.
Briefly describing the change, we have a new singleton called
HIDManagement, which is responsible to initialize the i8042 controller
if exists, and to enumerate its devices. I also abstracted a bit
things, so now every Human interface device is represented with the
HIDDevice class. Then, there are 2 types of it - the MouseDevice and
KeyboardDevice classes; both are responsible to handle the interface in
the DevFS.
PS2KeyboardDevice, PS2MouseDevice and VMWareMouseDevice classes are
responsible for handling the hardware-specific interface they are
assigned to. Therefore, they are inheriting from the IRQHandler class.
Almost a year after first working on this, it's finally done: an
implementation of Promises for LibJS! :^)
The core functionality is working and closely following the spec [1].
I mostly took the pseudo code and transformed it into C++ - if you read
and understand it, you will know how the spec implements Promises; and
if you read the spec first, the code will look very familiar.
Implemented functions are:
- Promise() constructor
- Promise.prototype.then()
- Promise.prototype.catch()
- Promise.prototype.finally()
- Promise.resolve()
- Promise.reject()
For the tests I added a new function to test-js's global object,
runQueuedPromiseJobs(), which calls vm.run_queued_promise_jobs().
By design, queued jobs normally only run after the script was fully
executed, making it improssible to test handlers in individual test()
calls by default [2].
Subsequent commits include integrations into LibWeb and js(1) -
pretty-printing, running queued promise jobs when necessary.
This has an unusual amount of dbgln() statements, all hidden behind the
PROMISE_DEBUG flag - I'm leaving them in for now as they've been very
useful while debugging this, things can get quite complex with so many
asynchronously executed functions.
I've not extensively explored use of these APIs for promise-based
functionality in LibWeb (fetch(), Notification.requestPermission()
etc.), but we'll get there in due time.
[1]: https://tc39.es/ecma262/#sec-promise-objects
[2]: https://tc39.es/ecma262/#sec-jobs-and-job-queues
This utility traces the route packets take to a user specified host.
QEMU user networking (the type of networking we currently have setup)
does not support sending "real" raw IPv4 packets, and as such we cant
specify a custom TTL value. As a result, this utility will only work
on real hardware, qemu setup with tun networking (requires root) and
other hypervisors that support bridged adapters (VirtualBox/VMWare).
Running the tests will be moved to a separate test command which can
then leverage the availability of different targets and run either unit
tests on the host or the image in QEMU in self-test mode. :^)
- Only call ensure_toolchain for non-lagom targets
- Use host addr2line, we can't expect the i686 toolchain's addr2line to
support the host's binary executable format
- Don't export SERENITY_ARCH and TOOLCHAIN_DIR, don't need them anymore
