Commit Graph

631 Commits

Author SHA1 Message Date
Andrew Kaster
e88761b2b9 Meta+LibUnicode: Move unicode_data helper to Meta/CMake
Moving this helper CMake file to the centralized Meta/CMake folder helps
to get a better grasp on what extra files are required for the build,
and what files are generated.

While we're at it, don't use add_compile_definitions for
ENABLE_UNICODE_DATA, which only needs to be seen by LibUnicode sources.
2021-08-28 08:44:17 +01:00
Andrew Kaster
47471c0ec2 Meta: Prefer SerenityOS_SOURCE_DIR to CMAKE_SOURCE_DIR
By using SerenityOS_SOURCE_DIR we can make custom targets and commands
agnostic to the actual location of the root CMakeLists directory.
All we care about is the root of the SerenityOS project.
2021-08-28 08:44:17 +01:00
Andrew Kaster
20e904d87c Meta: Move code generator helpers to their own CMake helper file
compile_gml, compile_ipc, and generate_state_machine all use host
tools to generate sources for the target build. As part of trying to
organize host tools into a common area, let's move these helper rules to
a common file that we can add other host tools to later. And, keep the
host tool helpers separate from the CMake target helpers for apps and
libraries.
2021-08-28 08:44:17 +01:00
Andrew Kaster
a83847f8bf Meta: Move components CMake helpers to their own file
It's hard to follow how all the functions in the utils.cmake helper file
flow together, so let's move the pieces that are related to each other
into specialized helpers. First up: all the ConfigureComponents related
properties and functions.
2021-08-28 08:44:17 +01:00
kleines Filmröllchen
2dc614127e Meta: Add System32 to the PATH so that reg.exe is always found
On my machine (c), /mnt/c/Windows/System32 is not on the PATH by
default. This causes reg.exe to fail, which is responsible for detecting
the presence of QEMU. By putting this path on the PATH on WSL, it will
always work regardless of the specific PATH configuration, and QEMU is
always detected.
2021-08-27 21:24:25 +01:00
Timothy Flynn
9663525542 CI: Ensure unzip is installed on the builders
The CLDR database comes in a .zip file.
2021-08-26 23:40:23 +02:00
Timothy Flynn
3127454642 CI: Skip Unicode caches for Fuzzer build
There is a bit of a race here between the Fuzzer and non-Fuzzer Lagom
builds. If the Unicode caches are empty, and the Fuzzer build completes
first, then the UCD and CLDR directories will be empty or won't exist.
Skip handling the Unicode caches for this build.
2021-08-26 22:04:09 +01:00
Timothy Flynn
e883792fd4 CI: Cache downloaded Unicode CLDR database 2021-08-26 22:04:09 +01:00
Andreas Kling
c915174563 Userland: Remove IRC Client
The IRC Client application made some sense while our main communication
hub was an IRC channel. Now that we've moved on, IRC is just a random
protocol with no particular relevance to this project.

This also has the benefit of removing one major client of the single-
process Web::InProcessWebView class.
2021-08-24 16:37:28 +02:00
Itamar
6f4324e64e Meta: Add 'copy-src' command to serenity.sh
This command copies the project's source tree to
/home/anon/Source/serenity in the built disk image.

This will be useful for working on serenity inside serenity :^)
2021-08-22 10:28:13 +02:00
Timothy Flynn
ddfd4cced7 CI: Set on-target test timeout to 60 minutes
The on-target pipelines have a timeout of 6 hours to allow time for a
clean toolchain + Serenity build. Tests should time out much sooner than
that though.
2021-08-20 10:34:52 +02:00
Timothy Flynn
25ae1f7f72 CI: Include a timestamp in the Azure ccache key
Caches on Azure are immutable - so if a cache changes, but its key does
not, then the cache is not updated. Include a timestamp in the ccache
key so that we always push an updated cache from the master branch. Then
use a subkey without the timestamp to pull the cache.

We use a similar trick on GitHub Actions.
2021-08-20 10:34:52 +02:00
Brian Gianforcaro
ca9619c750 Meta: Explicitly configure source directory in debug-kernel.sh
Previously debug-kernel.sh wouldn't detect source if you launched the
script from the wrong path. By explicitly setting the name, source will
be loaded by gdb in all situations.
2021-08-19 11:42:20 +02:00
networkException
446bd1e0de README+Meta: Update the screenshot :^) 2021-08-18 14:27:37 +02:00
Martin Åberg
0ed2a03450 Meta: Assume and find GNU du
We want to use use the 'du' option '--apparent-size' which is a
GNU coreutils extension. GNU coreutils is a build dependency so
we know it is available. With this commit we first try to pick up
du as 'gdu', and if that fails, try 'du' instead.
2021-08-18 10:59:50 +02:00
Martin Åberg
0bf867bb9a Meta: Create memory disk before mounting it on FreeBSD
There was previously a case where the build-image-qemu.sh script
decided to mount an existing disk image, but without creating the
memory disk device and recording its /dev file name.

After this commit, We create the memory disk device just before
it is used to mount the disk image.
2021-08-18 10:25:00 +02:00
Andrew Kaster
9bdb44c5d2 CI: Ensure relevant patch files are part of Toolchain cache hash
We were over-hashing for the GNU build on GitHub Actions by including
the LLVM patch as well. The GNU Toolchain doesn't care about our LLVM
patches.

For Azure, fix the inversion of the condition for which jobs check which
Build*.sh script, and add the Toolchain patch files to the cache
hash calculation.
2021-08-18 08:26:38 +02:00
Timothy Flynn
a8f9f160f9 CI: Print on-target debug logs when the tests fail 2021-08-18 08:47:23 +04:30
Timothy Flynn
e3d4778a63 CI: Build Lagom with fuzzing on Azure 2021-08-17 21:06:15 +01:00
Timothy Flynn
3ce6f26924 CI: Build and test SerenityOS with the Clang toolchain on Azure 2021-08-17 13:31:43 +01:00
Timothy Flynn
5eba8c948d CI: Use Azure caching for Lagom build
Adds a compiler cache and a cache for the UCD files.
2021-08-17 13:31:43 +01:00
Timothy Flynn
831f90c9d4 Meta: Semi-automatically detect ccache for Lagom builds
The top-level CMakeLists.txt already automatically detects ccache, but
CI will invoke CMake with Lagom's CMakeLists.txt. Add an option to Lagom
to do the same detection.
2021-08-17 13:31:43 +01:00
Timothy Flynn
27804a6e67 CI: Build and test Lagom (non-fuzzer) on Azure 2021-08-16 18:04:33 +01:00
Brian Gianforcaro
72f348a6cf Meta: Disable printing frame args when debugging the kernel with gdb
The function arguments almost always optimized away, so you never get
much value out of showing these in the default back trace view, it just
adds a bunch of extra stuff that you need to visual wade through.
So lets disable showing them.
2021-08-14 21:18:05 +02:00
Itamar
e57fdb63f8 Tests: Add regression tests for the LibCpp preprocessor
Similarly to the LibCpp parser regression tests, these tests run the
preprocessor on the .cpp test files under
Userland/LibCpp/Tests/preprocessor, and compare the output with existing
.txt ground truth files.
2021-08-14 12:40:55 +02:00
Itamar
a38c330c68 LibCpp: Move parser tests to Userland/Libraries/LibCpp/Tests/parser 2021-08-14 12:40:55 +02:00
sin-ack
1d2bde2403 Meta: Add new-project.sh :^)
This script will instantiate a HackStudio template into a project on the
host. It currently supports all templates used by HackStudio.

To avoid having to maintain compatibility between other shells and the
Serenity shell in the postcreate scripts, we build the Serenity shell
with Lagom and use that to run the script.
2021-08-13 21:12:17 +04:30
Jean-Baptiste Boric
f641cc6470 Meta: Tune default QEMU disk size
Having lots of small files in Base/ may require more inodes in the
ext2 filesystem than the format utility sets aside by default. Let's
make a more educated guess since we have a rough idea of how many
inodes we need by counting files and directories.
2021-08-13 08:04:59 +02:00
Ali Mohammad Pur
f0e7e5bbe8 Meta: Un-escape escaped strings when generating Wasm tests 2021-08-12 21:03:53 +02:00
Ali Mohammad Pur
799471d16f Meta: Don't roundtrip floats for i64/i32 hex literals in wasm tests 2021-08-12 21:03:53 +02:00
sin-ack
a871a2040b Meta: Properly quote some CMake variables in serenity_component
This probably isn't all of them, because I'm no CMake expert. :^)
It does however allow "/bin/false" to build now.
2021-08-12 07:48:50 +02:00
Tom
e3e52a4e61 Meta: Add SERENITY_CPUS environment variable to control number of CPUs 2021-08-09 21:02:23 +02:00
Daniel Bertalan
13e3df41de Meta: Add Clang support to the CMake build scripts 2021-08-08 10:55:36 +02:00
Daniel Bertalan
15e217ea68 Toolchain: Add LLVM patch and script for building it
This contains all the bits and pieces necessary to build a Clang binary
that will correctly compile SerenityOS.

I had some trouble with getting LLVM building with a single command, so
for now, I decided to build each LLVM component in a separate command
invocation. In the future, we can also make the main llvm build step
architecture-independent, but that would come with extra work to make
library and include paths work.

The binutils build invocation and related boilerplate is duplicated
because we only use `objdump` from GNU binutils in the Clang toolchain,
so most features can be disabled.
2021-08-08 10:55:36 +02:00
Gunnar Beutner
2fec891d8e Meta: Allow attaching GDB to the QEMU instance on the WSL2 host 2021-08-08 01:46:41 +02:00
Gunnar Beutner
45f2e9f3cf Meta: Don't use x86_64 QEMU for the i686 kernel
This seemed like a good idea at the time to avoid an unnecessary
dependency on qemu-system-i386. However this makes debugging the
kernel with GDB more difficult because GDB assumes that the QEMU
architectures matches the kernel architecture.
2021-08-08 01:46:41 +02:00
Andrew Kaster
891bc81f30 Meta: Remove syslog.h from clang-format exclusion list
Before Libraries was moved to Userland/Libraries syslog.h had a bunch
of manually aligned defines and array initializations.

Andreas seems to have formatted the file with clang-format as part of
that file move. Since syslog.h is now properly formatted, we don't
need to exclude it from the linter list.
2021-08-07 15:36:51 +02:00
sin-ack
0d468f2282 Kernel: Implement a ISO 9660 filesystem reader :^)
This commit implements the ISO 9660 filesystem as specified in ECMA 119.
Currently, it only supports the base specification and Joliet or Rock
Ridge support is not present. The filesystem will normalize all
filenames to be lowercase (same as Linux).

The filesystem can be mounted directly from a file. Loop devices are
currently not supported by SerenityOS.

Special thanks to Lubrsi for testing on real hardware and providing
profiling help.

Co-Authored-By: Luke <luke.wilde@live.co.uk>
2021-08-07 15:21:58 +02:00
Liav A
02c72de090 Meta: Remove legacy hardware components from Q35 machine
As this is a test machine I use personally to test "modern" hardware
setups, it feels quite comfortable to not care too much about VGA with
this type of machine.
Also, we don't actively use the IDE controller on this machine type, so
let's just remove it :^)
2021-08-07 11:49:07 +02:00
sin-ack
bb609cee7f Meta: Add run-local.sh
This allows one to set their desired parameters for run.sh without the
need to set them in every terminal session or add it to the user account
shell files. If a run-local.sh file exists at the repository root and is
executable, it will be sourced. The file can contain any variables that
are expected to be set in run.sh.
2021-08-07 11:47:07 +02:00
Mandar Kulkarni
566702ffa2 Meta: Fix wrong path for AK Tests in check-ak-test-files.sh 2021-08-05 19:58:55 +02:00
Gunnar Beutner
d9d2287a11 Meta: Detect nested KVM support for WSL2 and use that when available
This allows running QEMU inside WSL2 for hosts which have nested KVM
and WSLg support (e.g. Windows 11).

Running QEMU inside the WSL2 VM is slightly slower than running QEMU
on Windows, probably because of how WSLg handles screen updates.
2021-08-04 21:15:41 +02:00
Gunnar Beutner
723c89af67 Meta: Automatically enable WHPX when possible 2021-08-04 21:15:41 +02:00
Gunnar Beutner
5445155dba Meta: Auto-detect where QEMU is installed on Windows 2021-08-04 21:15:41 +02:00
Liav A
cca1498e09 Meta: Make the Q35 machine more realistic
Although it is nice to test the system without too many devices, in
reality bare metal hardware is far more complex than the default skeleon
that QEMU provides. As a preparation of supporting more devices, we
need to ensure we are capable of at least booting on complex hardware
setups without easily-observable problems. Later on, this can be the
foundations of testing new drivers :^)
2021-08-03 21:08:01 +02:00
Gunnar Beutner
31655c9486 Meta: Prefer to use the 64-bit QEMU binary if that's available
Some users might not have qemu-system-i386 installed.
2021-08-03 13:59:57 +01:00
Gunnar Beutner
b082488e94 Meta: Prefer to use -machine pcspk-audiodev for QEMU >=5.1
This gets rid of the following warning message from QEMU on startup:

qemu-system-i386: warning: '-soundhw pcspk' is deprecated, please set a
backend using '-machine pcspk-audiodev=<name>' instead

Fixes #4093.
2021-08-03 13:59:57 +01:00
Gunnar Beutner
74af43ed97 Meta: Improve WSL detection for the run.sh script
For users who use a custom kernel with WSL our previous method of
detecting WSL doesn't work. This new check instead detects WSL by
checking if the wslpath utility is available.
2021-08-02 11:47:29 +01:00
Andrew Kaster
40b0767d88 Meta: Add BUILD_SHARED_LIBS option for Lagom builds
This standard CMake option controls whether add_library() calls will
use STATIC or SHARED by default. The flag is set to on by default
since that's what we want for normal CI jobs and local builds and the
test262 runner, but disabled for oss-fuzz builds.

This should finally fix the oss-fuzz build after it was broken in #9017

oss-fuzz un-breakage was verified by running the following commands in
the oss-fuzz repo:

python infra/helper.py build_image serenity
python infra/helper.py build_fuzzers --sanitizer address --engine afl \
    --architecture x86_64 serenity /path/to/local/checkout/Meta/Lagom
python infra/helper.py check_build --sanitizer address --engine afl \
    --architecture x86_64 serenity
2021-08-02 09:05:28 +02:00
x-yl
8e8b1c8333 Meta: Explicitly specify the disk format in the QEMU options
Otherwise we're getting this warning:

WARNING: Image format was not specified for '_disk_image' and probing
         guessed raw. Automatically detecting the format is dangerous
         for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
2021-08-02 00:35:22 +02:00