Commit Graph

196 Commits

Author SHA1 Message Date
Daniel Bertalan
ee9125fa9a Meta: Use correct variable for checking if the mold linker is used
This variable was originally called USE_MOLD_LINKER, but it was changed
to ENABLE_MOLD_LINKER during review to be consistent with other
configuration options. I branched off the commits that added RELR
support before this change, and I failed to update the variable name
there.
2022-02-19 14:21:52 +02:00
czapek1337
64ff8af074 Meta: Add support for the Limine bootloader 2022-02-14 11:52:07 +01:00
Daniel Bertalan
ba5bbde7ee Meta: Enable RELR relocations
Also add a check to serenity.sh to ensure that the toolchain is new
enough for this feature to work.
2022-02-11 18:07:53 +01:00
Brian Gianforcaro
ee61739e0a Meta: Add install-native-partition CMake target installing to a real FS
While playing around with getting serenity to run on my main desktop
machine I wanted a way of easily updating my physical serenity
partition.

To use it you just need to:
- Create and format your local partition to ext4
- Set `SERENITY_TARGET_INSTALL_PARTITION` to the partition /dev path.
- Run the `install-native-partition` build target.

Example:

    $ export SERENITY_TARGET_INSTALL_PARTITION=/dev/nvme1n1p3
    $ cd serenity/Build/x86_64
    $ ninja install-native-partition
2022-02-04 12:44:50 +01:00
Timothy Flynn
6f8e89a905 Meta: Download USB and PCI ID data with fallible download function 2022-01-26 00:22:53 +00:00
Daniel Bertalan
4055c393fc Toolchain: Add support for building the userland with the mold linker
This commit adds support for building the SerenityOS userland with the
new [mold linker].

This is not enabled by default yet; to link using mold, run the
`Toolchain/BuildMold.sh` script to build the latest release of mold, and
set the `ENABLE_MOLD_LINKER` CMake variable to ON. This option relies on
toolchain support that has been added just recently, so you might need
to rebuild your toolchain for mold to work.

[mold linker]: https://github.com/rui314/mold
2022-01-24 10:41:47 +00:00
Itamar
f464ae84ce CMake: Add HACKSTUDIO_BUILD option for building from Hack Studio
If this option is set, we will not build all components.
Instead, we include an external CMake file passed in via a variable
named HACKSTUDIO_BUILD_CMAKE_FILE.

This will be used to build serenity components from Hack Studio.
2022-01-12 14:55:19 +01:00
Daniel Bertalan
4a428ba252 Meta: Do not allow undefined symbols in executables and shared objects
The `--allow-shlib-undefined` option is a bit of a misnomer. It actually
controls whether we should be allowed to have undefined references after
symbols from all dependencies have been resolved, so it applies both to
shared libraries and executables.

LLD defaults to allowing undefined references in shared libraries, but
not in executables. Previously, we had to disable this check for
executables too, as it caused a build failure due to the
LibC-LibPthread-libc++ and the LibCore-LibCrypto circular dependencies.

Now that those have been resolved, we can enable this warning, in the
hopes that it will prevent us from introducing circular libraries and
missing dependencies that might cause unexpected breakage.
2022-01-09 20:40:22 +01:00
Andrew Kaster
1a17425fb5 Meta: Remove clang warnings with no violations
At some point we probably needed these, but no code complains about them
with the current state of the codebase.
2022-01-04 07:51:31 +00:00
Andrew Kaster
5cbc734081 SystemMonitor: Remove unused functions and enable warning globally
These two helpers were the only unused functions clang could detect in
the entire codebase. Now that's commitment to no dead code :^)
2022-01-04 07:51:31 +00:00
Andrew Kaster
f8ea9cd64e Userland: Resolve unused-but-set-varaible warnings
These are almost always bugs, so enable globally.

Remove unused counter variables in SystemMonitor and disk_benchmark.
2022-01-04 07:51:31 +00:00
Andrew Kaster
d809637023 Userland: Resolve tautological-constant-out-of-range-compare warnings
Stop comparing platform-specific sized integer types to max() values of
other interger types. Enable the warning everywhere.
2022-01-04 07:51:31 +00:00
Andrew Kaster
a103a85ae6 Userland: Locally suppress -Wc99-designator and re-enable globally
There's only two places where we're using the C99 feature of array
designated initalizers. This feature seemingly wasn't included with
C++20 designated initalizers for classes and structs. The only two
places we were using this feature are suitably old and isolated that
it makes sense to just suppress the warning at the usage sites while
discouraging future array designated intializers in new code.
2022-01-04 07:51:31 +00:00
Andrew Kaster
c8cfd6661a Userland: Resolve -Woverloaded-virtual in LibGUI and SoundPlayer
Enable the warning project-wide. It catches when a non-virtual method
creates an overload set with a virtual method. This might cause
surprising overload resolution depending on how the method is invoked.
2022-01-04 07:51:31 +00:00
Andrew Kaster
ed3cb88898 Meta: Enable -Wexpansion-to-defined warning for Clang
The Clang implementation of this warning protects against some undefined
pre-processor behavior while ignoring function-like macros. The gcc
implementation also warns on function-like macros, and is therefore
noisy.
2022-01-04 07:51:31 +00:00
Andrew Kaster
ba2ca54660 Meta: Re-enable toolchain version checks
These were removed in the Superbuild conversion. Re-add the checks that
make sure that if there's a toolchain update, developers re-build their
toolchain.
2022-01-04 07:51:31 +00:00
tuftedocelot
4633f89217 Meta: Use portable gzip option for extracting PCI and USB ID files
gzip -c is supported in both Linux and BSD flavors of gzip. The -o flag
was introduced in a previous commit which is present in OpenBSD, but not
other flavors of Linux. -c will write to stdout which is redirected to
the target files. As a side benefit, we no longer need to copy files
anywhere
2021-12-29 15:03:47 -08:00
tuftedocelot
5810467c97 Build: Remove gzip -k usage in PCI/USB ID files and crypt for OpenBSD
OpenBSD gzip does not have the -k flag to keep the original after
extraction. Work around this by copying the original gzip to the dest
and then extracting. A bit of a hack, but only needs to be done for the
first-time or rebuilds

OpenBSD provides crypt in libc, not libcrypt. Adjust if/else to check
for either and proceed accordingly

Remove outdated OpenBSD checks when building the toolchain
2021-12-29 03:46:56 -08:00
Daniel Bertalan
52e4753722 Meta: Move compiler-specific warnings into the appropriate if clause 2021-12-11 13:14:15 -08:00
Daniel Bertalan
4a81b33c07 Everywhere: Fix -Winconsistent-missing-override warnings from Clang
This option is already enabled when building Lagom, so let's enable it
for the main build too. We will no longer be surprised by Lagom Clang
CI builds failing while everything compiles locally.

Furthermore, the stronger `-Wsuggest-override` warning is enabled in
this commit, which enforces the use of the `override` keyword in all
classes, not just those which already have some methods marked as
`override`. This works with both GCC and Clang.
2021-12-11 13:14:15 -08:00
Ben Wiederhake
8de696bdd0 HeaderCheck: It checks some of your headers 2021-10-20 09:20:18 +01:00
Daniel Bertalan
06fc64be13 Toolchain+Meta: Update LLVM version to 13.0.0
This commit updates the Clang toolchain's version to 13.0.0, which comes
with better C++20 support and improved handling of new features by
clang-format. Due to the newly enabled `-Bsymbolic-functions` flag, our
Clang binaries will only be 2-4% slower than if we dynamically linked
them, but we save hundreds of megabytes of disk space.

The `BuildClang.sh` script has been reworked to build the entire
toolchain in just three steps: one for the compiler, one for GNU
binutils, and one for the runtime libraries. This reduces the complexity
of the build script, and will allow us to modify the CI configuration to
only rebuild the libraries when our libc headers change.

Most of the compile flags have been moved out to a separate CMake cache
file, similarly to how the Android and Fuchsia toolchains are
implemented within the LLVM repo. This provides a nicer interface than
the heaps of command-line arguments.

We no longer build separate toolchains for each architecture, as the
same Clang binary can compile code for multiple targets.

The horrible mess that `SERENITY_CLANG_ARCH` was, has been removed in
this commit. Clang happily accepts an `i686-pc-serenity` target triple,
which matches what our GCC toolchain accepts.
2021-10-17 17:09:58 +01:00
Ben Wiederhake
0321034c87 Meta: Re-enable warnings for deprecated copies
This used to supress two true-positives, and zero false-positives.

That's good enough in my book to warrant re-activation.
2021-10-10 21:03:27 +01:00
Andrew Kaster
b5c98ede08 Meta: Switch to a SuperBuild that splits host and target builds
Replace the old logic where we would start with a host build, and swap
all the CMake compiler and target variables underneath it to trick
CMake into building for Serenity after we configured and built the Lagom
code generators.

The SuperBuild creates two ExternalProjects, one for Lagom and one for
Serenity. The Serenity project depends on the install stage for the
Lagom build. The SuperBuild also generates a CMakeToolchain file for the
Serenity build to use that replaces the old toolchain file that was only
used for Ports.

To ensure that code generators are rebuilt when core libraries such as
AK and LibCore are modified, developers will need to direct their manual
`ninja` invocations to the SuperBuild's binary directory instead of the
Serenity binary directory.

This commit includes warning coalescing and option style cleanup for the
affected CMakeLists in the Kernel, top level, and runtime support
libraries. A large part of the cleanup is replacing USE_CLANG_TOOLCHAIN
with the proper CMAKE_CXX_COMPILER_ID variable, which will no longer be
confused by a host clang compiler.
2021-09-15 19:04:52 +04:30
Andrew Kaster
904a268872 Meta: Move all options to targetname_options.cmake files
This common strategy of having a serenity_option() macro defined in
either the Lagom or top level CMakeLists.txt allows us to do two things:

First, we can more clearly see which options are Serenity-specific,
Lagom-specific, or common between the target and host builds.

Second, it enables the upcoming SuperBuild changes to set() the options
in the SuperBuild's CMake cache and forward each target's options to the
corresponding ExternalProject.
2021-09-15 19:04:52 +04:30
Andrew Kaster
6e7cc40b18 Meta: Add Meta/CMake to the CMAKE_MODULE_PATH for Serenity and Lagom
This makes it so we don't need to specify the full path to all the
helper scripts we include() from different places in the codebase and
feels a lot cleaner.
2021-09-15 19:04:52 +04:30
Andrew Kaster
a269a32a17 Meta: Use Lagom:: namespaced names for code generators
This will be required when we switch to a SuperBuild that has Lagom as
a configure time dependency, but is a distinct enough change to be
separate.
2021-09-15 19:04:52 +04:30
Andreas Kling
619ee99c34 Build: Add -fno-delete-null-pointer-checks
This prevents GCC and Clang from deleting null pointer checks for
optimization purposes. I think we're strictly better off crashing
in those cases instead of the compiler hiding errors from us.
2021-09-09 21:25:10 +02:00
Andreas Kling
fac0bbe739 Build: Pass "-z separate-code" to linker
This tells the linker to not combine read-only data and executable code,
instead favoring multiple PT_LOAD headers with more precise permissions.

This greatly reduces the amount of executable pages in all our programs
and libraries.

/usr/lib/libjs.so before:

  Type           Offset   VirtAddr   PhysAddr   FileSiz  MemSiz   Flg Align
  LOAD           0x000000 0x00000000 0x00000000 0x2fc77c 0x2fc77c R E 0x1000
  LOAD           0x2fc900 0x002fd900 0x002fd900 0x0c708  0x0dd1c  RW  0x1000

/usr/lib/libjs.so after:

  Type           Offset   VirtAddr   PhysAddr   FileSiz  MemSiz   Flg Align
  LOAD           0x000000 0x00000000 0x00000000 0x80e60  0x80e60  R   0x1000
  LOAD           0x081000 0x00081000 0x00081000 0x25f6c9 0x25f6c9 R E 0x1000
  LOAD           0x2e1000 0x002e1000 0x002e1000 0x1c27c  0x1c27c  R   0x1000
  LOAD           0x2fd900 0x002fe900 0x002fe900 0x0c708  0x0dd1c  RW  0x1000

As you can see, we go from 0x2fc77c bytes of executable memory down to
0x25f6c9 (a ~20% reduction!) The memory that was previous executable is
now simply read-only instead. :^)
2021-08-31 16:46:16 +02:00
Andrew Kaster
60e27dea9c Meta: Require unzip and tar explicitly in CMakeLists
This should help stem the tide of people hopping in the build problems
channel on discord because they don't have unzip installed.
2021-08-31 12:39:46 +02:00
Brian Gianforcaro
619200774b CMake: Add custom target to build only the generated sources
This is needed so all headers and files exist on disk, so that
the sonar cloud analyzer can find them when executing the compilation
commands contained in compile_commands.json, without actually building.

Co-authored-by: Andrew Kaster <akaster@serenityos.org>
2021-08-30 16:44:16 +02:00
Nico Weber
57f95bd724 CMake: Don't build Userland and Tests in aarch64 builds for now 2021-08-28 21:51:30 +01:00
Nico Weber
bbad4758b2 CMake: Let Meta/serenity.sh run aarch64 make it past cmake
This adds just enough scaffolding to make cmake succeed.
The build falls over immediately.
2021-08-28 14:43:07 +01:00
Andrew Kaster
834be1a895 Meta: Remove write-only-on-difference script
It's now been replaced with "${CMAKE_PROGRAM}" -E copy_if_different in
all previous usage locations.
2021-08-28 08:44:17 +01:00
Andrew Kaster
63956b36d0 Everywhere: Move all host tools into the Lagom/Tools subdirectory
This allows us to remove all the add_subdirectory calls from the top
level CMakeLists.txt that referred to targets linking LagomCore.

Segregating the host tools and Serenity targets helps us get to a place
where the main Serenity build can simply use a CMake toolchain file
rather than swapping all the compiler/sysroot variables after building
host libraries and tools.
2021-08-28 08:44:17 +01:00
Andrew Kaster
47471c0ec2 Meta: Prefer SerenityOS_SOURCE_DIR to CMAKE_SOURCE_DIR
By using SerenityOS_SOURCE_DIR we can make custom targets and commands
agnostic to the actual location of the root CMakeLists directory.
All we care about is the root of the SerenityOS project.
2021-08-28 08:44:17 +01:00
Daniel Bertalan
bd6dc5ccec Meta+LibC: Don't allow text relocations in SerenityOS libraries
The `-z,text` linker flag causes the linker to reject shared libraries
and PIE executables that have textrels. Our code mostly did not use
these except in one place in LibC, which is changed in this commit.
This makes GNU ld match LLD's behavior, which has this option enabled by
default.

TEXTRELs pose a security risk, as performing these relocations require
executable pages to be written to by the dynamic linker. This can
significantly weaken W^X hardening mitigations.

Note that after this change, TEXTRELs can still be used in ports, as the
dynamic loader code is not changed. There are also uses of it in the
kernel, removing which are outside the scope of this PR. To allow those,
`-z,notext` is added.
2021-08-18 18:01:22 +02:00
Daniel Bertalan
49d795b985 Meta: Enable nonnull-compare warnings
Previously, this was disabled because GCC flagged seemingly correct and
well-defined code. This was however not the case because GCC implicitly
marked some pointers non-null, even if we wanted to handle them
ourselves, and deleted null checks on them. By re-introducing this
warning, we will know if the compiler tries to discard our code again.
2021-08-12 21:10:44 +02:00
Daniel Bertalan
13e3df41de Meta: Add Clang support to the CMake build scripts 2021-08-08 10:55:36 +02:00
Andreas Kling
2362ebf483 Build: Add -fsized-deallocation to global CXXFLAGS
This is enabled by default for GCC (>=C++14) but apparently not for
Clang, which has been causing CLion to get confused about kernel code.
2021-08-08 00:03:45 +02:00
Gunnar Beutner
b7ca269b4d Kernel: Use our toolchain's c++filt tool for the kernel map
The host's version of c++filt might not work on some operating systems,
e.g. macOS.
2021-07-29 10:38:31 +02:00
Gunnar Beutner
9c431ae7ef Toolchain: Update GCC to version 11.2.0 2021-07-28 21:28:31 +02:00
Timothy Flynn
98d8274040 Meta: Add LibUnicode (and its tests) to Lagom
This is primarily to allow using LibUnicode within LibJS and its REPL.

Note: this seems to be the first time that a Lagom dependency requires
generated source files. For this to work, some of Lagom's CMakeLists.txt
commands needed to be re-organized to include the CMake files that fetch
and parse UnicodeData.txt. The paths required to invoke the generator
also differ depending on what is currently building (SerenityOS vs.
Lagom as part of the Serenity build vs. a standalone Lagom build).
2021-07-26 17:03:55 +01:00
Timothy Flynn
4dda3edc9e LibUnicode: Introduce a Unicode library for interacting with UCD files
The Unicode standard publishes the Unicode Character Database (UCD) with
information about every code point, such as each code point's upper case
mapping. LibUnicode exists to download and parse UCD files at build time
and to provide accessors to that data.

As a start, LibUnicode includes upper- and lower-case code point
converters.
2021-07-26 17:03:55 +01:00
Patrick Meyer
83f88df757 Kernel: Add option to build with coverage instrumentation and KCOV
GCC and Clang allow us to inject a call to a function named
__sanitizer_cov_trace_pc on every edge. This function has to be defined
by us. By noting down the caller in that function we can trace the code
we have encountered during execution. Such information is used by
coverage guided fuzzers like AFL and LibFuzzer to determine if a new
input resulted in a new code path. This makes fuzzing much more
effective.

Additionally this adds a basic KCOV implementation. KCOV is an API that
allows user space to request the kernel to start collecting coverage
information for a given user space thread. Furthermore KCOV then exposes
the collected program counters to user space via a BlockDevice which can
be mmaped from user space.

This work is required to add effective support for fuzzing SerenityOS to
the Syzkaller syscall fuzzer. :^) :^)
2021-07-26 17:40:28 +02:00
Gunnar Beutner
e4f05a9046 Kernel: Make new kernel build process work on macOS
Use objcopy from the toolchain so that the changes introduced in
7236584 will succeed on macOS.

Fixes #8768.
2021-07-15 11:04:30 +02:00
Daniel Bertalan
7f2eb2f332 Meta: Remove obsolete -fconcepts CFLAG
The GCC documentation says that since it's officially a part of C++20,
this flag does nothing. Clang, however, does complain that it does not
recognize it, so it's better to just remove it.
2021-07-14 13:12:25 +02:00
Andrew Kaster
ca920ba082 Meta+Documentation: Remove unused -DDEBUG from build 2021-07-12 12:26:52 +02:00
Max Wipfli
2d71eaadcd Meta: Add the ConfigureComponents utility
This adds a utility program which is essentially a command generator for
CMake. It reads the 'components.ini' file generated by CMake in the
build directory, prompts the user to select a build type and optionally
customize it, generates and runs a CMake command as well as 'ninja
clean' and 'rm -rf Root', which are needed to properly remove system
components.

The program uses whiptail(1) for user interaction.
2021-07-06 21:54:51 +02:00
Gunnar Beutner
e1ff30a360 Toolchain+Userland: Enable TLS for x86_64
This is not technically a toolchain change, but it does require
rebuilding the toolchain for x86_64 (and just that).
2021-07-04 01:07:28 +02:00