Commit Graph

138 Commits

Author SHA1 Message Date
Andrew Kaster
4a5a1e8648 Userland: Port UBSAN implementation to userspace
Take Kernel/UBSanitizer.cpp and make a copy in LibSanitizer.

We can use LibSanitizer to hold other sanitizers as people implement
them :^).

To enable UBSAN for LibC, DynamicLoader, and other low level system
libraries, LibUBSanitizer is built as a serenity_libc, and has a static
version for LibCStatic to use. The approach is the same as that taken in

Note that this means now UBSAN is enabled for code generators, Lagom,
Kernel, and Userspace with -DENABLE_UNDEFINED_SANTIZER=ON. In userspace
however, UBSAN is not deadly (yet).

Co-authored-by: ForLoveOfCats <ForLoveOfCats@vivaldi.net>
2021-05-27 15:18:03 +02:00
Ali Mohammad Pur
0e4431af33 Meta: Run the Wasm spec tests in CI
Since LibWasm is still not capable of passing all of the spec tests,
ignore failing tests, only fail the build if some segfault/abort/etc
occurs.
2021-05-27 17:28:41 +04:30
Andrew Kaster
2ec302ea22 Meta/CI: Add ENABLE_ALL_DEBUG_FACILITIES CMake option
This option replaces the use of ENABLE_ALL_THE_DEBUG_MACROS in CI runs,
and enables all debug options that might be broken by developers
unintentionally that are only used in specific debugging situations.
2021-05-27 10:21:30 +02:00
Andrew Kaster
dda8afcb90 Kernel: Add ENABLE_EXTRA_KERNEL_DEBUG_SYMBOLS option to set Og and ggdb3
When debugging kernel code, it's necessary to set extra flags. Normal
advice is to set -ggdb3. Sometimes that still doesn't provide enough
debugging information for complex functions that still get optimized.
Compiling with -Og gives the best optimizations for debugging, but can
sometimes be broken by changes that are innocuous when the compiler gets
more of a chance to look at them. The new CMake option enables both
compile options for kernel code.
2021-05-27 10:21:30 +02:00
Ali Mohammad Pur
24b2a6c93a LibWasm+Meta: Implement instantiation/execution primitives in test-wasm
This also optionally generates a test suite from the WebAssembly
testsuite, which can be enabled via passing `INCLUDE_WASM_SPEC_TESTS`
to cmake, which will generate test-wasm-compatible tests and the
required fixtures.
The generated directories are excluded from git since there's no point
in committing them.
2021-05-21 00:15:23 +01:00
Ali Mohammad Pur
b3c13c3e8a LibWasm+Meta: Add test-wasm and optionally test the conformance tests
This only tests "can it be parsed", but the goal of this commit is to
provide a test framework that can be built upon :)
The conformance tests are downloaded, compiled* and installed only if
the INCLUDE_WASM_SPEC_TESTS cmake option is enabled.
(*) Since we do not yet have a wast parser, the compilation is delegated
to an external tool from binaryen, `wasm-as`, which is required for the
test suite download/install to succeed.
This *does* run the tests in CI, but it currently does not include the
spec conformance tests.
2021-05-21 00:15:23 +01:00
Andreas Kling
a15c7b7944 Build: Stop using precompiled headers (PCH)
This had very bad interactions with ccache, often leading to rebuilds
with 100% cache misses, etc. Ali says it wasn't that big of a speedup
in the end anyway, so let's not bother with it.

We can always bring it back in the future if it seems like a good idea.
2021-05-17 19:30:12 +02:00
Daniel Bertalan
22195d965f DevTools: Add StateMachineGenerator utility
This program turns a description of a state machine that takes its input
byte-by-byte into C++ code. The state machine is described in a custom
format as specified below:

```
// Comments are started by two slashes, and cause the rest of the line
// to be ignored

@name ExampleStateMachine // sets the name of the generated class
@namespace Test           // sets the namespace (optional)
@begin Begin              // sets the state the parser will start in

// The rest of the file contains one or more states and an optional
// @anywhere directive. Each of these is a curly bracket delimited set
// of state transitions. State transitions contain a selector, the
// literal "=>" and a (new_state, action) tuple. Examples:
//     0x0a => (Begin, PrintLine)
//     [0x00..0x1f] => (_, Warn)      // '_' means no change
//     [0x41..0x5a] => (BeginWord, _) // '_' means no action

// Rules common to all states. These take precedence over rules in the
// specific states.
@anywhere {
    0x0a         => (Begin, PrintLine)
    [0x00..0x1f] => (_, Warn)
}

Begin {
    [0x41..0x5a] => (Word, _)
    [0x61..0x7a] => (Word, _)
    // For missing values, the transition (_, _) is implied
}

Word {
    // The entry action is run when we transition to this state from a
    // *different* state. @anywhere can't have this
    @entry IncreaseWordCount
    0x09 => (Begin, _)
    0x20 => (Begin, _)

    // The exit action is run before we transition to any *other* state
    // from here. @anywhere can't have this
    @exit EndOfWord
}
```

The generated code consists of a single class which takes a
`Function<Action, u8>` as a parameter in its constructor. This gets
called whenever an action is to be done. This is because some input
might not produce an action, but others might produce up to 3 (exit,
state transition, entry). The actions allow us to build a more
advanced parser over the simple state machine.

The sole public method, `void advance(u8)`, handles the input
byte-by-byte, managing the state changes and requesting the appropriate
Action from the handler.

Internally, the state transitions are resolved via a lookup table. This
is a bit wasteful for more complex state machines, therefore the
generator is designed to be easily extendable with a switch-based
resolver; only the private `lookup_state_transition` method needs to be
re-implemented.

My goal for this tool is to use it for implementing a standard-compliant
ANSI escape sequence parser for LibVT, as described on
<https://vt100.net/emu/dec_ansi_parser>
2021-05-16 11:50:56 +02:00
Brian Gianforcaro
8693c925a0 CMake: Fix message levels for error conditions during configuration
Make messages which should be fatal, actually fail the build.

- FATAL is not a valid mode keyword. The full list is available in the
  docs: https://cmake.org/cmake/help/v3.19/command/message.html

- SEND_ERROR doesn't immediately stop processing, FATAL_ERROR does.
  We should immediately stop if the Toolchain is not present.

- The app icon size validation was just a WARNING that is easy to
  overlook. We should promote it to a FATAL_ERROR so that people will
  not overlook the issue when adding a new application. We can only make
  the small icon message FATAL_ERROR, as there is currently one
  violation of the medium app icon validation.
2021-05-13 18:52:48 +02:00
DolphinChips
422ef7904e Build: Add extlinux-image target to CMake 2021-05-12 09:25:03 +01:00
Brian Gianforcaro
fd0dbd1ebf Tests: Establish root Tests directory, move Userland/Tests there
With the goal of centralizing all tests in the system, this is a
first step to establish a Tests sub-tree. It will contain all of
the unit tests and test harnesses for the various components in the
system.
2021-05-06 17:54:28 +02:00
Gunnar Beutner
ce77caf479 LibC: Move crypt() and crypt_r() to the right header file
According to POSIX.1 these should be in <crypt.h>.
2021-05-01 12:40:12 +02:00
Carlos César Neves Enumo
d142ca4c85 CMake: Fix building with AppleClang compiler 2021-04-30 09:24:14 +02:00
Gunnar Beutner
8ae0794584 CMake: Fix building libraries on macOS
When building libraries on macOS they'd be missing the SONAME
attribute which causes the linker to embed relative paths into
other libraries and executables:

Dynamic section at offset 0x52794 contains 28 entries:
 Type     Name/Value
(NEEDED) Shared library: [libgcc_s.so]
(NEEDED) Shared library: [Userland/Libraries/LibCrypt/libcrypt.so]
(NEEDED) Shared library: [Userland/Libraries/LibCrypto/libcrypto.so]
(NEEDED) Shared library: [Userland/Libraries/LibC/libc.so]
(NEEDED) Shared library: [libsystem.so]
(NEEDED) Shared library: [libm.so]
(NEEDED) Shared library: [libc.so]

The dynamic linker then fails to load those libraries which makes
the system unbootable.
2021-04-30 09:14:43 +02:00
Gunnar Beutner
6288ae2c37 Kernel: Add a CMake flag to enable LTO for the kernel 2021-04-29 20:26:36 +02:00
Gunnar Beutner
8cd62b5780 Toolchain+Ports: Update GCC to version 11.1.0 2021-04-29 10:33:44 +02:00
Brian Gianforcaro
2ef93a3c07 Build: Use variables when concatenating Toolchain paths.
Make this stuff a bit easier to maintain by using the
root level variables to build up the Toolchain paths.

Also leave a note for future editors of BuildIt.sh to
give them warning about the other changes they'll need
to make.
2021-04-27 13:07:04 +02:00
Ali Mohammad Pur
18ddfc19c6 Meta: Disable the use of PCH by default
While this has a rather significant impact for me, it appears to have
very minimal build time improvements (or in some cases, regressions).
Also appears to cause some issues when building on macOS.
So disable it by default, but leave the option so people that get
something out of it (seems to mostly be a case of "is reading the
headers fast enough") can turn it on for their builds.
2021-04-22 22:23:14 +02:00
Ali Mohammad Pur
468ac11f29 Meta: Add an option to precompile some very common AK headers
Until we get the goodness that C++ modules are supposed to be, let's try
to shave off some parse time using precompiled headers.
This commit only adds some very common AK headers, only to binaries,
libraries and the kernel (tests are not covered due to incompatibility
with AK/TestSuite.h).
This option is on by default, but can be disabled by passing
`-DPRECOMPILE_COMMON_HEADERS=OFF` to cmake, which will disable all
header precompilations.
This makes the build about 30 seconds faster on my machine (about 7%).
2021-04-21 14:29:46 +02:00
Panagiotis Vasilopoulos
e45e0eeb47 Everywhere: Replace SERENITY_ROOT with SERENITY_SOURCE_DIR 2021-04-20 15:27:52 +02:00
Lenny Maiorani
df916c9738 CMake: Quiet warnings about literal suffix
Problem:
- Newer versions of clang (ToT) have a similar `-Wliteral-suffix`
  warning as GCC. A previous commit enabled it for all compilers. This
  needs to be silenced for the entire build, but it currently only is
  silenced for some directories.

Solution:
- Move the `-Wno-literal-suffix` option up in the CMakeLists.txt so
  that it gets applied everywhere.
2021-04-19 00:38:31 +02:00
Lenny Maiorani
97f4aa166a CMake: Remove redundancies and support clang ToT
Problem:
- There are redundant options being set for some directories.
- Clang ToT fails to compile the project.

Solution:
- Remove redundancies.
- Fix clang error list.
2021-04-18 22:09:25 +02:00
Nicholas-Baron
73dd293ec4 Everywhere: Add -Wdouble-promotion warning
This warning informs of float-to-double conversions. The best solution
seems to be to do math *either* in 32-bit *or* in 64-bit, and only to
cross over when absolutely necessary.
2021-04-16 19:01:54 +02:00
Nicholas-Baron
c4ede38542 Everything: Add -Wnon-virtual-dtor flag
This flag warns on classes which have `virtual` functions but do not
have a `virtual` destructor.

This patch adds both the flag and missing destructors. The access level
of the destructors was determined by a two rules of thumb:
1. A destructor should have a similar or lower access level to that of a
   constructor.
2. Having a `private` destructor implicitly deletes the default
   constructor, which is probably undesirable for "interface" types
   (classes with only virtual functions and no data).

In short, most of the added destructors are `protected`, unless the
compiler complained about access.
2021-04-15 20:57:13 +02:00
Nicholas-Baron
7b502d113b Everywhere: Add "free" warnings
The following warnings do not occur anywhere in the codebase and so
enabling them is effectivly free:
 * `-Wcast-align`
 * `-Wduplicated-cond`
 * `-Wformat=2`
 * `-Wlogical-op`
 * `-Wmisleading-indentation`
 * `-Wunused`

These are taken as a strict subset of the list in #5487.
2021-04-15 10:21:45 +02:00
Peter Elliott
938924f36d Meta: Add install-ports CMake target
install-ports copys the necessary files from Ports/ to /usr/Ports. Also
refactor the compiler and destiation variables from .port_include.sh
into .hosted_defs.sh. .hosted_defs.sh does not exists when ports are
built in serenity
2021-04-12 14:06:24 +02:00
Andreas Kling
d454926e0f cmake: Hotfix the broken build
This regressed in #6000 and started complaining about bad literal
suffixes, so here's a quick and dirty partial revert to make things
build again.
2021-03-28 21:32:28 +02:00
Michel Hermier
d927b69082 cmake: Tidy compiler options.
Prior to this patch there was some long line of unreadable compiler
options. Now the long lines are deduplicated and there is only one
option per line to ease reading/maintenance.
2021-03-28 20:40:12 +02:00
Michel Hermier
a208cc3169 cmake: Group compile options together. 2021-03-28 20:40:12 +02:00
Brendan Coles
215375f2a5 Build: Enable --noexecstack
Build ELF executables with a zero length `GNU_STACK`
program header flagged non-executable.

The stack is never executable on SerenityOS regardless
of whether the `GNU_STACK` header is specified.

Specifically defining this header is more explicit,
as absence of this header implies an executable stack
on other systems (Linux).
2021-03-19 09:16:53 +01:00
Linus Groh
3775507613 Build: Download and uncompress gzipped version of pci.ids
Partially addresses #5611.
2021-03-04 11:21:55 +01:00
Linus Groh
15ae22f7cc Build: Add ENABLE_PCI_IDS_DOWNLOAD CMake option
This allows disabling the download of the pci.ids database at build
time.

Addresses concerns raised in #5410.
2021-03-04 11:21:55 +01:00
Andrew Kaster
e787738c24 Meta: Build AK and LibRegex tests in Lagom and for Serenity
These tests were never built for the serenity target. Move their Lagom
build steps to the Lagom CMakeLists.txt, and add serenity build steps
for them. Also, fix the build errors when building them with the
serenity cross-compiler :^)
2021-02-28 18:19:37 +01:00
Brian Gianforcaro
31e1b08e15 AK: Add support for AK::StringView literals with operator""sv
A new operator, operator""sv was added as of C++17 to support
string_view literals. This allows string_views to be constructed
from string literals and with no runtime cost to find the string
length.

See: https://en.cppreference.com/w/cpp/string/basic_string_view/operator%22%22sv

This change implements that functionality in AK::StringView.
We do have to suppress some warnings about implementing reserved
operators as we are essentially implementing STL functions in AK
as we have no STL :).
2021-02-24 14:38:31 +01:00
Linus Groh
6ad3454bfb AK: Rename {DBGLN_NO => ENABLE}_COMPILETIME_FORMAT_CHECK
This is no longer limited to dbgln(). Also invert it to match all the
other ENABLE_FOO options.
2021-02-24 13:07:57 +01:00
Andreas Kling
f27eb315fc Build: Build Userland with -O2, Kernel with -Os
For some reason I don't yet understand, building the kernel with -O2
produces a way-too-large kernel on some people's systems.

Since there are some really nice performance benefits from -O2 in
userspace, let's do a compromise and build Userland with -O2 but
put Kernel back into the -Os box for now.
2021-02-24 11:38:52 +01:00
Andreas Kling
84996c6567 Everywhere: Okay let's try that -O2 build again :^)
Now that the issue with the kernel outgrowing its slot is patched,
we should be able to boot a slightly larger kernel without trouble.
2021-02-23 21:52:26 +01:00
Andreas Kling
4ba36c6a49 Build: Revert back to building with -Os
-O2 kernels are failing to boot on other people's machines for some
reason that we need to investigate. In the meantime, let's revert.
2021-02-23 21:22:20 +01:00
Andreas Kling
bc029a6314 Everywhere: Build with -O2 :^)
Let's try going faster instead of smaller.
2021-02-23 19:43:44 +01:00
Andreas Kling
87bb00f6ab Build: Only use -fstack-clash-protection with GCC
This is not yet supported by Clang, so let's disable it for non-GCC
compilers for now. (CLion was whining about it.)
2021-02-23 17:41:03 +01:00
Itamar
7df61e2c9b Toolchain: Use -ftls-model=initial-exec by default
Our TLS implementation relies on the TLS model being "initial-exec".
We previously enforced this by adding the '-ftls-model=initial-exec'
flag in the root CmakeLists file, but that did not affect ports - So
now we put that flag in the gcc spec files.

Closes #5366
2021-02-19 15:21:24 +01:00
Andreas Kling
7142562310 Everywhere: Build with -fstack-clash-protection
This option causes GCC to generate code to prevent "stack clash" style
attacks where a very large stack allocation is used in to jump over the
stack guard page and into whatever's next to it.
2021-02-19 09:12:30 +01:00
Andreas Kling
713b3b36be DynamicLoader+Userland: Enable RELRO for shared libraries as well :^)
To support this, I had to reorganize the "load_elf" function into two
passes. First we map all the dynamic objects, to get their symbols
into the global lookup table. Then we link all the dynamic objects.

So many read-only GOT's! :^)
2021-02-19 00:03:03 +01:00
Andreas Kling
fa4c249425 LibELF+Userland: Enable RELRO for all userland executables :^)
The dynamic loader will now mark RELRO segments read-only after
performing relocations. This is pretty cool!

Note that this only applies to main executables so far,.
RELRO support for shared libraries will require some reorganizing
of the dynamic loader.
2021-02-18 18:55:19 +01:00
AnotherTest
5729e76c7d Meta: Make it possible to (somewhat) build the system inside Serenity
This removes some hard references to the toolchain, some unnecessary
uses of an external install command, and disables a -Werror flag (for
the time being) - only if run inside serenity.

With this, we can build and link the kernel :^)
2021-02-15 17:32:56 +01:00
Brian Gianforcaro
566b916364 CMake: Add 'setup-and-run' target to perform all prereqs and run the image
Running 'ninja install && ninja image && ninja run` is kind of
annoying. I got tired, and came up with this instead, which does the
right thing and I don't have to type out the incantation.
2021-02-15 12:25:31 +01:00
Brian Gianforcaro
96943ab07c Kernel: Initial integration of Kernel Address Sanitizer (KASAN)
KASAN is a dynamic analysis tool that finds memory errors. It focuses
mostly on finding use-after-free and out-of-bound read/writes bugs.

KASAN works by allocating a "shadow memory" region which is used to store
whether each byte of memory is safe to access. The compiler then instruments
the kernel code and a check is inserted which validates the state of the
shadow memory region on every memory access (load or store).

To fully integrate KASAN into the SerenityOS kernel we need to:

 a) Implement the KASAN interface to intercept the injected loads/stores.

      void __asan_load*(address);
      void __asan_store(address);

 b) Setup KASAN region and determine the shadow memory offset + translation.
    This might be challenging since Serenity is only 32bit at this time.

    Ex: Linux implements kernel address -> shadow address translation like:

      static inline void *kasan_mem_to_shadow(const void *addr)
      {
          return ((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT)
                  + KASAN_SHADOW_OFFSET;
      }

 c) Integrating KASAN with Kernel allocators.
    The kernel allocators need to be taught how to record allocation state
    in the shadow memory region.

This commit only implements the initial steps of this long process:
- A new (default OFF) CMake build flag `ENABLE_KERNEL_ADDRESS_SANITIZER`
- Stubs out enough of the KASAN interface to allow the Kernel to link clean.

Currently the KASAN kernel crashes on boot (triple fault because of the crash
in strlen other sanitizer are seeing) but the goal here is to just get started,
and this should help others jump in and continue making progress on KASAN.

References:
* ASAN Paper: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37752.pdf
* KASAN Docs: https://github.com/google/kasan
* NetBSD KASAN Blog: https://blog.netbsd.org/tnf/entry/kernel_address_sanitizer_part_3
* LWN KASAN Article: https://lwn.net/Articles/612153/
* Tracking Issue #5351
2021-02-15 11:41:53 +01:00
Andreas Kling
1cec5f3d4c Build: Allow setting DBGLN_NO_COMPILETIME_FORMAT_CHECK via CMake flag 2021-02-08 18:27:28 +01:00
Linus Groh
dff808d087 Base: Remove /res/pci.ids and download at build time instead
This is an external file from https://pci-ids.ucw.cz that's being updated
daily, which was imported a while ago but probably shouldn't live in the
SerenityOS repository in the first place (or else would need manual
maintenance). The legal aspects of redistributing this file as we
currently do are not quite clear to me, they require either GPL (version
2 or later) or 3-clause BSD - Serenity is 2-clause BSD...

The current version we use is 2019.08.08, so quite outdated - and while
most of these devices are obviously not supported, we're still capable
of *listing* them, so having an up-to-date version with recent additions
and fixes would be nice.

This updates the root CMakeLists.txt to check for existence of the file
and download it if not found - effectively on every fresh build. Do note
that this is not a critical file, and the system runs just fine should
this ever fail. :^)
2021-02-07 01:14:36 +01:00
Andreas Kling
e87eac9273 Userland: Add LibSystem and funnel all syscalls through it
This achieves two things:

- Programs can now intentionally perform arbitrary syscalls by calling
  syscall(). This allows us to work on things like syscall fuzzing.

- It restricts the ability of userspace to make syscalls to a single
  4KB page of code. In order to call the kernel directly, an attacker
  must now locate this page and call through it.
2021-02-05 12:23:39 +01:00