ladybird/Documentation/RunningTests.md

Running SerenityOS Tests

There are two classes of tests built during a SerenityOS build: host tests and target tests. Host tests run on the build machine, and use Lagom to build SerenityOS userspace libraries for the host platform. Target tests run on the SerenityOS machine, either emulated or bare metal.

Running Host Tests

There are two ways to build host tests: from a full build, or from a Lagom-only build. The only difference is the CMake command used to initialize the build directory.

For a full build, pass -DBUILD_LAGOM=ON to the CMake command.

mkdir -p Build/lagom
cd Build/lagom
cmake ../.. -GNinja -DBUILD_LAGOM=ON

For a Lagom-only build, pass the Lagom directory to CMake. The BUILD_LAGOM CMake option is still required.

mkdir BuildLagom
cd BuildLagom
cmake ../Meta/Lagom -GNinja -DBUILD_LAGOM=ON

In both cases, the tests can be run via ninja after doing a build. Note that test-js requires the SERENITY_SOURCE_DIR environment variable to be set to the root of the serenity source tree when running on a non-SerenityOS host.

# /path/to/serenity repository
export SERENITY_SOURCE_DIR=${PWD}/..
ninja && ninja test

To see the stdout/stderr output of failing tests, the recommended way is to set the environment variable CTEST_OUTPUT_ON_FAILURE to 1.

CTEST_OUTPUT_ON_FAILURE=1 ninja test

# or, using ctest directly...
ctest --output-on-failure

Running with Sanitizers

CI runs host tests with Address Sanitizer and Undefined Sanitizer instrumentation enabled. These tools catch many classes of common C++ errors, including memory leaks, out of bounds access to stack and heap allocations, and signed integer overflow. For more info on the sanitizers, check out the Address Sanitizer wiki page, or the Undefined Sanitizer documentation from clang.

Note that a sanitizer build will take significantly longer than a non-santizer build, and will mess with caches in tools such as ccache. The sanitizers can be enabled with the -DENABLE_FOO_SANITIZER set of flags. Sanitizers are only supported for Lagom tests, as SerenityOS support for gcc's libsanitizer is not yet implemented.

mkdir BuildLagom
cd BuildLagom
cmake ../Meta/Lagom -GNinja -DBUILD_LAGOM=ON -DENABLE_ADDRESS_SANITIZER=ON -DENABLE_UNDEFINED_SANITIZER=ON
ninja
CTEST_OUTPUT_ON_FAILURE=1 SERENITY_SOURCE_DIR=${PWD}/.. ninja test

To ensure that Undefined Sanitizer errors fail the test, the halt_on_error flag should be set to 1 in the environment variable UBSAN_OPTIONS.

UBSAN_OPTIONS=halt_on_error=1 CTEST_OUTPUT_ON_FAILURE=1 SERENITY_SOURCE_DIR=${PWD}/.. ninja test

Running Target Tests

Tests built for the SerenityOS target get installed either into /usr/Tests or /bin. /usr/Tests is preferred, but some system tests are installed into /bin for historical reasons.

The easiest way to run all of the known tests in the system is to use the run-tests-and-shutdown.sh script that gets installed into /home/anon/tests. When running in CI, the environment variable $DO_SHUTDOWN_AFTER_TESTS is set, which will run shutdown -n after running all the tests.

For completeness, a basic on-target test run will need the SerenityOS image built and run via QEMU.

mkdir Build/i686
cd Build/i686
cmake ../.. -GNinja
ninja install && ninja image && ninja run

In the initial terminal, one can easily run the test runner script:

courage ~ $ ./tests/run-tests-and-shutdown.sh
=== Running Tests on SerenityOS ===
...

CI runs the tests in self-test mode, using the 'ci' run options and the TestRunner entry in /etc/SystemServer.ini to run tests automatically on startup.

The system server entry looks as below:

[TestRunner@ttyS0]
Executable=/home/anon/tests/run-tests-and-shutdown.sh
StdIO=/dev/ttyS0
Environment=DO_SHUTDOWN_AFTER_TESTS=1 TERM=xterm PATH=/usr/local/bin:/usr/bin:/bin
User=anon
WorkingDirectory=/home/anon
BootModes=self-test

/dev/ttyS0 is used as stdio because that serial port is connected when qemu is run with -display none and -nographic, and output to it will show up in the stdout of the qemu window. Separately, the CI run script redirects the serial debug output to ./debug.log so that both stdout of the tests and the dbgln from the kernel/tests can be captured.

To run with CI's TestRunner system server entry, SerenityOS needs booted in self-test mode. Running the following shell lines will boot SerenityOS in self-test mode, run tests, and exit.

export SERENITY_RUN=ci
export SERENITY_KERNEL_CMDLINE="boot_mode=self-test"
ninja run