Commit Graph

344 Commits

Author SHA1 Message Date
Tim Schumacher
0abc21132a Toolchain: Update mold to 1.3.1 2022-07-16 20:45:06 +01:00
Tim Schumacher
fb877effb8 Meta+Ports: Automatically generate a meson cross file that we can use 2022-07-13 21:22:52 +01:00
Andrew Kaster
010190beb8 Toolchain: Remove workarounds for initializer_list and _aligned_alloc
These are no longer required to use libc++ on Serenity
2022-06-30 12:34:16 +01:00
Andrew Kaster
efc1805b8f Toolchain: Create separate serenity xlocale support file 2022-06-30 12:34:16 +01:00
Andrew Kaster
d889294036 Toolchain: Format llvm patches per new patch format
This removes the patch series 1/N comments, git version at the bottom of
the patch, and zeros out the original commit hash.
2022-06-30 12:34:16 +01:00
Andrew Kaster
809852aeb1 Toolchain+Ports: Install host ruby into Local/ruby, not Local/$ARCH
Following the pattern for qemu, mold, and clang, we should install the
host ruby required to build the ruby port into its own install tree
rather than forcing it into the GNU compiler's bindir.
2022-06-30 12:29:18 +01:00
Andrew Kaster
dcc0f299be Toolchain+Ports: Install host python into Local/python, not Local/$ARCH
Following the pattern for qemu, mold, and clang, we should install the
host python required to build the python port into its own install tree
rather than forcing it into the GNU compiler's bindir.
2022-06-30 12:29:18 +01:00
Tim Schumacher
242bbf1be4 Toolchain: Set '/' as default division character for binutils
While we are at it, regenerate the patch and remove some outdated parts
of the description.
2022-06-30 12:27:17 +01:00
Andrew Kaster
05058d185c Toolchain: Update Platform files to match those in upstream cmake
Serenity support was merged into the CMake master branch for the 3.25.0
milestone (https://gitlab.kitware.com/cmake/cmake/-/merge_requests/6837)

Remove some settings that are now redundant from the Toolchain files.
2022-06-29 13:36:38 +01:00
Tim Schumacher
2df56f840f Revert "Toolchain: Load x64 executables at a higher address"
Now that the lower pages can be unmapped and more of the virtual
address range is available to us, we can actually use the default
mapping address of x86_64 again.

This reverts commit 292398b585.
2022-06-21 22:38:15 +01:00
kleines Filmröllchen
026cea990a Toolchain: Update mold to 1.0.3 2022-06-19 15:45:14 +01:00
Daniel Bertalan
d4d6f2d945 Toolchain: Add $arch-pc-serenity-clang symlink
This will allow us to build ports that don't allow the --target and
--sysroot compiler arguments to be specified in $CC/$CXX.
2022-06-11 22:40:23 +01:00
Daniel Bertalan
52710c2672 Toolchain: Fix shellcheck warning 2022-05-20 12:44:12 +01:00
Daniel Bertalan
d0edf2627c Toolchain+Ports: Split the GCC patches
This shouldn't cause any breaking changes, so a toolchain rebuild is not
required.

As per Hendiadyoin's request, math errno is disabled by default, which
should enable some extra compiler optimizations in LibGL and LibSoftGPU
code that uses math functions heavily.

Co-Authored-By: Ali Mohammad Pur <mpfard@serenityos.org>
2022-05-19 20:17:10 +04:30
Daniel Bertalan
6cf260c7af Toolchain: Upgrade to GCC 12.1.0
This release brings support for various C++23 constructs like `if
consteval` and multidimensional subscript operators. Vectorization is
now enabled for O2 too, and `-ftrivial-auto-var-init` has been added
which can help us find and prevent security issues coming from
uninitialized variables.

Toolchain/Patches/gcc.patch is now significanly smaller as some unused,
autoconf-generated code has been removed.
2022-05-12 13:12:37 +02:00
Lucas CHOLLET
ebb35e03be Toolchain: Allow BuildQemu.sh to resume downloads
Use -C option for curl to resume if a partially downloaded file already
exists.
2022-05-08 17:48:05 +02:00
Andrew Kaster
4334bd06b8 Toolchain: Enable LLVM Profile instrumentation
Add a patch to let llvm's InstrProfiling modules know serenity supports
all the Unix-y features required to make -fprofile-instr-generate and
-fcoverage-mapping work properly on target.
2022-05-02 01:46:18 +02:00
Andrew Kaster
29fefe2d3e Toolchain: Explicitly install llvm-config and FileCheck
Also disable the "toolchain only build" to ensure we install llvm
headers for when we want to build host tools that link against llvm.
2022-04-25 10:45:46 +02:00
EWouters
ab37791431 Toolchain: Update BuildQemu.sh to qemu-7.0
QEMU 7.0 was released on April 19th.
Release Notes: https://wiki.qemu.org/ChangeLog/7.0
2022-04-23 10:44:39 -07:00
Daniel Bertalan
01b31d9858 Toolchain+Ports: Update LLVM to 14.0.1
Besides a version bump, the following changes have been made to our
toolchain infrastructure:
- LLVM/Clang is now built with -march=native if the host compiler
  supports it. An exception to this is CI, as the toolchain cache is
  shared among many different machines there.
- The LLVM tarball is not re-extracted if the hash of the applied
  patches doesn't differ.
- The patches have been split up into atomic chunks.
- Port-specific patches have been integrated into the main patches,
  which will aid in the work towards self-hosting.
- <sysroot>/usr/local/lib is now appended to the linker's search path by
  default.
- --pack-dyn-relocs=relr is appended to the linker command line by
  default, meaning ports take advantage of RELR relocations without any
  patches or additional compiler flags.

The formatting of LLVM port's package.sh has been bothering me, so I
also indented the arguments to the CMake invocation.
2022-04-23 10:43:32 -07:00
James Mintram
81b6386c87 Toolchain: Fix the gdb build for aarch64 target on macOS host 2022-04-01 19:39:56 +01:00
James Mintram
11bcde0bbd Toolchain: Add a gcc patch to fix linker error on M1 host 2022-03-31 19:22:53 -07:00
jd
ba6ba67fa0 Toolchain: Fix serenity.nix
* x11 package name has changed to xlibsWrapper.
* texinfo is necessary for Makeinfo.
* e2fsprogs was recenty fixed on nicpkgs to include fuse2fs to mount
serenity images without root access but it needed some configuration.
2022-03-29 16:36:28 -07:00
Daniel Bertalan
5550905c00 Toolchain: Update gdb to 11.2 2022-03-24 20:08:08 -07:00
Tim Schumacher
2788d46ae0 Toolchain: Make wint_t unsigned for Clang 2022-03-19 00:11:45 +00:00
Tim Schumacher
5d0869d5ca Toolchain: Enable IFUNC support for GCC 2022-03-19 00:11:45 +00:00
Tim Schumacher
b55a3a504e Toolchain: Enforce correct sizes for size_t and ptrdiff_t 2022-03-19 00:11:45 +00:00
Jonathan Marler
97c0a8c063 Toolchain: Move nix script from Documentation into Toolchain
Moves the nix script to setup the build environment from Documentation
into the Toolchain as a callable script.  I also modified the script
to accept a "pkgs" argument to make it easy to override the nixpkgs
version from the command-line when calling the script.
2022-03-08 23:31:45 +01:00
Ali Mohammad Pur
4bd01b7fe9 Kernel: Add support for SA_SIGINFO
We currently don't really populate most of the fields, but that can
wait :^)
2022-03-04 20:07:05 +01:00
Andrew Kaster
7b0f3b6811 Toolchain: Set CMAKE_NM for Clang and GNU toolchain files
Make sure that we set CMAKE_NM, it's possible that some version of
CMake could choose a host nm binary instead of the ones we just built.

It's unlikely that host nm will understand our .dyn.relr segments unless
it's from binutils 2.38 or higher, so it might complain.
2022-02-19 21:27:46 +01:00
Andrew Kaster
76e99fa6c8 Toolchain: Update Dockerfile to use gcc 11 and add texinfo package
Missed this in the gcc 11 version update, but if anyone is using this
then they'll need gcc 11 to build Lagom.
2022-02-14 12:52:06 +03:30
Brian Gianforcaro
d386d3946a Toolchain: Fix QEMU build with latest gcc by disabling -fcf-protection
I noticed after upgrading my machine that the QEMU is no longer building
due to GCC enabling `-fcf-protection` by default, even for targets that
don't support it.

The included patch came from the QEMU development list, but hasn't be
included in any patch releases at the time of writing.
https://lore.kernel.org/all/20220208211937.79580-1-vineetg@rivosinc.com/

Until QEMU patches, lets fix it on our end by patching before we build.
2022-02-13 17:11:53 +00:00
Daniel Bertalan
85780fafcd Toolchain: Update binutils to version 2.38
This version comes with general fixes and, more importantly, introduces
support for the packed RELR relocation format.
2022-02-11 18:07:53 +01:00
Daniel Bertalan
4055c393fc Toolchain: Add support for building the userland with the mold linker
This commit adds support for building the SerenityOS userland with the
new [mold linker].

This is not enabled by default yet; to link using mold, run the
`Toolchain/BuildMold.sh` script to build the latest release of mold, and
set the `ENABLE_MOLD_LINKER` CMake variable to ON. This option relies on
toolchain support that has been added just recently, so you might need
to rebuild your toolchain for mold to work.

[mold linker]: https://github.com/rui314/mold
2022-01-24 10:41:47 +00:00
Daniel Bertalan
ad649c48da Toolchain: Backport support for the mold linker
The `-fuse-ld=mold` flag was only added in the not-yet-released GCC 12.

Cherry-picked from the following upstream commit:
https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=ad964f7eaef9c03ce68a01cfdd7fde9d56524868
2022-01-24 10:41:47 +00:00
Daniel Bertalan
bd3ed700f2 Toolchain+Ports: Use the GNU hash format by default
This hash format offers faster symbol lookup than the System V hash.

We've been using it in all our shared libraries for a long time, but did
not have it enabled by default in our toolchain, so ports couldn't make
use of it.
2022-01-24 10:41:47 +00:00
Daniel Bertalan
5576e9c4c5 Kernel+Toolchain: Use .init_array section for global constructors
Before this change, our dynamic linker's global constructor handler
relied on the GNU linker implicitly including the content of `.ctors`
section inside `.init_array`. The mold linker does not do this, so
global constructors would fail to be called in the mold-built userland.

There is no point in sticking to `.ctors`, as most other systems already
use the superior `.init_array` scheme. This commit changes the kernel
linker script to not discard this new section, and enables it by default
in our toolchain.
2022-01-24 10:41:47 +00:00
Daniel Bertalan
9e8c4bb072 Toolchain: Enable the -rdynamic flag in the gcc driver
Although we handle it in the linker spec file, gcc will actually reject
this argument unless it's also enabled in the option file.

CMake adds this flag if the minimum required version is 3.3 or less (see
CMP0065), so old projects would fail to compile because of this
unrecognized option.
2022-01-12 14:54:12 +01:00
Daniel Bertalan
fbccf77d37 Toolchain: Enable std::filesystem support in libc++
Now that the last missing function (`fchmodat(2)`) has been added to
LibC, libc++ compiles successfully with std::filesystem support enabled.
2022-01-12 14:54:12 +01:00
Daniel Bertalan
162606f358 Toolchain+CI: Link LLVM with LLD if available
According to most benchmarks, LLD is faster than GNU ld and the macOS
linker, so let's use it if possible in order to speed up the toolchain
build.
2022-01-10 09:55:45 +03:30
Daniel Bertalan
43008267b2 Toolchain: Do not overwrite libraries with stubs when rebuilding LLVM
We erroneously appended ".so" after the base name for the library,
so we ended up checking for the existence of e.g. `libc.so.so`,
which obviously didn't exist, so we overwrote the existing libraries
when we rebuilt the toolchain.
2022-01-10 09:55:45 +03:30
Daniel Bertalan
b3dbf204fc Toolchain: Don't use GNU objcopy in the LLVM toolchain
Our build of LLVM's objcopy now supports the single missing feature
(--update-section) that previously forced us to use the one from GNU
Binutils. This means that there is no reason anymore to build Binutils
alongside LLVM's tools.
2022-01-10 09:55:45 +03:30
Daniel Bertalan
b19cc3cdcb Toolchain: Backport --update-section support to llvm-objcopy
This commit backports the LLVM commit that adds support for the
`--update-section` flag to llvm-objcopy. We use this feature of GNU
objcopy to embed the symbol map in the kernel.

The corresponding LLVM Phabricator Differential Revision can be found
here: https://reviews.llvm.org/D112116

This patch is identical to the upstream commit, except for two hunks
that had to be changed as they didn't apply cleanly.
2022-01-10 09:55:45 +03:30
Daniel Bertalan
20e29c403e Toolchain: Fix macOS build failure due to a malformed patch
Discord user aesophor pointed out that the GCC toolchain fails to build
on macOS, and traced the issue back to 41ea37f2, which is the latest
change to `gcc.patch`. Similarly, when I tried to run BuildIt.sh in the
`--dev` mode, `git apply` complained about the patch being malformed.

I regenerated the patch by manually applying the changes of 41ea37f2 on
top of a known good GCC source tree, and I sent the new file to them.
They reported that this fixed the build issue they were having.
2022-01-09 14:15:24 +01:00
Martin Bříza
41ea37f25a Toolchain/GCC: Declare we don't have memalign
I'm not even sure if this is a hack. However, we don't implement
memalign so it's necessary to tell GCC so it doesn't go looking for it
in their implementation of `new`
2022-01-07 18:38:32 +01:00
Daniel Bertalan
84c6d6649e Toolchain: Fix building the aarch64 toolchain
The `aarch64/t-aarch64` makefile fragment needs to be included for the
aarch64-specific parts of GCC to be built. Before 738e52da5, this was
done implicitly, but now it is not. This caused the following error when
building the toolchain: "aarch64-builtins.o: No such file or directory".
2022-01-04 17:40:09 +00:00
Jean-Paul Balabanian
913511249b Toolchain: Add support for Clang on MacOS
This commit adds Darwin as a possible host for building the toolchain
with Clang.
2022-01-04 00:40:11 +03:30
Andrew Kaster
eb672aef31 Toolchain: Add CMAKE_PLATFORM_USES_PATH_WHEN_NO_SONAME to Platform
This property tells CMake that if a library is missing a SONAME field,
the link editor(s) we use will insert the full path to the library into
the binary. This is the behaivor of GNU ld compatible linkers, so let's
avoid that possiblity by telling CMake that it really doesn't want to
let the linker embed the full path to the lib. This is especially
important when cross-compiling things for ports and such, as the full
path to the lib will have absolutely nothing to do with the runtime path
2022-01-03 11:08:45 +00:00
Andrew Kaster
c5898806d2 Toolchain: Use Platform/SerenityOS.cmake in LLVM toolchain build
By setting CMAKE_MODULE_PATH in the LLVM initial cache scripts, we can
make the "SerenityOS" CMAKE_SYSTEM_NAME usable in the builds of
compiler-rt, libunwind, libcxxabi and libcxx.

This simplifies some toolchain patches and brings the cross-compiler
patches closer to the Port's patches, and closer to something
upstreamable.
2022-01-03 11:08:45 +00:00
tuftedocelot
5810467c97 Build: Remove gzip -k usage in PCI/USB ID files and crypt for OpenBSD
OpenBSD gzip does not have the -k flag to keep the original after
extraction. Work around this by copying the original gzip to the dest
and then extracting. A bit of a hack, but only needs to be done for the
first-time or rebuilds

OpenBSD provides crypt in libc, not libcrypt. Adjust if/else to check
for either and proceed accordingly

Remove outdated OpenBSD checks when building the toolchain
2021-12-29 03:46:56 -08:00
Daniel Bertalan
738e52da56 Toolchain: Link libgcc_s on an as-needed basis
If we pass `-lgcc_s` explicitly to the linker, it will be added as a
dependency even if no functions are used from it. This behavior is not
consistent with other systems. GCC can already handle passing the
correct flags, so let's rely on that instead.

As an added benefit, we now get support for the `-static-libgcc` flag;
and `-static-pie` will no longer mistakenly link us against the dynamic
version of libgcc.

No toolchain rebuild is required.
2021-12-16 21:27:03 +02:00
Brian Gianforcaro
fb53eacefd Toolchain: Update BuildQemu.sh to qemu-6.2
QEMU 6.2 was released on December 14th.
Release Notes: https://wiki.qemu.org/ChangeLog/6.2
2021-12-16 03:15:40 -08:00
Daniel Bertalan
36bd230ffa Toolchain/Clang: Fix CMake using utilities from the LLVM port
If we have the LLVM port installed, CMake might pick up some of the
tools installed as part of it (`llvm-ar`, `llvm-strip`, etc.) instead of
the ones belonging to the host toolchain. These, of course, can't be run
on the host platform, so builds would eventually fail. This made it
impossible to rebuild the LLVM toolchain.

We now set these variables explicitly when compiling the LLVM runtime
libraries in order to avoid this issue.
2021-11-28 09:38:57 -08:00
Daniel Bertalan
b1f6bfca7f Toolchain/Clang: Support using libstdc++ as the C++ standard library
This will come in handy if we want to use the LLVM port with a GNU host
compiler.

As of version 13, libc++ uses `__attribute__((using_if_exists))` to
import global LibC functions into the `std` namespace, which allows some
symbols to be absent. GCC does not support this attribute, so it fails
to build libc++ due to some obscure `wchar.h` functions. This means that
cross-compiling libc++ is not possible; and on-target builds would be
tedious, so we'll be better off using the toolchain's `libstdc++`.
2021-11-28 09:38:57 -08:00
Daniel Bertalan
ce3b219021 Toolchain/Clang: Allow setting a default unwind library
This will let us use `libgcc` for unwinding when we build the LLVM port
with the GNU toolchain.
2021-11-28 09:38:57 -08:00
Daniel Bertalan
ea8335cda1 Toolchain/Clang: Pick up includes when running inside Serenity
The toolchain should work without setting `--sysroot` when we build
inside SerenityOS.
2021-11-28 09:38:57 -08:00
Daniel Bertalan
91546f42f3 Toolchain/Clang: Add support for -static-libstdc++
This option is already used by our GNU toolchain to avoid creating a
circular dependency between LibC and `lib(std)c++`.
2021-11-28 09:38:57 -08:00
Jelle Raaijmakers
8292061f53 Toolchain: Remove git clone of project from Dockerfile
The goal of these more recent additions to the Dockerfile is to provide
a working copy of SerenityOS with the toolchain prebuilt. To me, these
additions feel misplaced:

- The toolchain is built assuming the i686 architecture, which may not
  be what you want.
- You get a shallow clone of the project limiting you in your abilities
  to navigate through the project's history or bisect.
- There's this awkward directory structure of `/serenity/serenity-git`
  and `/serenity/out`.

The Dockerfile is immensely useful for building SerenityOS in a
containerized environment, separate from the host's environment. If we
want to automate builds, we can always use CI or extend this image to
do so. For now, let's remove the `git clone` and associated actions.

Fixes #9310.
2021-11-20 21:16:51 +00:00
Jelle Raaijmakers
cff3add897 Toolchain: Remove unused gettext package from Dockerfile
It is no longer used to build the git port, it seems.
2021-11-20 21:16:51 +00:00
Jelle Raaijmakers
b08a265840 Toolchain: Remove unused wget package from Dockerfile 2021-11-20 21:16:51 +00:00
Jelle Raaijmakers
8744086781 Toolchain: Add Qemu build dependencies to Dockerfile
Also add `libsdl2-dev` as a required dependency and reorder the
list of packages passed to `apt-get`.
2021-11-20 21:16:51 +00:00
Jelle Raaijmakers
b7bf1a4330 Toolchain: Update Dockerfile to ubuntu:21.10 2021-11-20 21:16:51 +00:00
Andrew Kaster
6f5b9cc031 Toolchain: Enable clang-tools-extra for LLVM
If we want to use clang-tidy on the codebase, we'll need to build
clang-tidy from an LLVM that has been patched and built with Serenity
cross-compilation support.
2021-11-14 22:52:35 +01:00
Tim Schumacher
292398b585 Toolchain: Load x64 executables at a higher address
Serenity defines a protected range of memory that must not be mmapped,
and is apparently reserved for kernel tasks. In this case, the protected
range is anything below 0x800000.

However, in its default setting, binutils chooses the memory address
0x400000 as the mapping address for executables that do not have PIE
enabled, resulting in mmap being unable to map the file unless the load
address has been overwritten at link time or if it's a PIE.

To mitigate this, move the default base address somewhere outside of
that range (and preferably not anywhere close near the beginning of the
useable virtual memory space, to avoid running into it during sequential
allocations).
2021-11-14 22:25:15 +01:00
Daniel Bertalan
061fa8a1ca Toolchain: Add aligned allocation support to libc++
Now that we have `_aligned_malloc` and `_aligned_free`, we can finally
enable C++17 aligned allocation support.
2021-11-14 16:46:21 +00:00
Tim Schumacher
22562b4b17 CMake: Assume working compiler instead of using static linking
We were previously using TRY_COMPILE_TARGET_TYPE to bypass the compiler
check at the beginning of the CMake build, since we don't have LibC
available and therefore can't link at that point.

However, this breaks a lot of assumptions in try_compile when it comes
to library checks. While this was the main idea behind our usage of the
flag, it also has some really nasty side effects when software wants
to find out what library a symbol is in.

Instead, just manually tell CMake that our compiler works as intended
and keep the target type setting at its default.
2021-11-07 20:11:23 -08:00
Tim Schumacher
8f060bed17 Toolchain: Use dynamic paths for cross-compile CMake toolchains
`CMAKE_INSTALL_PREFIX` is supposed to be the in-system installation
path. The sysroot path on the host doesn't belong there, since other
applications will duplicate that path when applying their respective
sysroot.
2021-10-31 12:09:25 +01:00
Daniel Bertalan
06fc64be13 Toolchain+Meta: Update LLVM version to 13.0.0
This commit updates the Clang toolchain's version to 13.0.0, which comes
with better C++20 support and improved handling of new features by
clang-format. Due to the newly enabled `-Bsymbolic-functions` flag, our
Clang binaries will only be 2-4% slower than if we dynamically linked
them, but we save hundreds of megabytes of disk space.

The `BuildClang.sh` script has been reworked to build the entire
toolchain in just three steps: one for the compiler, one for GNU
binutils, and one for the runtime libraries. This reduces the complexity
of the build script, and will allow us to modify the CI configuration to
only rebuild the libraries when our libc headers change.

Most of the compile flags have been moved out to a separate CMake cache
file, similarly to how the Android and Fuchsia toolchains are
implemented within the LLVM repo. This provides a nicer interface than
the heaps of command-line arguments.

We no longer build separate toolchains for each architecture, as the
same Clang binary can compile code for multiple targets.

The horrible mess that `SERENITY_CLANG_ARCH` was, has been removed in
this commit. Clang happily accepts an `i686-pc-serenity` target triple,
which matches what our GCC toolchain accepts.
2021-10-17 17:09:58 +01:00
Daniel Bertalan
c524f58290 Toolchain: Introduce stubs for core system libraries
This allows the linker to link against these dynamic libraries when
compiling libc++/libunwind, without having to do a separate
bootstrapping LibC build.

Without this change, libc++ would fail to pick up the need to link to
`LibPthread` if no prior builds of it existed. Because of this, we'd
immediately have an assertion failure in SystemServer, as mutexes are
used for the safe construction of function-local static variables.
2021-10-17 17:09:58 +01:00
Ben Wiederhake
52e9f25403 Everywhere: Change from http to https where feasible
I used "git grep -FIn http://" to find all occurrences, and looked at
each one. If an occurrence was really just a link, and if a https
version exists, and if our Browser can access it at least as well as the
http version, then I changed the occurrence to https.

I'm happy to report that I didn't run into a single site where Browser
can't deal with the https version.
2021-10-05 02:08:08 +02:00
Eric Seifert
01a06dde0e Ports: Add Ruby 3.0.2 2021-09-24 10:04:47 +02:00
Timothy Flynn
5a2f41fff0 Toolchain: Add --ci option to BuildClang to enable ccache 2021-09-21 15:39:17 +03:00
Tim Schumacher
dea43d88e7 Toolchain: Remove breaking mbstate_t define from clang 2021-09-17 00:14:57 +00:00
Andrew Kaster
b9e3647e66 Meta+Toolchain: Rename CMAKE_CXXFILT to SERENITY_CXXFILT
The "CMAKE_<foo>" variable namespace is reserved, and CXXFILT is not
currently a variable known to upstream CMake.
2021-09-15 19:04:52 +04:30
Andrew Kaster
b5c98ede08 Meta: Switch to a SuperBuild that splits host and target builds
Replace the old logic where we would start with a host build, and swap
all the CMake compiler and target variables underneath it to trick
CMake into building for Serenity after we configured and built the Lagom
code generators.

The SuperBuild creates two ExternalProjects, one for Lagom and one for
Serenity. The Serenity project depends on the install stage for the
Lagom build. The SuperBuild also generates a CMakeToolchain file for the
Serenity build to use that replaces the old toolchain file that was only
used for Ports.

To ensure that code generators are rebuilt when core libraries such as
AK and LibCore are modified, developers will need to direct their manual
`ninja` invocations to the SuperBuild's binary directory instead of the
Serenity binary directory.

This commit includes warning coalescing and option style cleanup for the
affected CMakeLists in the Kernel, top level, and runtime support
libraries. A large part of the cleanup is replacing USE_CLANG_TOOLCHAIN
with the proper CMAKE_CXX_COMPILER_ID variable, which will no longer be
confused by a host clang compiler.
2021-09-15 19:04:52 +04:30
Nico Weber
17ab44e7d7 Meta: Make BuildClang.sh produce less output when running in a TTY
Ninja disables its fancy output mode when it's not writing to a TTY.
So don't pipe its output into something else, so that it writes to
a TTY if the invoking terminal is a TTY.
2021-09-07 20:29:22 +01:00
Nico Weber
eb7b8a7bbe Meta: Don't pass LLVM_LLVM_BUILD_LLVM_DYLIB to cmake
`LLVM_LLVM_BUILD_LLVM_DYLIB` does not exist, so passing this does
nothing but make CMake warn.

However, since we pass `LLVM_LINK_LLVM_DYLIB`, `LLVM_BUILD_LLVM_DYLIB`
(the correct spelling) defaults to true anyways.  So let's pass fewer
flags.

No behavior change, but fixes a CMake warning.
2021-09-07 20:29:22 +01:00
Nico Weber
1ed1a57707 Meta: Make serenity.sh rebuild-toolchain aarch64 clang work 2021-09-07 20:29:22 +01:00
Idan Horowitz
783a58dbc7 Toolchain: Build aarch64-gdb for cross-debugging on x86 2021-09-07 12:58:46 +02:00
Tom
77953a937d Meta: Add the ability to specify clang with serenity.sh
This enables maintaining gcc and clang builds side-by-side.
2021-09-03 23:12:17 +02:00
Nico Weber
a900792921 Toolchain: Also build aarch64-softmmu in BuildQemu.sh 2021-08-28 21:51:30 +01:00
Nico Weber
7052f403c8 Toolchain: Enough to make rebuild-toolchain aarch64 work
The gcc patch might not be completely correct, but at least the
toolchain completes building.
2021-08-28 14:43:07 +01:00
Nico Weber
61b6f69947 Toolchain: Regenerate binutils.patch and gcc.patch
I locally modified Meta/serenity.sh to pass `--dev` to BuildIt.sh
in build_toolchain(). Then I ran `Meta/serenity.sh rebuild-toolchain`,
cd'd into Toolchain/Tarballs/binutils-2.37, `git add`ed unadded files in
`git status`, and then ran `git diff > ../../Patches/binutils.patch`.
Then I did the same for Toolchain/Tarballs/gcc-11.2.0 (and was careful
not to `git add` serenity-kernel.h, since that's created by
Toolchain/BuildIt.sh).

No behavior change. This just rewrites the patch like git writes it.
2021-08-28 14:43:07 +01:00
Brian Gianforcaro
aceea42ba4 Toolchain: Update BuildQemu.sh to latest 6.1.0
Change Log: https://wiki.qemu.org/ChangeLog/6.1
2021-08-25 12:13:13 +02:00
Timothy Flynn
7ea55c883b Toolchain: Make Kernel/API headers available to Clang toolchain build
Copied from 9b79867909.
2021-08-17 13:31:43 +01:00
Andreas Kling
9b79867909 Toolchain: Make Kernel/API/ headers available during toolchain build 2021-08-14 20:25:56 +02:00
Daniel Bertalan
657fbc1e6c Toolchain: Remove static LLVM libraries
We link against these dynamically anyways, so having them around is not
useful. Removing them frees precious storage space on CI.
2021-08-08 23:54:00 +02:00
Daniel Bertalan
d3595477d8 Toolchain: Build libLLVM as a dynamic library
This library is used by virtually all executables in the Clang
toolchain. By default, it is linked statically, which leads to huge
file sizes and us running out of artifact storage disk space on CI.
2021-08-08 23:54:00 +02:00
Gunnar Beutner
a29c18f969 Toolchain+Ports: Let -nostdlib make us not link against libgcc_s
For now this is not a mandatory toolchain rebuild.
2021-08-08 16:41:51 +02:00
Daniel Bertalan
9809e685c1 Toolchain: Add caching to BuildClang.sh for CI 2021-08-08 10:55:36 +02:00
Daniel Bertalan
15e217ea68 Toolchain: Add LLVM patch and script for building it
This contains all the bits and pieces necessary to build a Clang binary
that will correctly compile SerenityOS.

I had some trouble with getting LLVM building with a single command, so
for now, I decided to build each LLVM component in a separate command
invocation. In the future, we can also make the main llvm build step
architecture-independent, but that would come with extra work to make
library and include paths work.

The binutils build invocation and related boilerplate is duplicated
because we only use `objdump` from GNU binutils in the Clang toolchain,
so most features can be disabled.
2021-08-08 10:55:36 +02:00
Linus Groh
a613e00caa Ports: Change Python's auth_type to sha256 2021-08-03 21:01:07 +01:00
Gunnar Beutner
5df8c274bb Toolchain+Ports: Fix building ports on x86_64
Fixes #8377.
2021-07-28 21:28:31 +02:00
Gunnar Beutner
9c431ae7ef Toolchain: Update GCC to version 11.2.0 2021-07-28 21:28:31 +02:00
Gunnar Beutner
cfeffbe524 Toolchain: Update binutils to version 2.37 2021-07-28 21:28:31 +02:00
Gunnar Beutner
12bfc7b3fc Toolchain+Ports: Ignore -arch on macOS M1 hosts
CMake specifies -arch arm64 for our toolchain. Unfortunately that's an
option GCC only understands when built for macOS. This causes the build
to fail.

I haven't been able to get CMake to not specify that option so this adds
a dummy option to GCC.
2021-07-24 14:04:13 +02:00
Gunnar Beutner
9bc3ad75b4 Toolchain: Use correct variable when deleting the QEMU tarball 2021-07-10 11:13:56 +01:00
modmuss50
036f2a14ed Toolchain: Fix building toolchain on arm64 macs
This patch is based off:
https://github.com/osx-cross/homebrew-avr/pull/248 and
https://github.com/riscv/riscv-gnu-toolchain/issues/800
2021-07-08 16:07:14 +02:00
Gunnar Beutner
d86275d271 Kernel+Toolchain: Remove the kernel-specific toolchain
This is no longer necessary now that the kernel doesn't use libsupc++
anymore.
2021-07-06 19:08:22 +02:00
Gunnar Beutner
128e504de6 Meta: Change the QEMU binary directory to Toolchain/Local/qemu
Previously we'd place the QEMU binaries into the architecture-specific
toolchain directory. This is a problem because the BuildIt.sh script
clears those directories which also removes the QEMU binaries users
may have built earlier. Also, the QEMU binaries are not specific to
the target architecture.
2021-07-03 12:06:20 +02:00
kleines Filmröllchen
7ff363127b Toolchain: Allow containerized work with Serenity via Docker
Docker is a nice way of doing build automation, or just
containerizing builds for increased safety and isolating unstable
packages. The old Dockerfile in the toolchain did not satisfy these
needs. The new Dockerfile is known to run successfully on Docker
version 20.10.7. It clones the SerenityOS repo and builds the
toolchain. In this way, it is intended to be a starting point for other
Docker images that can e.g. run builds. For example, one can simply run
this docker image as-is, exec a shell in it and run a build there.
2021-06-29 15:56:02 +02:00
Idan Horowitz
5b909a94e7 Toolchain: Build the x86_64 target in addition to i386 in BuildQemu.sh 2021-06-26 17:59:54 +02:00
Jelle Raaijmakers
3067ee9c4c Toolchain: Add gettext as a dependency to Dockerfile
We need `msgfmt` inside of the `gettext` package in order to build the
git port.
2021-06-07 22:46:53 +02:00
Jelle Raaijmakers
6af9d87258 Toolchain: Add ccache to Dockerfile
Following up on 2d38d56e, we were missing this in our Dockerfile.
2021-06-07 19:22:03 +02:00
Gunnar Beutner
988dfa7f33 Toolchain+Ports: Fix building binutils on FreeBSD
This imports the upstream patch from
https://sourceware.org/bugzilla/show_bug.cgi?id=27382

Fixes #7407.
2021-06-07 09:51:12 +02:00
Jelle Raaijmakers
68f0170bc6 Toolchain: Add ImageMagick to Dockerfile 2021-06-04 12:54:27 +02:00
Hediadyoin1
6fe7d4d7b8 Toolchain: Use gcc's ar
the vanilla versions might not handle all things, that gcc can do;
For example is lto not really supported by the vanilla versions
source:
https://gcc.gnu.org/wiki/LinkTimeOptimizationFAQ
2021-06-03 00:24:06 +01:00
Oleg Kosenkov
839aad6e5b
Toolchain: Set CMAKE_AR to the right executable 2021-06-02 23:25:44 +01:00
Gunnar Beutner
7dedf09f3d Toolchain: Remove dependency check for texinfo
Turns out we don't need that after all.
2021-05-30 23:36:10 +01:00
Gunnar Beutner
259822493f Toolchain: Check whether required tools and libraries are available
Rather than having the toolchain build fail half-way through we should
check whether the user has installed all the required tools and
libraries early on.
2021-05-30 13:06:28 +01:00
Gunnar Beutner
afbab621aa Toolchain: Use set -o pipefail to the toolchain build script
Previously the buildstep function would obscure error codes because
the return value of the function was the exit code for the sed command
which caused us to continue execution even though one of the build
steps had failed.

With set -o pipefail the return value of the buildstep function is
the real command's exit code.
2021-05-27 08:58:37 +01:00
Idan Horowitz
bcfde43849 Toolchain: Replace the -march=native flag with -mtune=native
This ensures inter-machine compatibility by not emitting any processor
specific instructions. This fixes the issue raised by the non AVX-512
supporting GitHub actions runners.
2021-05-21 15:23:07 +02:00
Brian Gianforcaro
17ff895e1c Toolchain: Enable native host optimizations when building toolchain
-march=native specializes the binaries for the CPU features available on
the CPU the binary is being compiled on. This matches the needs of the
Toolchain, as it's always built and used on that machine only.

This should be safe for the github actions VMs as well, as they all run
on a standard VM SKU in "the cloud".

I saw small but notable improvements in end-2-end build times in my
local testing. Each compilation unit is on average around a second
faster on my Intel(R) Core(TM) i7-8705G CPU @ 3.10GHz.
2021-05-21 10:04:49 +01:00
Gunnar Beutner
b223233b15 Toolchain+Ports: Skip link tests for libstdc++v3
This makes stdlib.h and stdio.h functions available in the std
namespace for C++.

libstdc++v3's link tests can fail if you don't have an up-to-date
build directory, for example:

1. Have libc with missing _Exit symbol because you haven't done
   a build since that was added.

2. Run toolchain rebuild. libstdc++v3's configure script will
   realize that it can do link tests in general but will fail
   later on when it tries to link a program that tests for _Exit.

Even though this is a toolchain patch this does not necessarily
require rebuilding the toolchain right away. This is only required
once we start using any of these new members in the std namespace,
e.g. for ports.
2021-05-09 15:35:01 +02:00
Gunnar Beutner
1889006b90 Toolchain: Make -nodefaultlibs not link against libgcc_s
This fixes the -nodefaultlibs flag for gcc which previously
linked against libgcc_s anyway. Even though this is a toolchain
patch we don't need to rebuild the toolchain right away.
2021-05-07 15:35:50 +02:00
Brian Gianforcaro
8b856bd5a1 Toolchain: Fix expansion bugs and make BuildIt.sh shellcheck compliant
BuildIt.sh had a bunch of SC2086 errors, where we were not quoting
variables in variable expansions. The logic being:

    Quoting variables prevents word splitting and glob expansion,
    and prevents the script from breaking when input contains spaces,
    line feeds, glob characters and such.

    Reference: https://github.com/koalaman/shellcheck/wiki/SC2086

As bcoles noticed in #6772, shellcheck actually found a real bug here,
where the user's build directory included spaces.

Close: #6772
2021-05-05 21:26:37 +02:00
Brian Gianforcaro
f558a44610 Toolchain: Make BuildFuseExt2.sh shellcheck compliant
BuildFuseExt2.sh was saying it should be run under /bin/sh but it is
using bash extensions like pushd/popd, ${BASH_SOURCE[0]}, etc. So just
run it under bash to avoid any potential issues.
2021-05-05 21:26:37 +02:00
Brian Gianforcaro
b4927a753d Toolchain: Make BuildPython.sh shellcheck compliant
Shellcheck is unable to source non-literal includes,
so inform shellcheck to just ignore this include.
2021-05-05 21:26:37 +02:00
Gunnar Beutner
d685db6eb6 Toolchain: Enable building all code with -fPIC
Ordinarily this would force the compiler to not inline certain
symbols and call them via the PLT instead. To counteract this
I've also added -fno-semantic-interposition which disables
ELF symbol interposition. Our dynamic loader doesn't support
this anyway and we might even consider not implementing this
at all.

Even though this is a toolchain change this doesn't require
rebuilding the toolchain unless you're planning to build
for the x86_64 arch.
2021-05-03 08:42:39 +02:00
Brian Gianforcaro
5acac54085 Toolchain: Upgrade to QEMU 6.0 release 2021-04-30 08:36:50 +02:00
Gunnar Beutner
19064eaaca Toolchain: Update the binutils patch file to match the port
Previously the toolchain's binutils would not have been able to
build binaries on 32-bit host systems (not that this would be
much of an issue nowadays) because one of the #ifdefs was in
the wrong place.

I moved the #ifdef in the port's patch and this now updates
the toolchain's patch file to match the port's patch.
2021-04-29 10:33:44 +02:00
Gunnar Beutner
8cd62b5780 Toolchain+Ports: Update GCC to version 11.1.0 2021-04-29 10:33:44 +02:00
Brian Gianforcaro
d19180433f Toolchain: Update QEMU to 6.0.0-rc5
Changes since rc4:

0cef06d187: Update version for v6.0.0-rc5 release
5351fb7cb2: hw/block/nvme: fix invalid msix exclusive uninit
ffa090bc56: target/s390x: fix s390_probe_access to check PAGE_WRITE_ORG
bc38e31b4e: net: check the existence of peer before trying to pad
2021-04-28 09:43:42 +02:00
Brian Gianforcaro
2ef93a3c07 Build: Use variables when concatenating Toolchain paths.
Make this stuff a bit easier to maintain by using the
root level variables to build up the Toolchain paths.

Also leave a note for future editors of BuildIt.sh to
give them warning about the other changes they'll need
to make.
2021-04-27 13:07:04 +02:00
Jelle Raaijmakers
03569f9686 Toolchain/Dockerfile: Add rsync and unzip
Both utilies are used in the .port_include.sh file.
2021-04-27 08:58:29 +02:00
Brian Gianforcaro
1399048662 Toolchain: Update QEMU to 6.0-rc4 2021-04-26 09:10:53 +02:00
Gunnar Beutner
e6953d14d8 Toolchain: Update cmake platform definition
This fixes building cmake and other ports which use cmake
to detect whether we have -ldl.
2021-04-25 10:14:50 +02:00
Jelle Raaijmakers
7b6954afe3 Toolchain/Dockerfile: Update to Ubuntu 21.04, add git 2021-04-22 12:30:02 +02:00
Gunnar Beutner
d71f64b97f Toolchain: Build GCC with optimizations 2021-04-20 23:23:23 +02:00
Gunnar Beutner
75d41657d5 Toolchain+Ports: Enable threads for gcc
This enables POSIX threads for GCC and makes the -pthread
argument available.
2021-04-20 21:08:17 +02:00
Panagiotis Vasilopoulos
e45e0eeb47 Everywhere: Replace SERENITY_ROOT with SERENITY_SOURCE_DIR 2021-04-20 15:27:52 +02:00
xackus
6e2f2cd8b1 Toolchain: Don't produce debug symbols
Previously debug symbols were produced and then stripped.
2021-04-19 09:49:14 +02:00
Emanuele Torre
685556ae84 Toolchain/BuildIt.sh: let's have colours on macOS too :^)
MacOS's sed doesn't support "\x1b" expansion unlike GNU sed, but we can
still use bash's $'\x1b' to expand it.
2021-04-18 19:00:49 +02:00
Gunnar Beutner
05d140e7bd Toolchain: Add missing buildstep calls 2021-04-18 15:51:56 +02:00
Gunnar Beutner
8a50c8431b Toolchain: Cache the output of uname -s 2021-04-18 15:51:56 +02:00
Gunnar Beutner
c60f02fbc2 Toolchain: Build fix for macOS
Unfortunately their sed behaves a tiny bit different to how
GNU sed does.
2021-04-18 15:51:56 +02:00
Gunnar Beutner
a1e0cf80e8 Toolchain: Colorize the toolchain build script's output 2021-04-18 10:55:25 +02:00
Gunnar Beutner
d7978a3317 Toolchain: Enable -fexceptions and build a separate libstdc++ for the kernel
This enables building usermode programs with exception handling. It also
builds a libstdc++ without exception support for the kernel.

This is necessary because the libstdc++ that gets built is different
when exceptions are enabled. Using the same library binary would
require extensive stubs for exception-related functionality in the
kernel.
2021-04-18 10:55:25 +02:00
Panagiotis Vasilopoulos
3b6454b9c2 Toolchain: Updated QEMU from 5.2.0 to 6.0.0-rc3 2021-04-17 16:11:42 +02:00
Gunnar Beutner
7edfe02dff Toolchain: Fix cmake definition for CMAKE_DL_LIBS
We don't have libdl (yet) so update the CMAKE_DL_LIBRARY
definition to match.
2021-04-17 11:40:56 +02:00
Gunnar Beutner
c9d5358685 Ports: Make sure ports are installed into /usr/local 2021-04-16 19:04:24 +02:00
Gunnar Beutner
594d480391 Toolchain+Ports: Move the CMake toolchain file into a subdirectory 2021-04-16 19:04:24 +02:00
Gunnar Beutner
c6c1e2037b Toolchain: Add platform definition for CMake
This also ensures that pkg-config finds packages in /usr/local
and changes the install prefix to /usr/local.
2021-04-16 19:04:24 +02:00
Gunnar Beutner
50e4cad4a0 Toolchain+LibC: Don't link LibC against crt0
Instead GCC should be used to automatically link against crt0
and crt0_shared depending on the type of object file that is being
built.

Unfortunately this requires a rebuild of the toolchain as well
as everything that has been built with the old GCC.
2021-04-16 17:56:12 +02:00
Brian Gianforcaro
73fab93ef5 Toolchain: Update to the latest gcc release 10.3.0
10.3.0 is a bugfix release, with 178 bugs fixed.
See: https://gcc.gnu.org/bugzilla/buglist.cgi?bug_status=RESOLVED&resolution=FIXED&target_milestone=10.3
2021-04-14 21:49:54 +02:00
Brian Gianforcaro
5d59affc18 Toolchain: Update binutils to version to latest 2.36.1 release
This updates binutils to the latest release. It looks like some
lines where shuffled around, so I had to fixup out patch to
apply cleanly again.
2021-04-14 21:49:54 +02:00
Gunnar Beutner
e95cb252be Toolchain: Install system headers before building GCC
GCC determines whether the system's <limits.h> header is usable
and installs a different version of its own <limits.h> header
depending on whether the system header file exists.

If the system header is missing GCC's <limits.h> header does not
include the system header via #include_next.

For this to work we need to install LibC's headers before
attempting to build GCC.

Also, re-running BuildIt.sh "hides" this problem because at that
point the sysroot directory also already has a <limits.h> header
file from the previous build.
2021-04-12 22:37:34 +02:00
Ben Wiederhake
100ce8fc1f Toolchain: Always strip compiler, not just on CI 2021-03-13 22:42:20 +01:00
Linus Groh
f45e16a6a2 Toolchain: Update build paths in CMakeToolchain.txt 2021-03-11 21:01:29 +01:00
Ben Wiederhake
c436c3e13d Toolchain: Remove unnecessary file from commit
This file is never read by gcc, and the filename looks like it was never meant to be added.
2021-03-11 12:32:53 +01:00