Commit Graph

373 Commits

Author SHA1 Message Date
Gunnar Beutner
4aaf6386ae Meta: Only allow sha256 and sig for auth_type in ports 2021-04-25 13:30:18 +02:00
Andrew Kaster
35c0a6c54d AK+Userland: Move AK/TestSuite.h into LibTest and rework Tests' CMake
As many macros as possible are moved to Macros.h, while the
macros to create a test case are moved to TestCase.h. TestCase is now
the only user-facing header for creating a test case. TestSuite and its
helpers have moved into a .cpp file. Instead of requiring a TEST_MAIN
macro to be instantiated into the test file, a TestMain.cpp file is
provided instead that will be linked against each test. This has the
side effect that, if we wanted to have test cases split across multiple
files, it's as simple as adding them all to the same executable.

The test main should be portable to kernel mode as well, so if
there's a set of tests that should be run in self-test mode in kernel
space, we can accomodate that.

A new serenity_test CMake function streamlines adding a new test with
arguments for the test source file, subdirectory under /usr/Tests to
install the test application and an optional list of libraries to link
against the test application. To accomodate future test where the
provided TestMain.cpp is not suitable (e.g. test-js), a CUSTOM_MAIN
parameter can be passed to the function to not link against the
boilerplate main function.
2021-04-25 09:36:49 +02:00
Gunnar Beutner
511ffa8d68 Meta: Support using rsync to install the root filesystem
Using rsync is significantly faster when we're using an existing
image because we only have to copy files which are new or have
been updated. This cuts down the image creation time to about
a second or two assuming an old image exists.
2021-04-23 22:34:05 +02:00
Gunnar Beutner
479905be6c Meta: Re-use existing disk image where possible
This adds support for re-using and re-sizing existing disk images.
Disk images are checked with e2fsck prior to re-use and a new disk
image is automatically created when that check fails.
2021-04-23 22:34:05 +02:00
Gunnar Beutner
b3db01e20e Ports: Detect more types of errors in the AvailablePorts.md file
This adds support for detecting incorrect version numbers and links
in the ports list.

Also, unlike before it doesn't parse the package.sh script but executes
it instead which allows us to detect syntax errors.
2021-04-23 16:11:48 +02:00
Linus Groh
43f56ced73 Meta: Remove sync-local.sh move warning from build-root-filesystem.sh 2021-04-22 23:08:20 +02:00
Brian Gianforcaro
1682f0b760 Everything: Move to SPDX license identifiers in all files.
SPDX License Identifiers are a more compact / standardized
way of representing file license information.

See: https://spdx.dev/resources/use/#identifiers

This was done with the `ambr` search and replace tool.

 ambr --no-parent-ignore --key-from-file --rep-from-file key.txt rep.txt *
2021-04-22 11:22:27 +02:00
Linus Groh
696f23d7a0 Meta: Add basic Zsh completions for serenity.sh
This patch adds a basic Zsh completion script for the commands and
targets provided by Meta/serenity.sh. There's some room for improvement
here, e.g. we could provide completion for available CMake targets -
currently completion stops after serenity.sh <command> <target>.

You can enable it by adding this to your .zshrc before completions are
loaded:

    fpath=($SERENITY_SOURCE_DIR/Meta/ShellCompletions/zsh $fpath)
2021-04-22 09:55:48 +02:00
Gunnar Beutner
232e64c10a Meta: Add GDB pretty printers 2021-04-22 00:45:43 +02:00
Ali Mohammad Pur
468ac11f29 Meta: Add an option to precompile some very common AK headers
Until we get the goodness that C++ modules are supposed to be, let's try
to shave off some parse time using precompiled headers.
This commit only adds some very common AK headers, only to binaries,
libraries and the kernel (tests are not covered due to incompatibility
with AK/TestSuite.h).
This option is on by default, but can be disabled by passing
`-DPRECOMPILE_COMMON_HEADERS=OFF` to cmake, which will disable all
header precompilations.
This makes the build about 30 seconds faster on my machine (about 7%).
2021-04-21 14:29:46 +02:00
Marco Biscaro
8124719c3d Tests: Reorganize LibCompress unit tests
Move LibCompress unit tests to LibCompress/Tests directory and register
them with CMake's add_test. This allows us to run these tests with
ninja test instead of running a separate executable.

Also split the existing tests in 3 test files that better follow the
source code structure (inspired by AK tests).
2021-04-21 08:00:32 +02:00
Timothy Flynn
110cd98c0a Lagom: Add sql utility and LibSQL unit tests 2021-04-20 18:28:34 +02:00
Timothy Flynn
55b7b8e930 AK+Meta: Add a debug option for LibSQL 2021-04-20 18:28:34 +02:00
Gunnar Beutner
2d62cf8b79 Ports: Update the lint-ports.py script to ignore missing auth_type for SerenityOS repositories 2021-04-20 17:04:51 +02:00
Panagiotis Vasilopoulos
e45e0eeb47 Everywhere: Replace SERENITY_ROOT with SERENITY_SOURCE_DIR 2021-04-20 15:27:52 +02:00
Brian Gianforcaro
8f6a5a1c39 Meta: Add a rebuild-world command to serenity.sh
This was brought up as something that would be useful by
`RealKC` on the discord, and I happened to agree that it
would be useful. Especially given the abundance of Toolchain
changes recently.
2021-04-20 08:10:21 +02:00
Brian Gianforcaro
c195ee6fd2 Meta: Add allow-list for icon size detection in CMake
The only icons we are currently warning about are designed
and rendered as small icons intentionally, as their only use
is in desktop applets, and thus are exempt to this rule.

This reduces build spam back down to a minimum.
I should have just done this in the first place, back in #4729
2021-04-20 08:10:21 +02:00
Brendan Coles
3c4dc5cb11 Meta: Remove references to Userland/Libraries/LibELF/exec_elf.h 2021-04-19 09:19:31 +02:00
FalseHonesty
969c73660b Meta: Mark x86 debug register macros as false positives for CI 2021-04-18 17:02:40 +02:00
Linus Groh
a48d365046 Meta: Replace /bin/env with /usr/bin/envin serenity.sh shebang
This is more portable.
2021-04-18 14:01:52 +02:00
Gunnar Beutner
aa254ffd66 Meta: Update check-style.sh to ignore CVS $id$ tags
Userlands/Libraries/LibC/sys/exec_elf.h has an $id$ tag
which fails this lint check.
2021-04-18 10:55:25 +02:00
Gunnar Beutner
6cb28ecee8 LibC+LibELF: Implement support for the dl_iterate_phdr helper
This helper is used by libgcc_s to figure out where the .eh_frame sections
are located for all loaded shared objects.
2021-04-18 10:55:25 +02:00
Linus Groh
2ac9790883 Meta: Remove stray newline from serenity.sh help text 2021-04-17 19:16:19 +02:00
Idan Horowitz
e4d9fa914e Kernel: Add base support for VirtRNG (VirtIO based Hardware RNG)
This is a very basic implementation that only requests 4096 bytes
of entropy from the host once, but its still high quality entropy
so it should be a good fix for #4490 (boot-time entropy starvation)
for virtualized environments.

Co-authored-by: Sahan <sahan.h.fernando@gmail.com>
2021-04-17 10:21:23 +02:00
Idan Horowitz
d1f7a2f9a5 Kernel: Finish base implementation of VirtQueues
This commit includes a lot of small changes and additions needed to
finalize the base implementation of VirtIOQueues and VirtDevices:
* The device specific driver implementation now has to handle setting
up the queues it needs before letting the base device class know it
finised initialization
* Supplying buffers to VirtQueues is now done via ScatterGatherLists
instead of arbitary buffer pointers - this ensures the pointers are
physical and allows us to follow the specification in regards to the
requirement that individual descriptors must point to physically
contiguous buffers. This can be further improved in the future by
implementating support for the Indirect-Descriptors feature (as
defined by the specification) to reduce descriptor usage for very
fragmented buffers.
* When supplying buffers to a VirtQueue the driver must supply a
(temporarily-)unique token (usually the supplied buffer's virtual
address) to ensure the driver can discern which buffer has finished
processing by the device in the case in which the device does not
offer the F_IN_ORDER feature.
* Device drivers now handle queue updates (supplied buffers being
returned from the device) by implementing a single pure virtual
method instead of setting a seperate callback for each queue
* Two new VirtQueue methods were added to allow the device driver
to either discard or get used/returned buffers from the device by
cleanly removing them off the descriptor chain (This also allows
the VirtQueue implementation to reuse those freed descriptors)

This also includes the necessary changes to the VirtIOConsole
implementation to match these interface changes.

Co-authored-by: Sahan <sahan.h.fernando@gmail.com>
2021-04-17 10:21:23 +02:00
Idan Horowitz
ecfa7cb824 Kernel: Implement a naive version of virtconsole by memcpying to physical page
This patch allocates a physical page for each of the virtqueues and
memcpys to it when receiving a buffer to get a physical, aligned
contiguous buffer as required by the virtio specification.

Co-authored-by: Sahan <sahan.h.fernando@gmail.com>
2021-04-17 10:21:23 +02:00
Idan Horowitz
62303d46d1 Kernel: Add base support for VirtIO devices
Based on pull #3236 by tomuta, this adds helper methods for generic
device initialization, and partily-broken virtqueue helper methods

Co-authored-by: Tom <tomut@yahoo.com>
Co-authored-by: Sahan <sahan.h.fernando@gmail.com>
2021-04-17 10:21:23 +02:00
Brian Gianforcaro
253114e3b1 Meta: Allow serenity.sh to be used outside of assumed working directory
This change fixes two bugs:

1) If you run `serenity.sh` outside of your serenity git clone, the
   `get_top_dir()` function won't be able to auto detect the serenity
    root dir and we'll error out. By allowing the script to use an
    existing $SERENITY_ROOT vlaue if it exists, we can solve
    this problem.

2) If tried to run a command which ends up calling `build_toolchain()`
   and you were outside of the root of the project, it should just
   fail. Fix this by utilizing `$SERENITY_ROOT`.
2021-04-17 09:27:19 +02:00
Brendan Coles
bb6bb7bc5e Meta: lint-ports: Check package files for required properties 2021-04-16 21:14:00 +02:00
Brian Gianforcaro
265e155181 Meta: Use basename for serenity.sh help message
I have this symlinked into ~/bin, when looking at the help/usage
it would previously print the fully qualified path to the script
making the help very difficult to read.
2021-04-15 17:47:09 +02:00
Peter Elliott
938924f36d Meta: Add install-ports CMake target
install-ports copys the necessary files from Ports/ to /usr/Ports. Also
refactor the compiler and destiation variables from .port_include.sh
into .hosted_defs.sh. .hosted_defs.sh does not exists when ports are
built in serenity
2021-04-12 14:06:24 +02:00
Andreas Kling
251018676f Everywhere: Update references from ReadMe.md => README.md 2021-04-11 10:52:25 +02:00
Andreas Kling
19987a9112 Meta: Update the screenshot in the ReadMe :^)
It's been almost a year since the last one, and things look a bit
different so let's add a new screenshot.
2021-04-04 09:57:36 +02:00
Liav A
8e3e3a71cb Kernel: Introduce a new HID subsystem
The end goal of this commit is to allow to boot on bare metal with no
PS/2 device connected to the system. It turned out that the original
code relied on the existence of the PS/2 keyboard, so VirtualConsole
called it even though ACPI indicated the there's no i8042 controller on
my real machine because I didn't plug any PS/2 device.
The code is much more flexible, so adding HID support for other type of
hardware (e.g. USB HID) could be much simpler.

Briefly describing the change, we have a new singleton called
HIDManagement, which is responsible to initialize the i8042 controller
if exists, and to enumerate its devices. I also abstracted a bit
things, so now every Human interface device is represented with the
HIDDevice class. Then, there are 2 types of it - the MouseDevice and
KeyboardDevice classes; both are responsible to handle the interface in
the DevFS.

PS2KeyboardDevice, PS2MouseDevice and VMWareMouseDevice classes are
responsible for handling the hardware-specific interface they are
assigned to. Therefore, they are inheriting from the IRQHandler class.
2021-04-03 11:57:23 +02:00
Brendan Coles
666aeecaa2 Meta: Resolve some pylint violations in Python lint scripts
Resolves:

* all: consider-using-sys-exit
* all: wrong-import-order
* all: TODO: Require that a few keys are set? (fixme)
* some: missing-function-docstring
* some: line-too-long
2021-04-02 23:07:58 +02:00
Linus Groh
f418115f1b LibJS: Add initial support for Promises
Almost a year after first working on this, it's finally done: an
implementation of Promises for LibJS! :^)

The core functionality is working and closely following the spec [1].
I mostly took the pseudo code and transformed it into C++ - if you read
and understand it, you will know how the spec implements Promises; and
if you read the spec first, the code will look very familiar.

Implemented functions are:

- Promise() constructor
- Promise.prototype.then()
- Promise.prototype.catch()
- Promise.prototype.finally()
- Promise.resolve()
- Promise.reject()

For the tests I added a new function to test-js's global object,
runQueuedPromiseJobs(), which calls vm.run_queued_promise_jobs().
By design, queued jobs normally only run after the script was fully
executed, making it improssible to test handlers in individual test()
calls by default [2].

Subsequent commits include integrations into LibWeb and js(1) -
pretty-printing, running queued promise jobs when necessary.

This has an unusual amount of dbgln() statements, all hidden behind the
PROMISE_DEBUG flag - I'm leaving them in for now as they've been very
useful while debugging this, things can get quite complex with so many
asynchronously executed functions.

I've not extensively explored use of these APIs for promise-based
functionality in LibWeb (fetch(), Notification.requestPermission()
etc.), but we'll get there in due time.

[1]: https://tc39.es/ecma262/#sec-promise-objects
[2]: https://tc39.es/ecma262/#sec-jobs-and-job-queues
2021-04-02 10:47:40 +02:00
Idan Horowitz
3b201da473 Utilities: Add simple traceroute utility
This utility traces the route packets take to a user specified host.

QEMU user networking (the type of networking we currently have setup)
does not support sending "real" raw IPv4 packets, and as such we cant
specify a custom TTL value. As a result, this utility will only work
on real hardware, qemu setup with tun networking (requires root) and
other hypervisors that support bridged adapters (VirtualBox/VMWare).
2021-03-31 23:42:24 +02:00
Luke
1ff6f2b4b2 Lagom/Fuzzers: Add fuzzer for zip file parser 2021-03-27 16:28:34 +01:00
Linus Groh
ad4cc19ca0 Meta: Pass arguments to 'serenity.sh run lagom' executable 2021-03-24 20:38:05 +01:00
Andreas Kling
d20e3dbe8b LibGUI: Remove one ancient GMENU_DEBUG debug log 2021-03-23 20:37:39 +01:00
Itamar
4cb38f6dd8 CI: Don't fail check-symbols if symbol is defiend somewhere else 2021-03-19 22:55:53 +01:00
Andreas Kling
d792200a55 LibJS: Rename GlobalObject::initialize() => initialize_global_object()
This function was shadowing Object::initialize() which cannot be called
on global objects and has a different set of parameters.
2021-03-17 16:53:35 +01:00
Luke
c983e42e8c Lagom/Fuzzers: Add fuzzers for the new Gzip and Deflate compressors 2021-03-14 11:28:12 +01:00
Luke
d27e6f0961 Lagom/Fuzzers: Add fuzzers for Latin 1, Latin 2 and UTF16-BE
No fuzzer for UTF-8 as it (currently) just returns the input.
2021-03-14 11:27:52 +01:00
Jean-Baptiste Boric
800dca3834 Kernel: Implement triply indirect block support in Ext2FSInode 2021-03-13 09:27:18 +01:00
Linus Groh
89602c6fc0 Meta: Support 'serenity.sh gdb lagom' 2021-03-12 21:27:09 +01:00
Linus Groh
eb5b2b4b4f Lagom: Add 'test' command to serenity.sh 2021-03-12 21:27:09 +01:00
Linus Groh
04fc4a8134 Meta: Make 'serenity.sh run lagom' run Lagom executables
Running the tests will be moved to a separate test command which can
then leverage the availability of different targets and run either unit
tests on the host or the image in QEMU in self-test mode. :^)
2021-03-12 21:27:09 +01:00
Linus Groh
1c85ce0324 Meta: Don't depend on toolchain for lagom target in serenity.sh
- Only call ensure_toolchain for non-lagom targets
- Use host addr2line, we can't expect the i686 toolchain's addr2line to
  support the host's binary executable format
- Don't export SERENITY_ARCH and TOOLCHAIN_DIR, don't need them anymore
2021-03-12 21:27:09 +01:00
Linus Groh
cab2982feb Meta: Add lagom_unsupported helper function to serenity.sh 2021-03-12 21:27:09 +01:00