LadybirdBrowser/ladybird
1
1
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2024-11-11 01:06:01 +03:00
Code Issues Packages Projects Releases Wiki Activity
37bfaf4e7c
ladybird/Kernel/Syscalls/futex.cpp

381 lines
14 KiB
C++
Raw Normal View History

Kernel: Move syscall implementations out of Process.cpp This is something I've been meaning to do for a long time, and here we finally go. This patch moves all sys$foo functions out of Process.cpp and into files in Kernel/Syscalls/. It's not exactly one syscall per file (although it could be, but I got a bit tired of the repetitive work here..) This makes hacking on individual syscalls a lot less painful since you don't have to rebuild nearly as much code every time. I'm also hopeful that this makes it easier to understand individual syscalls. :^)
2020-07-31 00:38:15 +03:00
/*
* Copyright (c) 2018-2020, Andreas Kling <kling@serenityos.org>
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* 1. Redistributions of source code must retain the above copyright notice, this
* list of conditions and the following disclaimer.
*
* 2. Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
* SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
* CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
* OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
Everywhere: Deprecate dbg().
2021-01-23 17:30:51 +03:00
#include <AK/Debug.h>
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
#include <AK/Singleton.h>
Kernel: Move syscall implementations out of Process.cpp This is something I've been meaning to do for a long time, and here we finally go. This patch moves all sys$foo functions out of Process.cpp and into files in Kernel/Syscalls/. It's not exactly one syscall per file (although it could be, but I got a bit tired of the repetitive work here..) This makes hacking on individual syscalls a lot less painful since you don't have to rebuild nearly as much code every time. I'm also hopeful that this makes it easier to understand individual syscalls. :^)
2020-07-31 00:38:15 +03:00
#include <AK/Time.h>
#include <Kernel/Process.h>
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
#include <Kernel/VM/MemoryManager.h>
Kernel: Move syscall implementations out of Process.cpp This is something I've been meaning to do for a long time, and here we finally go. This patch moves all sys$foo functions out of Process.cpp and into files in Kernel/Syscalls/. It's not exactly one syscall per file (although it could be, but I got a bit tired of the repetitive work here..) This makes hacking on individual syscalls a lot less painful since you don't have to rebuild nearly as much code every time. I'm also hopeful that this makes it easier to understand individual syscalls. :^)
2020-07-31 00:38:15 +03:00
namespace Kernel {
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
static SpinLock<u8> g_global_futex_lock;
static AK::Singleton<HashMap<VMObject*, FutexQueues>> g_global_futex_queues;
FutexQueue::FutexQueue(FlatPtr user_address_or_offset, VMObject* vmobject)
: m_user_address_or_offset(user_address_or_offset)
, m_is_global(vmobject != nullptr)
{
Everywhere: Debug macros instead of constexpr. This was done with the following script: find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/dbgln<debug_([a-z_]+)>/dbgln<\U\1_DEBUG>/' {} \; find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/if constexpr \(debug_([a-z0-9_]+)/if constexpr \(\U\1_DEBUG/' {} \;
2021-01-24 01:59:27 +03:00
dbgln<FUTEX_DEBUG>("Futex @ {}{}",
Everywhere: Deprecate dbg().
2021-01-23 17:30:51 +03:00
this,
m_is_global ? " (global)" : " (local)");
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
if (m_is_global) {
// Only register for global futexes
m_vmobject = vmobject->make_weak_ptr();
vmobject->register_on_deleted_handler(*this);
}
}
FutexQueue::~FutexQueue()
{
if (m_is_global) {
if (auto vmobject = m_vmobject.strong_ref())
vmobject->unregister_on_deleted_handler(*this);
}
Everywhere: Debug macros instead of constexpr. This was done with the following script: find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/dbgln<debug_([a-z_]+)>/dbgln<\U\1_DEBUG>/' {} \; find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/if constexpr \(debug_([a-z0-9_]+)/if constexpr \(\U\1_DEBUG/' {} \;
2021-01-24 01:59:27 +03:00
dbgln<FUTEX_DEBUG>("~Futex @ {}{}",
Everywhere: Deprecate dbg().
2021-01-23 17:30:51 +03:00
this,
m_is_global ? " (global)" : " (local)");
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
}
void FutexQueue::vmobject_deleted(VMObject& vmobject)
{
ASSERT(m_is_global); // If we got called we must be a global futex
// Because we're taking ourselves out of the global queue, we need
// to make sure we have at last a reference until we're done
NonnullRefPtr<FutexQueue> own_ref(*this);
Everywhere: Debug macros instead of constexpr. This was done with the following script: find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/dbgln<debug_([a-z_]+)>/dbgln<\U\1_DEBUG>/' {} \; find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/if constexpr \(debug_([a-z0-9_]+)/if constexpr \(\U\1_DEBUG/' {} \;
2021-01-24 01:59:27 +03:00
dbgln<FUTEX_DEBUG>("Futex::vmobject_deleted @ {}{}",
Everywhere: Deprecate dbg().
2021-01-23 17:30:51 +03:00
this,
m_is_global ? " (global)" : " (local)");
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
// Because this is called from the VMObject's destructor, getting a
// strong_ref in this function is unsafe!
m_vmobject = nullptr; // Just to be safe...
{
ScopedSpinLock lock(g_global_futex_lock);
g_global_futex_queues->remove(&vmobject);
}
bool did_wake_all;
auto wake_count = wake_all(did_wake_all);
Everywhere: Deprecate dbg().
2021-01-23 17:30:51 +03:00
Everywhere: Debug macros instead of constexpr. This was done with the following script: find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/dbgln<debug_([a-z_]+)>/dbgln<\U\1_DEBUG>/' {} \; find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/if constexpr \(debug_([a-z0-9_]+)/if constexpr \(\U\1_DEBUG/' {} \;
2021-01-24 01:59:27 +03:00
if constexpr (FUTEX_DEBUG) {
Everywhere: Deprecate dbg().
2021-01-23 17:30:51 +03:00
if (wake_count > 0)
dbgln("Futex @ {} unblocked {} waiters due to vmobject free", this, wake_count);
}
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
ASSERT(did_wake_all); // No one should be left behind...
}
void Process::clear_futex_queues_on_exec()
Kernel: Move syscall implementations out of Process.cpp This is something I've been meaning to do for a long time, and here we finally go. This patch moves all sys$foo functions out of Process.cpp and into files in Kernel/Syscalls/. It's not exactly one syscall per file (although it could be, but I got a bit tired of the repetitive work here..) This makes hacking on individual syscalls a lot less painful since you don't have to rebuild nearly as much code every time. I'm also hopeful that this makes it easier to understand individual syscalls. :^)
2020-07-31 00:38:15 +03:00
{
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
ScopedSpinLock lock(m_futex_lock);
for (auto& it : m_futex_queues) {
bool did_wake_all;
it.value->wake_all(did_wake_all);
ASSERT(did_wake_all); // No one should be left behind...
}
m_futex_queues.clear();
Kernel: Move syscall implementations out of Process.cpp This is something I've been meaning to do for a long time, and here we finally go. This patch moves all sys$foo functions out of Process.cpp and into files in Kernel/Syscalls/. It's not exactly one syscall per file (although it could be, but I got a bit tired of the repetitive work here..) This makes hacking on individual syscalls a lot less painful since you don't have to rebuild nearly as much code every time. I'm also hopeful that this makes it easier to understand individual syscalls. :^)
2020-07-31 00:38:15 +03:00
}
Kernel: Use Userspace<T> for the futex syscall Utilizie Userspace<T> for the syscall argument itself, as well as internally in the SC_futex_params struct. We were double validating the SC_futex_params.timeout validation, that was removed as well.
2020-08-05 09:53:23 +03:00
int Process::sys$futex(Userspace<const Syscall::SC_futex_params*> user_params)
Kernel: Move syscall implementations out of Process.cpp This is something I've been meaning to do for a long time, and here we finally go. This patch moves all sys$foo functions out of Process.cpp and into files in Kernel/Syscalls/. It's not exactly one syscall per file (although it could be, but I got a bit tired of the repetitive work here..) This makes hacking on individual syscalls a lot less painful since you don't have to rebuild nearly as much code every time. I'm also hopeful that this makes it easier to understand individual syscalls. :^)
2020-07-31 00:38:15 +03:00
{
REQUIRE_PROMISE(thread);
Syscall::SC_futex_params params;
Kernel: Make copy_to/from_user safe and remove unnecessary checks Since the CPU already does almost all necessary validation steps for us, we don't really need to attempt to do this. Doing it ourselves doesn't really work very reliably, because we'd have to account for other processors modifying virtual memory, and we'd have to account for e.g. pages not being able to be allocated due to insufficient resources. So change the copy_to/from_user (and associated helper functions) to use the new safe_memcpy, which will return whether it succeeded or not. The only manual validation step needed (which the CPU can't perform for us) is making sure the pointers provided by user mode aren't pointing to kernel mappings. To make it easier to read/write from/to either kernel or user mode data add the UserOrKernelBuffer helper class, which will internally either use copy_from/to_user or directly memcpy, or pass the data through directly using a temporary buffer on the stack. Last but not least we need to keep syscall params trivial as we need to copy them from/to user mode using copy_from/to_user.
2020-09-12 06:11:07 +03:00
if (!copy_from_user(&params, user_params))
Kernel: Move syscall implementations out of Process.cpp This is something I've been meaning to do for a long time, and here we finally go. This patch moves all sys$foo functions out of Process.cpp and into files in Kernel/Syscalls/. It's not exactly one syscall per file (although it could be, but I got a bit tired of the repetitive work here..) This makes hacking on individual syscalls a lot less painful since you don't have to rebuild nearly as much code every time. I'm also hopeful that this makes it easier to understand individual syscalls. :^)
2020-07-31 00:38:15 +03:00
return -EFAULT;
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
Thread::BlockTimeout timeout;
u32 cmd = params.futex_op & FUTEX_CMD_MASK;
switch (cmd) {
case FUTEX_WAIT:
case FUTEX_WAIT_BITSET:
case FUTEX_REQUEUE:
case FUTEX_CMP_REQUEUE: {
if (params.timeout) {
timespec ts_stimeout { 0, 0 };
if (!copy_from_user(&ts_stimeout, params.timeout))
return -EFAULT;
clockid_t clock_id = (params.futex_op & FUTEX_CLOCK_REALTIME) ? CLOCK_REALTIME_COARSE : CLOCK_MONOTONIC_COARSE;
bool is_absolute = cmd != FUTEX_WAIT;
timeout = Thread::BlockTimeout(is_absolute, &ts_stimeout, nullptr, clock_id);
}
if (cmd == FUTEX_WAIT_BITSET && params.val3 == FUTEX_BITSET_MATCH_ANY)
cmd = FUTEX_WAIT;
break;
case FUTEX_WAKE_BITSET:
if (params.val3 == FUTEX_BITSET_MATCH_ANY)
cmd = FUTEX_WAKE;
break;
}
}
bool is_private = (params.futex_op & FUTEX_PRIVATE_FLAG) != 0;
auto& queue_lock = is_private ? m_futex_lock : g_global_futex_lock;
auto user_address_or_offset = FlatPtr(params.userspace_address);
auto user_address_or_offset2 = FlatPtr(params.userspace_address2);
// If this is a global lock, look up the underlying VMObject *before*
// acquiring the queue lock
RefPtr<VMObject> vmobject, vmobject2;
if (!is_private) {
if (!Kernel::is_user_range(VirtualAddress(user_address_or_offset), sizeof(u32)))
Kernel: Make copy_to/from_user safe and remove unnecessary checks Since the CPU already does almost all necessary validation steps for us, we don't really need to attempt to do this. Doing it ourselves doesn't really work very reliably, because we'd have to account for other processors modifying virtual memory, and we'd have to account for e.g. pages not being able to be allocated due to insufficient resources. So change the copy_to/from_user (and associated helper functions) to use the new safe_memcpy, which will return whether it succeeded or not. The only manual validation step needed (which the CPU can't perform for us) is making sure the pointers provided by user mode aren't pointing to kernel mappings. To make it easier to read/write from/to either kernel or user mode data add the UserOrKernelBuffer helper class, which will internally either use copy_from/to_user or directly memcpy, or pass the data through directly using a temporary buffer on the stack. Last but not least we need to keep syscall params trivial as we need to copy them from/to user mode using copy_from/to_user.
2020-09-12 06:11:07 +03:00
return -EFAULT;
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
auto region = MM.find_region_from_vaddr(*Process::current(), VirtualAddress(user_address_or_offset));
if (!region)
return -EFAULT;
vmobject = region->vmobject();
user_address_or_offset = region->offset_in_vmobject_from_vaddr(VirtualAddress(user_address_or_offset));
Kernel: Move syscall implementations out of Process.cpp This is something I've been meaning to do for a long time, and here we finally go. This patch moves all sys$foo functions out of Process.cpp and into files in Kernel/Syscalls/. It's not exactly one syscall per file (although it could be, but I got a bit tired of the repetitive work here..) This makes hacking on individual syscalls a lot less painful since you don't have to rebuild nearly as much code every time. I'm also hopeful that this makes it easier to understand individual syscalls. :^)
2020-07-31 00:38:15 +03:00
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
switch (cmd) {
case FUTEX_REQUEUE:
case FUTEX_CMP_REQUEUE:
case FUTEX_WAKE_OP: {
if (!Kernel::is_user_range(VirtualAddress(user_address_or_offset2), sizeof(u32)))
return -EFAULT;
auto region2 = MM.find_region_from_vaddr(*Process::current(), VirtualAddress(user_address_or_offset2));
if (!region2)
Kernel: Move some time related code from Scheduler into TimeManagement Use the TimerQueue to expire blocking operations, which is one less thing the Scheduler needs to check on every iteration. Also, add a BlockTimeout class that will automatically handle relative or absolute timeouts as well as overriding timeouts (e.g. socket timeouts) more consistently. Also, rework the TimerQueue class to be able to fire events from any processor, which requires Timer to be RefCounted. Also allow creating id-less timers for use by blocking operations.
2020-11-15 21:58:19 +03:00
return -EFAULT;
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
vmobject2 = region2->vmobject();
user_address_or_offset2 = region->offset_in_vmobject_from_vaddr(VirtualAddress(user_address_or_offset2));
break;
}
}
}
auto find_global_futex_queues = [&](VMObject& vmobject, bool create_if_not_found) -> FutexQueues* {
auto& global_queues = *g_global_futex_queues;
auto it = global_queues.find(&vmobject);
if (it != global_queues.end())
return &it->value;
if (create_if_not_found) {
// TODO: is there a better way than setting and finding it again?
auto result = global_queues.set(&vmobject, {});
ASSERT(result == AK::HashSetResult::InsertedNewEntry);
it = global_queues.find(&vmobject);
ASSERT(it != global_queues.end());
return &it->value;
Kernel: Move syscall implementations out of Process.cpp This is something I've been meaning to do for a long time, and here we finally go. This patch moves all sys$foo functions out of Process.cpp and into files in Kernel/Syscalls/. It's not exactly one syscall per file (although it could be, but I got a bit tired of the repetitive work here..) This makes hacking on individual syscalls a lot less painful since you don't have to rebuild nearly as much code every time. I'm also hopeful that this makes it easier to understand individual syscalls. :^)
2020-07-31 00:38:15 +03:00
}
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
return nullptr;
};
Kernel: Move syscall implementations out of Process.cpp This is something I've been meaning to do for a long time, and here we finally go. This patch moves all sys$foo functions out of Process.cpp and into files in Kernel/Syscalls/. It's not exactly one syscall per file (although it could be, but I got a bit tired of the repetitive work here..) This makes hacking on individual syscalls a lot less painful since you don't have to rebuild nearly as much code every time. I'm also hopeful that this makes it easier to understand individual syscalls. :^)
2020-07-31 00:38:15 +03:00
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
auto find_futex_queue = [&](VMObject* vmobject, FlatPtr user_address_or_offset, bool create_if_not_found) -> RefPtr<FutexQueue> {
ASSERT(is_private || vmobject);
auto* queues = is_private ? &m_futex_queues : find_global_futex_queues(*vmobject, create_if_not_found);
if (!queues)
return {};
auto it = queues->find(user_address_or_offset);
if (it != queues->end())
return it->value;
if (create_if_not_found) {
auto futex_queue = adopt(*new FutexQueue(user_address_or_offset, vmobject));
auto result = queues->set(user_address_or_offset, futex_queue);
ASSERT(result == AK::HashSetResult::InsertedNewEntry);
return futex_queue;
}
return {};
};
auto remove_futex_queue = [&](VMObject* vmobject, FlatPtr user_address_or_offset) {
auto* queues = is_private ? &m_futex_queues : find_global_futex_queues(*vmobject, false);
if (queues) {
queues->remove(user_address_or_offset);
if (!is_private && queues->is_empty())
g_global_futex_queues->remove(vmobject);
}
};
auto do_wake = [&](VMObject* vmobject, FlatPtr user_address_or_offset, u32 count, Optional<u32> bitmask) -> int {
if (count == 0)
return 0;
auto futex_queue = find_futex_queue(vmobject, user_address_or_offset, false);
if (!futex_queue)
return 0;
bool is_empty;
u32 woke_count = futex_queue->wake_n(count, bitmask, is_empty);
if (is_empty) {
// If there are no more waiters, we want to get rid of the futex!
remove_futex_queue(vmobject, user_address_or_offset);
}
return (int)woke_count;
};
ScopedSpinLock lock(queue_lock);
auto do_wait = [&](u32 bitset) -> int {
auto user_value = user_atomic_load_relaxed(params.userspace_address);
if (!user_value.has_value())
return -EFAULT;
if (user_value.value() != params.val) {
Everywhere: Deprecate dbg().
2021-01-23 17:30:51 +03:00
dbgln("futex wait: EAGAIN. user value: {:p} @ {:p} != val: {}", user_value.value(), params.userspace_address, params.val);
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
return -EAGAIN;
}
atomic_thread_fence(AK::MemoryOrder::memory_order_acquire);
auto futex_queue = find_futex_queue(vmobject.ptr(), user_address_or_offset, true);
ASSERT(futex_queue);
// We need to release the lock before blocking. But we have a reference
// to the FutexQueue so that we can keep it alive.
lock.unlock();
Thread::BlockResult block_result = futex_queue->wait_on(timeout, bitset);
lock.lock();
if (futex_queue->is_empty()) {
// If there are no more waiters, we want to get rid of the futex!
remove_futex_queue(vmobject, user_address_or_offset);
}
if (block_result == Thread::BlockResult::InterruptedByTimeout) {
Kernel: Move syscall implementations out of Process.cpp This is something I've been meaning to do for a long time, and here we finally go. This patch moves all sys$foo functions out of Process.cpp and into files in Kernel/Syscalls/. It's not exactly one syscall per file (although it could be, but I got a bit tired of the repetitive work here..) This makes hacking on individual syscalls a lot less painful since you don't have to rebuild nearly as much code every time. I'm also hopeful that this makes it easier to understand individual syscalls. :^)
2020-07-31 00:38:15 +03:00
return -ETIMEDOUT;
}
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
return 0;
};
auto do_requeue = [&](Optional<u32> val3) -> int {
auto user_value = user_atomic_load_relaxed(params.userspace_address);
if (!user_value.has_value())
return -EFAULT;
if (val3.has_value() && val3.value() != user_value.value())
return -EAGAIN;
atomic_thread_fence(AK::MemoryOrder::memory_order_acquire);
int woken_or_requeued = 0;
if (auto futex_queue = find_futex_queue(vmobject.ptr(), user_address_or_offset, false)) {
RefPtr<FutexQueue> target_futex_queue;
bool is_empty, is_target_empty;
woken_or_requeued = futex_queue->wake_n_requeue(
params.val, [&]() -> FutexQueue* {
// NOTE: futex_queue's lock is being held while this callback is called
// The reason we're doing this in a callback is that we don't want to always
// create a target queue, only if we actually have anything to move to it!
target_futex_queue = find_futex_queue(vmobject2.ptr(), user_address_or_offset2, true);
return target_futex_queue.ptr();
},
params.val2, is_empty, is_target_empty);
if (is_empty)
remove_futex_queue(vmobject, user_address_or_offset);
if (is_target_empty && target_futex_queue)
remove_futex_queue(vmobject2, user_address_or_offset2);
}
return woken_or_requeued;
};
switch (cmd) {
case FUTEX_WAIT:
return do_wait(0);
Kernel: Move syscall implementations out of Process.cpp This is something I've been meaning to do for a long time, and here we finally go. This patch moves all sys$foo functions out of Process.cpp and into files in Kernel/Syscalls/. It's not exactly one syscall per file (although it could be, but I got a bit tired of the repetitive work here..) This makes hacking on individual syscalls a lot less painful since you don't have to rebuild nearly as much code every time. I'm also hopeful that this makes it easier to understand individual syscalls. :^)
2020-07-31 00:38:15 +03:00
case FUTEX_WAKE:
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
return do_wake(vmobject.ptr(), user_address_or_offset, params.val, {});
case FUTEX_WAKE_OP: {
Optional<u32> oldval;
u32 op_arg = _FUTEX_OP_ARG(params.val3);
auto op = _FUTEX_OP(params.val3);
if (op & FUTEX_OP_ARG_SHIFT) {
op_arg = 1 << op_arg;
op &= FUTEX_OP_ARG_SHIFT;
Kernel: Move syscall implementations out of Process.cpp This is something I've been meaning to do for a long time, and here we finally go. This patch moves all sys$foo functions out of Process.cpp and into files in Kernel/Syscalls/. It's not exactly one syscall per file (although it could be, but I got a bit tired of the repetitive work here..) This makes hacking on individual syscalls a lot less painful since you don't have to rebuild nearly as much code every time. I'm also hopeful that this makes it easier to understand individual syscalls. :^)
2020-07-31 00:38:15 +03:00
}
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
atomic_thread_fence(AK::MemoryOrder::memory_order_release);
switch (op) {
case FUTEX_OP_SET:
oldval = user_atomic_exchange_relaxed(params.userspace_address2, op_arg);
break;
case FUTEX_OP_ADD:
oldval = user_atomic_fetch_add_relaxed(params.userspace_address2, op_arg);
break;
case FUTEX_OP_OR:
oldval = user_atomic_fetch_or_relaxed(params.userspace_address2, op_arg);
break;
case FUTEX_OP_ANDN:
oldval = user_atomic_fetch_and_not_relaxed(params.userspace_address2, op_arg);
break;
case FUTEX_OP_XOR:
oldval = user_atomic_fetch_xor_relaxed(params.userspace_address2, op_arg);
break;
default:
return -EINVAL;
}
if (!oldval.has_value())
return -EFAULT;
atomic_thread_fence(AK::MemoryOrder::memory_order_acquire);
int result = do_wake(vmobject.ptr(), user_address_or_offset, params.val, {});
if (params.val2 > 0) {
bool compare_result;
switch (_FUTEX_CMP(params.val3)) {
case FUTEX_OP_CMP_EQ:
compare_result = (oldval.value() == _FUTEX_CMP_ARG(params.val3));
break;
case FUTEX_OP_CMP_NE:
compare_result = (oldval.value() != _FUTEX_CMP_ARG(params.val3));
break;
case FUTEX_OP_CMP_LT:
compare_result = (oldval.value() < _FUTEX_CMP_ARG(params.val3));
break;
case FUTEX_OP_CMP_LE:
compare_result = (oldval.value() <= _FUTEX_CMP_ARG(params.val3));
break;
case FUTEX_OP_CMP_GT:
compare_result = (oldval.value() > _FUTEX_CMP_ARG(params.val3));
break;
case FUTEX_OP_CMP_GE:
compare_result = (oldval.value() >= _FUTEX_CMP_ARG(params.val3));
break;
default:
return -EINVAL;
}
if (compare_result)
result += do_wake(vmobject2.ptr(), user_address_or_offset2, params.val2, {});
}
return result;
Kernel: Move syscall implementations out of Process.cpp This is something I've been meaning to do for a long time, and here we finally go. This patch moves all sys$foo functions out of Process.cpp and into files in Kernel/Syscalls/. It's not exactly one syscall per file (although it could be, but I got a bit tired of the repetitive work here..) This makes hacking on individual syscalls a lot less painful since you don't have to rebuild nearly as much code every time. I'm also hopeful that this makes it easier to understand individual syscalls. :^)
2020-07-31 00:38:15 +03:00
}
Kernel: Some futex improvements This adds support for FUTEX_WAKE_OP, FUTEX_WAIT_BITSET, FUTEX_WAKE_BITSET, FUTEX_REQUEUE, and FUTEX_CMP_REQUEUE, as well well as global and private futex and absolute/relative timeouts against the appropriate clock. This also changes the implementation so that kernel resources are only used when a thread is blocked on a futex. Global futexes are implemented as offsets in VMObjects, so that different processes can share a futex against the same VMObject despite potentially being mapped at different virtual addresses.
2020-12-22 09:21:58 +03:00
case FUTEX_REQUEUE:
return do_requeue({});
case FUTEX_CMP_REQUEUE:
return do_requeue(params.val3);
case FUTEX_WAIT_BITSET:
ASSERT(params.val3 != FUTEX_BITSET_MATCH_ANY); // we should have turned it into FUTEX_WAIT
if (params.val3 == 0)
return -EINVAL;
return do_wait(params.val3);
case FUTEX_WAKE_BITSET:
ASSERT(params.val3 != FUTEX_BITSET_MATCH_ANY); // we should have turned it into FUTEX_WAKE
if (params.val3 == 0)
return -EINVAL;
return do_wake(vmobject.ptr(), user_address_or_offset, params.val, params.val3);
}
return -ENOSYS;
Kernel: Move syscall implementations out of Process.cpp This is something I've been meaning to do for a long time, and here we finally go. This patch moves all sys$foo functions out of Process.cpp and into files in Kernel/Syscalls/. It's not exactly one syscall per file (although it could be, but I got a bit tired of the repetitive work here..) This makes hacking on individual syscalls a lot less painful since you don't have to rebuild nearly as much code every time. I'm also hopeful that this makes it easier to understand individual syscalls. :^)
2020-07-31 00:38:15 +03:00
}
}
Reference in New Issue Copy Permalink