Summary: The same as < 500 references, except this time for targets with < 1600 references.
Reviewed By: yfeldblum
Differential Revision: D6540895
fbshipit-source-id: 40fa46c32abd6bc1c3c652a0396d6478b947f69b
Summary:
There's no technical reason to block an open() request until the data
load / materialization returns. This change returns immediately from
open() and then waits if necessary in a subsequent write() call.
Reviewed By: wez
Differential Revision: D6391486
fbshipit-source-id: 862f87e3c3a0d760bacb0f8ca7acc479037fec2f
Summary:
Replace the initLoggingGlogStyle() function with a more generic initLogging()
function that accepts a log config string to be parsed with parseLogConfig().
Reviewed By: bolinfest, yfeldblum
Differential Revision: D6342086
fbshipit-source-id: fb1bffd11f190b70e03e2ccbf2b30be08d655242
Summary:
Always call `umount2()` with both `MNT_FORCE` and `MNT_DETACH`. Without
`MNT_DETACH` I see unmount operations still fail or hang in some cases on
edenfs shutdown. This should hopefully help fully eliminate situations where
disconnected mount points are sometimes left behind when eden shuts down.
Reviewed By: bolinfest, chadaustin
Differential Revision: D6455971
fbshipit-source-id: 5abe456e17a33a0080ad94b4d315540bce0c2f82
Summary:
Add a method to LogHandler to return its current configuration. This will
make it possible to query the LoggerDB for its current configuration state.
Reviewed By: bolinfest
Differential Revision: D6200563
fbshipit-source-id: 2b8b9752bbeb26c8aac28d1a73b7e2312fd198c8
Summary:
There's a bug in some combination of Eden and FUSE where open(O_TRUNC)
followed by a sequence of writes over an existing file does not flush
the kernel's VFS page cache, which manifests as an mmap larger than
the file's size not zeroing the data beyond the file's size. These
tests attempt capture that use case, but they are fiddly.
Disabling ATOMIC_O_TRUNC seems to resolve the issue.
Reviewed By: wez
Differential Revision: D6430152
fbshipit-source-id: f7626e268e778ebab60c66322e0ce42bce746ae1
Summary:
The gtest macros in this file were moved to folly/test/TestUtils.h
Update everything to just use folly/test/TestUtils.h directly.
Reviewed By: chadaustin
Differential Revision: D6301759
fbshipit-source-id: 7f2841c12d5bea15376f782fb3bf3bfef16039c7
Summary:
Per discussion with bolinfest, this brings Eden in line with clang-format.
This diff was generated with `find . \( -iname '*.cpp' -o -iname '*.h' \) -exec bash -c "yes | arc lint {}" \;`
Reviewed By: bolinfest
Differential Revision: D6232695
fbshipit-source-id: d54942bf1c69b5b0dcd4df629f1f2d5538c9e28c
Summary:
In previous discussions, it has been pointed out that `folly::RequestContext::setContext` consumes considerable amount of CPU cycles in production environments. After some investigation, we thought that might be caused by looping over all `RequestData` instances can calling the virtual `onSet` and `onUnset` callback of them. Both the iteration and invoking virtual methods are not cheap.
As you can see from this change, most of the derived classes of `RequestData` don't override the `onSet` and `onUnset` methods. Mostly likely they are only used for per-Request tracking. So the natural idea is to skip those instances when iterating and avoid the unnecessary virtual method invoke.
I have explored the solution to dynamically examine if the `RequestData` instance added has `onSet` and `onUnset` method overridden. That is possible with GCC's PMF extension, but not [[ http://lists.llvm.org/pipermail/llvm-bugs/2015-July/041164.html | for Clang ]] and probably many other compilers. This definitely won't be very good for `folly`'s probability, even if we gate it by compiler flags.
Therefore, this Diff adds the `hasCallback` method to `RequestData` class indicating whether the instance would have `onSet` and `onUnset` overridden. To make it clear to users that they need to correctly override it in order for their `onSet` and `onUnset` callback to work, making it abstract so that user must override it to something and would aware of that.
Also made some improvements on documentation.
Reviewed By: myreg
Differential Revision: D6144049
fbshipit-source-id: 4c9fd72e9efaeb6763d55f63760eaf582ee4839e
Summary:
[Wangle] Dead Code: `wangle/concurrent/SerialExecutor.h`.
It was just a shim around `folly/executors/SerialExecutor.h` but the shim is no longer required.
Reviewed By: Orvid
Differential Revision: D6075932
fbshipit-source-id: a5df32c7a27f8c60427120325016890544ffceba
Summary:
This should help ensure that `eden shutdown` exits cleanly without resorting to
the use of `SIGKILL`. See D6080929 for context.
Note that D5367407 changed things so that we used `MNT_DETACH` by default, but
we've seen cases where `eden shutdown` fails to terminate within the default
timeout, so we're taking a more aggressive approach by switching to `MNT_FORCE`.
Reviewed By: wez
Differential Revision: D6085508
fbshipit-source-id: 807e4927eec640aed0c3f7d3d0d3119023ac7dd0
Summary:
D6003897 introduced a bug in the PrivHelperConn code where we declared a
ControlMsgBuffer object in a small if statement scope, attached it to a msghdr
object, and then used the msghdr outside of this if block, after the
ControlMsgBuffer had already been destroyed.
We probably could make this problem less likely by also adding a wrapper class
around `struct msghdr`, and making that wrapper class responsible for owning
any ControlMsgBuffer that needs to be sent with the message. Maybe it would be
worth doing that in a future diff, but for now this applies a simple fix of
just declaring the ControlMsgBuffer outside of the if block.
Reviewed By: bolinfest
Differential Revision: D6028339
fbshipit-source-id: 0e9c415edc165460c4453705602ddb85454594d6
Summary:
Set the O_CLOEXEC flag when opening /dev/fuse inside the privhelper process.
We already set O_CLOEXEC on this FD inside the main edenfs process when we
receive it from the privhelper, but we probably should set it inside the
privhelper too.
The privhelper process never forks or execs so this probably shouldn't really
matter, but it seems safer to always set it anyway.
Reviewed By: bolinfest, wez
Differential Revision: D6003898
fbshipit-source-id: 452572c626b34ce0c4cda9a16fb9b44ab9bf9fa8
Summary:
Add a new helper class for working with cmsghdr structs.
This updates the PrivHelperConn code to use this new class, and I will also use
this in the upcoming code for implementing graceful mount point takeover on
edenfs restart.
Reviewed By: bolinfest, wez
Differential Revision: D6003897
fbshipit-source-id: 634e9d8f1dc73010c9e9336872cf1270a344bdd2
Summary:
CodeMod: Change `#include`'s of `wangle/concurrent/GlobalExecutor.h` to use `folly`.
The file in `wangle/` is just a shim around the same file in `folly/executors/GlobalExecutor.h`. Just codemod all the `#include` sites.
Reviewed By: Orvid
Differential Revision: D5981467
fbshipit-source-id: ad7f0dce959e2760d3977b04925945e0447abc1d
Summary:
We'll need to gate portions of our shutdown so that we don't
tear down the database until after in-flight requests have completed.
This seems like the easiest way to go about it.
Reviewed By: simpkins
Differential Revision: D5796593
fbshipit-source-id: 49e695826ae68cc2b1d724a8da53ce5d884ff9ff
Summary:
This is a mechanical and dumb move of the code from MountPoint
and into the EdenMount class.
Of note, it doesn't merge together the two different state/status fields
into a unified thing; that will be tackled in a follow on diff.
Reviewed By: bolinfest
Differential Revision: D5778212
fbshipit-source-id: 6e91a90a5cc760429d87a475ec12f81b93f87be0
Summary:
This is leading up to folding the MountPoint code into
the EdenMount class.
There's still a mention of the MountPoint in Dispatcher.h; that is
being dealt with in the following diff.
Reviewed By: bolinfest
Differential Revision: D5778215
fbshipit-source-id: 996640b3773988a4738ad55bb13de45e1ffe1880
Summary:
The higher level goal is to make it easier to deal
with the graceful restart scenario.
This diff removes the SessionDeleter class and effectively renames
the Channel class to FuseChannel. The FuseChannel represents
the channel to the kernel; it can be constructed from a fuse
device descriptor that has been obtained either from the privhelper
at mount time, or from the graceful restart procedure. Importantly
for graceful restart, it is possible to move the fuse device
descriptor out of the FuseChannel so that it can be transferred
to a new eden process.
The graceful restart procedure requires a bit more control over
the lifecycle of the fuse event loop so this diff also takes over
managing the thread pool for the worker threads. The threads
are owned by the MountPoint class which continues to be responsible
for starting and stopping the fuse session and notifying EdenServer
when it has finished. A nice side effect of this change is that
we can remove a couple of inelegant aspects of the integration;
the stack size env var stuff and the redundant extra thread
to wait for the loop to finish.
I opted to expose the dispatcher ops struct via an `extern` to
simplify the code in the MountPoint class and avoid adding special
interfaces for passing the ops around; they're constant anyway
so this doesn't feel especially egregious.
Reviewed By: bolinfest
Differential Revision: D5751521
fbshipit-source-id: 5ba4fff48f3efb31a809adfc7787555711f649c9
Summary:
The serialized data for each file handle needs to be enough
to re-construct the handle when we load it into a new process later
on. We need the inode number, the file handle number that we communicated
to the kernel and a flag to let us know whether it is a file or a dir.
Note that the file handle allocation strategy already accomodates the
idea of migrating to a new process; we don't need to serialize anything
like a next file handle id number.
This doesn't implement instantiating the handles from the loaded state,
it is just the plumbing for saving and loading that state information.
Reviewed By: bolinfest
Differential Revision: D5733079
fbshipit-source-id: 8fb8afb8ae9694d013ce7a4a82c31bc876ed33c9
Summary:
Update all of the code using ThreadLocal<EdenStats> to pass in a non-default
Tag parameter to the ThreadLocal template.
A non-default tag parameter is required to use the accessAllThreads() method on
the ThreadLocal object. We need to use accessAllThreads() to perform stats
aggregation correctly. Currently the EdenServer code is only performing
aggregation for stats in the FunctionScheduler.
This diff only updates the ThreadLocal<EdenStats> type, but does not contain
any behavior changes. I will fix the stats aggregation in a subsequent diff.
Reviewed By: bolinfest
Differential Revision: D5657268
fbshipit-source-id: bc4b6f56324eb8d3052c023fd3f6f64f99b1d4e0
Summary:
Recently, autodeps switched to requiring simple types for the various dep
parameters in order to support proper annotation parsing (D5387184), which
started breaking on `eden/fs/fuse/TARGETS`.
Reviewed By: simpkins
Differential Revision: D5620249
fbshipit-source-id: 16fa2c73421ff7e9929c71290a662babde1289ec
Summary:
Move the dropPrivileges() function from the PrivHelper code to the UserInfo
class, and update it to correctly set supplementary groups when dropping
privileges.
Reviewed By: bolinfest
Differential Revision: D5501254
fbshipit-source-id: 62d7a91685ae72a73e848236f6e6f636b6203481
Summary:
This adds a new UserInfo class to store the UID, GID, as well as the username
and home directory. This moves the determineUid() and determineGid() functions
from main.cpp into this new class, and makes the logic somewhat smarter now.
In addition to looking up the UID and GID, we now look up the username. This
information is not used yet, but will be used in an upcoming diff to set
supplementary groups.
This also stores the home directory in the UserInfo class. The home directory
is usually necessary to find the user's ~/.edenrc file. Computing it as part
of UserInfo makes the most sense since we will likely have already looked up
the user's passwd entry.
Reviewed By: bolinfest
Differential Revision: D5501252
fbshipit-source-id: 1cb4be9f6c1493de4362da3393034e78bedd9db2
Summary: This uses `clang-tidy` to comment out unused parameters (in functions, methods and lambdas) in fbcode. Cases that the tool failed to handle are fixed manually.
Reviewed By: igorsugak
Differential Revision: D5454343
fbshipit-source-id: 5dee339b4334e25e963891b519a5aa81fbf627b2
Summary:
Always call umount2() with the MNT_DETACH flag. This prevents the unmount
operation from failing if the mount point is still busy.
This is pretty much always necessary in Facebook's environment, since chg
processes tend to hold open long-lived file handles inside the mount point.
In the future we could potentially add back an option to only unmount if the
mount point is not busy, but I'll wait to do that until we find a use case for
it.
Reviewed By: akushner
Differential Revision: D5367407
fbshipit-source-id: 6805fe37346b9b35af3c40ab84c0187d8deafbd0
Summary:
This is mostly needed because of the `folly:file` target split, but there are a few other that had changes that weren't reflected.
This also re-enables auto-deps for `multifeed/aggregator/processor/TARGETS` as it works again.
Reviewed By: yfeldblum
Differential Revision: D5362875
fbshipit-source-id: f9d2793249c0bfe644d0504f19839c108648ea3b
Summary: It doesn't need to exist anymore
Reviewed By: yfeldblum
Differential Revision: D5318746
fbshipit-source-id: c70b184f4b3fc12ede4632d6b3d43de16ed758c7
Summary:
Format all of the TARGETS files under eden/fs with the autodeps tool.
A few rocksdb include statements require comments so that autodeps can
correctly tell which dependency this include comes from. The rocksdb library's
source file structure unfortunately does not match the layout of how its header
files get installed, so autodeps cannot figure this out automatically.
Reviewed By: wez
Differential Revision: D5316000
fbshipit-source-id: f8163adca79ee4a673440232d6467fb83e56aa10
Summary: This includes adjusting them to conform to Folly's style for single file sources and headers, making targets as small as possible, not providing aggregate targets, and using auto-deps.
Reviewed By: yfeldblum
Differential Revision: D5312078
fbshipit-source-id: 1c7f5aa04da3bad236ffa23000c7bda47b82e3ef
Summary:
Update eden to log via the new folly logging APIs rather than with glog.
This adds a new --logging flag that takes a logging configuration string.
By default we set the log level to INFO for all eden logs, and WARNING for
everything else. (I suspect we may eventually want to run with some
high-priority debug logs enabled for some or all of eden, but this seems like a
reasonable default to start with.)
Reviewed By: wez
Differential Revision: D5290783
fbshipit-source-id: 14183489c48c96613e2aca0f513bfa82fd9798c7
Summary:
This change makes it so that all of the C++ code related to the edenfs daemon
is now contained in the eden/fs subdirectory.
Reviewed By: bolinfest, wez
Differential Revision: D4889053
fbshipit-source-id: d0bd4774cc0bdb5d1d6b6f47d716ecae52391f37
