Summary:
Update `RocksHandles` to call `RepairDB()` if we get an error opening the
database.
We have seen errors opening the DB in some cases after hard server reboots.
In every case so far `ldb repair` has been able to repair it with no adverse
effects. This simply makes `edenfs` automatically attempt to perform this DB
repair step.
Reviewed By: chadaustin, strager
Differential Revision: D14452216
fbshipit-source-id: 10c0cb0ff9cea3c3bbe485a4e489e4a6df640803
Summary:
When opening the RocksDB, write one entry to each column family and then
explicitly flush that column family. This ensures that the column family
information has actually been flushed to an SST file. Without this some
column families may only have been written out to the write-ahead log files.
(Even calling `db->Flush()` does not appear to be sufficient; each column
family has to be explicitly flushed.)
The RocksDB' `RepairDB()` function (used by `ldb repair`) currently ends up
deleting column families that do not have any data defined in an SST file.
The repair tool ends up deleting column families that only have data in log
files.
The fact that we haven't been doing these explicit flushes previously probably
isn't too much of a concern in practice: once we write out enough data RocksDB
will automatically trigger a flush. This only matters in cases where we have
not yet written out enough data to trigger an automatic flush.
Note that with this change we re-write these `id` keys each time we open the
RocksDB store, even if they were already present.
Reviewed By: chadaustin, strager
Differential Revision: D14452214
fbshipit-source-id: 3f1b17e240cc89fe00e3d31105d16452795e754d
Summary:
folly::readFile() doesn't work correctly on Windows. It was returning the incorrect size. I didn't check if the read contents were correct, because I was testing it on a binary data file.
Moreover we would need wide char API which won't be a part of Folly for sometime.
Reviewed By: strager
Differential Revision: D14205306
fbshipit-source-id: 55859dca1a6399d5b5f106b4c6757e582ba1375d
Summary:
Rename `main.cpp` to `EdenMain.cpp` now that this contains the `EdenMain`
class rather than the top-level `main()` function, and rename `RunServer.cpp`
to `main.cpp`
Reviewed By: pkaush
Differential Revision: D14435306
fbshipit-source-id: e1528a773e0724c6bd50e31f6b33a1762d7bd49e
Summary:
This restructures `main.cpp` to turn our current top-level `main()` into a
method of a virtual class.
This allows us to eliminate the slightly awkward way that `main.cpp` depends
on `RunServer.cpp`. This required us to list `main.cpp` in the sources
parameter of multiple buck targets rules, which confuses some other tools like
autodeps. With this new change, `RunServer.cpp` depends on `main.cpp` instead
of the other way around. The real top-level `main()` function now lives in
`RunServer.cpp`
Reviewed By: pkaush
Differential Revision: D14435309
fbshipit-source-id: 402d00db0d8aa8d473d51a4f0e9d9d80c97a0134
Summary:
The systemctl command requires XDG_RUNTIME_DIR to be configured. If it's not configured, 'eden start' should pick a sane default. The sane default includes the user's UID (e.g. /run/user/6986). I want this default to be configurable via Eden's config files.
Expose the ${USER_ID} token to Eden configs. This will let administrators can customize XDG_RUNTIME_DIR's fallback value in the future.
Reviewed By: wez
Differential Revision: D13811732
fbshipit-source-id: 7933e078dd5f2b3bbbb0299730220a129c257256
Summary:
Implement `flush()` and `fsyncdir()` in `EdenDispatcher` and explicitly return
ENOSYS. We intentionally do not need to do any work for these methods.
Providing our an implementation that returns ENOSYS avoids the error log
message in the default `Dispatcher` implementation about the these calls being
unimplemented.
Reviewed By: chadaustin
Differential Revision: D14453245
fbshipit-source-id: 71efe6de6af73a5d705dace0f3439ba2466a50a8
Summary:
Update the FuseChannel code to explicitly handle `FUSE_LSEEK` and `FUSE_POLL`
to avoid emitting error log messages about these calls not being implemented.
We intentionally do not implement these operations at the moment.
Also drop the log about other unknown opcodes from an error log to a warning
message.
Reviewed By: chadaustin
Differential Revision: D14453246
fbshipit-source-id: 2605cf7e5c160cda92460a80187438ac3549ade5
Summary:
I want to change some of the StartupLogger behavior to automatically show all
INFO and higher level log messages to the user during start-up. These two
messages from the FUSE code are currently logged at the INFO level, but don't
really seem important enough to show to the user.
This drops their level to DBG1. These messages will therefore still be shown
in the normal `edenfs.log` file, since it includes everything from DBG2 and
up.
Reviewed By: chadaustin
Differential Revision: D14453247
fbshipit-source-id: 5d79766f87e658b807029d82ac035cb94ae68832
Summary:
The Windows headers and source files currently aren't mentioned in any buck
build rule, which confuses autodeps. Mark them all `manual` to tell
autodeps to simply ignore them.
Reviewed By: strager
Differential Revision: D14435307
fbshipit-source-id: a10b9be779f232cc5ae74356d7f54e77ca5414cb
Summary:
- Wait to call `curl_global_init()` until after we have dropped root
privileges. `curl_global_init()` performs a non-trivial amount of work,
including loading and processing OpenSSL configuration files from disk. To
guard against any possible security issues in this code we should wait until
after we have dropped root privileges to do this.
- Move the call to `curl_global_cleanup()` to after the main `EdenServer`
object has been destroyed. Since the EdenServer object will likely contain
backing store objects that use curl it seems like we want to make sure we
wait to clean up the curl library until after the `EdenServer` has been
destroyed. This change uses `SCOPE_EXIT` to perform the cleanup as one of
the last steps of `main()`, and this also reduces the number of `#ifdef`
blocks that we need.
Reviewed By: strager, fanzeyi
Differential Revision: D14435308
fbshipit-source-id: 6c277e4471f0f93decebd4fc741639c6a047d62a
Summary: We don't run this binary anymore, no reason to build and ship it.
Reviewed By: quark-zju
Differential Revision: D14437317
fbshipit-source-id: dd6da521783f18a2a518a7aa042be98950894e89
Summary:
The future returned by EdenMount::shutdown() can continue on an arbitrary thread. This has caused at least one deadlock in the past (D14337058).
Prevent users of EdenMount::shutdown() from continuing on the wrong thread by making shutdown return a SemiFuture instead of a Future.
This diff should not change behavior for production code, since all users of EdenMount::shutdown() already use Future<>::via().
Reviewed By: simpkins
Differential Revision: D14389607
fbshipit-source-id: 821d206d01ea3dcb0261cd0c7ca2d591b2c84e7a
Summary:
If TreeInode::startLoadingInode() is in progress, and EdenServer::startTakeoverShutdown() is called, edenfs can deadlock:
1. Thread A: A FUSE request calls TreeInode::readdir() -> TreeInode::prefetch() -> TreeInode::startLoadingInode() on the children TreeInode-s -> RocksDbLocalStore::getFuture().
2. Thread B: A takeover request calls EdenServer::performTakeoverShutdown() -> InodeMap::shutdown().
3. Thread C: RocksDbLocalStore::getFuture() (called in step 1) completes -> TreeInode::inodeLoadComplete(). (The inodeLoadComplete continuation was registered by TreeInode::registerInodeLoadComplete().)
4. Thread C: After TreeInode::inodeLoadComplete() returns, the TreeInode's InodePtr is destructed, dropping the reference count to 0.
5. Thread C: InodeMap::onInodeUnreferenced() -> InodeMap::shutdownComplete() -> EdenMount::shutdown() (called in step 2) completes -> EdenServer::performTakeoverShutdown().
6. Thread C: EdenServer::performTakeoverShutdown() -> localStore_.reset() -> RocksDbLocalStore::~RocksDbLocalStore().
7. Thread C: RocksDbLocalStore::~RocksDbLocalStore() signals the thread pool to exit and waits for the pool's threads to exit. Because thread C is one of the threads managed by RocksDbLocalStore's thread pool, the signal is never handled and RocksDbLocalStore::~RocksDbLocalStore() never finishes.
Fix this deadlock by executing EdenServer::shutdown()'s callback (in EdenServer::performTakeoverShutdown()) on a different thread.
Reviewed By: simpkins
Differential Revision: D14337058
fbshipit-source-id: 1d63b4e7d8f5103a2dde31e329150bf763be3db7
Summary:
FuseChannel::initialize throws runtime_error when the FUSE connection is interrupted. I want EdenMount::startFuse to handle unexpected-unmount errors from FuseChannel::initialize, but catching runtime_error would catch unrelated errors too.
When the FUSE connection is interrupted during initialization, make FuseChannel throw FuseDeviceUnmountedDuringInitialization instead of runtime_error.
Reviewed By: chadaustin
Differential Revision: D14077848
fbshipit-source-id: ed7b7d370a83ed1a9c36a443d8bb06ba940dc032
Summary:
In this change, we separate the low-level code that manipulates the overlay
into the FsOverlay class. The Overlay class makes use of the FsOverlay and
InodeMetaData table to support its Overlay interfaces. The FsOverlay class
is decoupled from the Overlay class, allowing other classes to manipulate
the overlay independently. We have a need for this in order to add
fsck to the c++ code base : described in T40728883.
Reviewed By: simpkins
Differential Revision: D14218281
fbshipit-source-id: 66c587f2b341579b8075ca5e5eeb4da6ffadf6f5
Summary: Since `CurlClient` is going to be used in multiple threads and each thread has its own instance of CurlClient, we are now able to assign one curl_easy handle for each of the client. So curl would be able to reuse the existing connections to Mononoke API Server.
Reviewed By: chadaustin
Differential Revision: D14334486
fbshipit-source-id: aacd210773c44dea6f1d9dd7f9b09d765a9c7a0a
Summary: Put `CurlClient` into its own thread pool (like how `HgBackingStore` uses `HgImporter`)
Reviewed By: chadaustin
Differential Revision: D14334483
fbshipit-source-id: 3ce2a8245761c11bfa0f66691d10a66004b46fe6
Summary: This diff enables `MononokeCurlBackingStore` in HgBackingStore. It also adds a new config value `mononoke:connection-type` (values: `http`, `curl` and `thrift`).
Reviewed By: chadaustin
Differential Revision: D14135028
fbshipit-source-id: cc6fb95d94b3d98996d872d22ad1a95a97d562ab
Summary: This diff add a MononokeBackingStore implemented based on libcurl so we could add Mononoke support on macOS.
Reviewed By: chadaustin
Differential Revision: D14096313
fbshipit-source-id: 2bcb0c49002dcbb0a0190ba505a6e814e74db747
Summary:
This diff added a configurable value "mononoke:connector" to EdenConfig, so that we can specify a particular type of connection we need when using Mononoke as backing store.
This diff also moved the logic of initializing the existing `MononokeHttpBackingStore` to `initializeHttpMononokeBackingStore`.
Reviewed By: wez, strager
Differential Revision: D14141725
fbshipit-source-id: c7208295d7b3853740d7b0e5166f8854457fcf8e
Summary:
Not sure why gflags is automatically included on Linux and not in
mode/mac, but add the includes nonetheless!
Reviewed By: strager
Differential Revision: D14389458
fbshipit-source-id: 27811ec4bb65b03e73b15bb51de1264dbfe053dc
Summary: Internal Facebook infrastructure is nagging me about some files not having a Facebook copyright notice. Add a notice to these files to make the nagging stop.
Reviewed By: simpkins
Differential Revision: D14173944
fbshipit-source-id: 7234431224fcf4f86ea56ca2f9108f47ef959d87
Summary:
This updates `edenfs` to automatically create the mount point directory
if it does not exist.
Previously the `eden mount` CLI command would automatically create the mount
directory in the Python logic. This adds similar logic to the C++ code, which
handles more situations. In particular this makes it so that `eden start`
will now automatically create missing mount point directories.
Note that the C++ code does not create the `README_EDEN.txt` symlink inside
the mount point if it is missing. We could move that functionality into the
C++ code in the future if needed.
Reviewed By: strager
Differential Revision: D14254699
fbshipit-source-id: bad5634f57fba6e7af3b6a3830eb51ac099b435e
Summary:
Update `Overlay::initialization()` to perform the bulk of the initialization
logic in the GC thread. We re-use the GC thread simply for convenience, since
it already exists and won't have any work to do until initialization has
completed anyway.
After an unclean shutdown this allows edenfs to start and perform the overlay
scans for all of its checkouts in parallel rather than serially.
Reviewed By: wez
Differential Revision: D14154868
fbshipit-source-id: c177cb9f950a6317fd7ea06888bd5b326a55ace7
Summary:
Add an Overlay::initialize() function, and consolidate all Overlay
initializtion logic in this function. Previously some initialization was done
by the Overlay constructor, and some was done in `EdenMount::initialize()`
Overlay::initialize() returns a folly::SemiFuture as it may perform a
non-trivial amount of disk I/O. However, at the moment it currently performs
all I/O in the current thread before it returns. I plan to move this work to
a separate thread in a subsequent diff.
Reviewed By: strager
Differential Revision: D13981140
fbshipit-source-id: b59eaef88012a8e74fcb770a9c93ca3f9bde32a0
Summary:
This updates the `EdenServer` class so that the existing `getMount()` and
`getMountPoints()` APIs only return mounts that have finished initializing.
These APIs are primarily used by the thrift interfaces. In most cases the
callers did not intend to operate on mounts that were still initializing, and
doing so was unsafe. The code could potentially dereference a null pointer if
it tried to access the mount's root inode before the root inode object had
been created.
New `getMountUnsafe()` and `getAllMountPoints()` APIs have been added for call
sites that explicitly want to be able to access mounts that may still be
initializing. Currently the `listMounts()` thrift API is the only location
that needs this.
Reviewed By: strager
Differential Revision: D13981139
fbshipit-source-id: e6168d7a15694c79ca2bcc129dda46f82382e8e9
Summary:
Add a flag to tell edenfs to report successful start-up as soon as the thrift
server is running, without waiting for all mount points to finish being
remounted.
In the future I plan to have edenfs automatically perform an fsck scan of the
overlay for checkouts that were not shut down cleanly. This may cause the
remount to take a significant amount of extra start-up time in some cases.
(This is already true today in some cases even with the simpler scan we do to
re-compute the max inode number.)
I think we will probably want to have systemd invoke edenfs with this option,
so that we do not time out during system start up if some mount points need to
be rescanned.
Reviewed By: strager
Differential Revision: D13522040
fbshipit-source-id: 6f183770c25efee34c4805c9bad42a9cce51039e
Summary:
In the future, I want some tests to mock PrivHelper::fuseUnmount. The existing classes for mocking PrivHelper::fuseMount, MountPromiseDelegate and FailingMountDelegate, work poorly for fuseUnmount.
Combine MountPromiseDelegate and FailingMountDelegate to create MockMountDelegate, which is more flexible and allows a mocking fuseUnmount to be implemented easily.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14319712
fbshipit-source-id: 6eecf121e3b44f39cd5dffbeafaf0f3aab75cb76
Summary: Switch to using cpptoml from third-party on all platforms.
Reviewed By: igorsugak
Differential Revision: D14351179
fbshipit-source-id: 183bedc7d27e9c0f9216f3d0cca58ada75ac74e7
Summary: Ensure that an EdenMount is RUNNING after it takes over an EdenMount from another process.
Reviewed By: simpkins
Differential Revision: D14178500
fbshipit-source-id: 2813b024b6815dc5d31f3e3bf89cffa98ad0f1c3
Summary:
I want manipulate a FakeFuseMountDelegate in a test, but FakeFuseMountDelegate is private to FakePrivHelper.cpp. Make FakeFuseMountDelegate usable outside FakePrivHelper.cpp by putting its declaration in FakePrivHelper.h.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14319711
fbshipit-source-id: d7af18c316f63febe1d60f6e5aadcd4f4827cca5
Summary:
Update `edenfs` to automatically create the bind mount source directories if
they are missing. Previously Eden would report an error and would not be able
to mount the checkout if some of the bind mount source directories were
missing.
Reviewed By: strager
Differential Revision: D14253771
fbshipit-source-id: 87ad091ccf2c0f0f72aebb50437fd7680ddbfd1c
Summary:
[Folly] Stop checking `EventBase::runInEventBaseThread` result, as the function will soon be changed not to return any result.
It returned `false` when failing to enqueue a task. But it cannot really fail anyway besides allocation failure, unless in the `EventBase` destructor and while draining and the `AlwaysEnqueue` variant is called.
Reviewed By: andriigrynenko
Differential Revision: D14254969
fbshipit-source-id: a6a9199cbafa18b61488a240e4318ce946953f51
Summary:
We're not that far from building privhelper on mode/mac but it does
require figuring out how to depend on osxfuse from the Buck build, so
bypass that by breaking the inodes target's dependency on privhelper's
<fuse_ioctl.h> include.
Reviewed By: strager
Differential Revision: D14218709
fbshipit-source-id: edbb2a21df06d6f2a4f860ef13718ad05d445e98
Summary: Fix some missing includes that showed up in my mode/mac builds of parts of Eden.
Reviewed By: simpkins, strager
Differential Revision: D14213572
fbshipit-source-id: 54e070e89afdfb8479abdaa122ba76ca1d2d9ba9
Summary:
Throw a slightly nicer error message if the hg import helper exits before
returning a `CMD_STARTED` response or an error response.
When switching from the `hg_import_helper.py` script to the
`hg debugedenimporthelper` subcommand we are slightly more likely to encounter
errors during startup. For instance, if the repository path was not a valid
repository the `hg_import_helper.py` code would send an error message back
that the `HgImporter.cpp` code could parse. However, we unfortunately can't
catch errors loading the repository the same way when using an `hg`
subcommand. If it fails to load the repository it will simply print an error
message to stderr and then bail out before it invokes the
`debugedenimporthelper` code.
Reviewed By: chadaustin
Differential Revision: D14222266
fbshipit-source-id: cd60e5a61725d45a816b74f63b9969b29ade2160
Summary:
EdenMount calls fuseMount, but EdenServer calls fuseUnmount. I think only one of EdenMount or EdenServer should call fuseMount/fuseUnmount. Splitting the responsibility is confusing.
Move the call to fuseUnmount into EdenMount by creating a member function called unmount.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14115385
fbshipit-source-id: 845c4a87cdba9f270c0eacaf01916ad91c3b7c0e
Summary:
Update `EdenMount::initialize()` to perform a fault injection check. This
allows test code to inject delays and errors into the mount initialization
flow.
Reviewed By: strager
Differential Revision: D14079491
fbshipit-source-id: be80135b0833c8f0300104524473cc3e949fec34
Summary:
This reverts the change in D14188312 that moved `hg_import_helper.py` from
`bin/` to `libexec/eden/`, and instead updates edenfs to look for
`hg_import_helper.py` at `../../bin/hg_import_helper.py`
We cannot remove `hg_import_helper.py` from `/usr/local/bin` without breaking
existing running `edenfs` daemons as they will continue to look for
`hg_import_helper.py` in this location.
Rather than keeping a copy in both `bin/` and `libexec/eden` it seems better
to simply update edenfs so that it can correctly find `hg_import_helper.py`
even when edenfs itself has been installed in `libexec/eden/`
Reviewed By: strager
Differential Revision: D14197390
fbshipit-source-id: 2845bd79343bc29af9d3acbacecac4501cdb9032
Summary: We should only attempt to include selinux on Linux, not macOS.
Reviewed By: strager
Differential Revision: D14181109
fbshipit-source-id: be47b7bdadc3409577fa114559e905214848ebd8
Summary:
The Eden CLI tool is really a control program for `edenfs`. Rename it to
`edenfsctl` to free up the `eden` name for future use.
The Eden daemon shouldn't really be on the user's path, and instead belongs in
`libexec`.
For transition compatibility, `eden` is symlinked to `edenfsctl`.
Reviewed By: simpkins
Differential Revision: D13888875
fbshipit-source-id: 435cc63e92b85b1f28b8691e4846fbcb05bc450e
Summary:
futures::sleep returning a Future leads to continuations easily being run on
the Timekeeper's callback. The goal is to change the return type so that
futures::sleep returns a folly::SemiFuture.
This codemod is part of the first stage:
* Migrate all call sites off of futures::sleep and onto futures::sleepUnsafe.
This will be followed by:
* Changing the return type of futures::sleep to folly::SemiFuture.
* Migrating callers, where clearly safe such as when they follow with .via or
.semi() from sleepUnsafe to sleep.
* Migrating remaining callers.
Reviewed By: yfeldblum
Differential Revision: D14152623
fbshipit-source-id: bc6874e4320e4a7352ac61b20d9750458e2cbbff
