Summary: Since `CurlClient` is going to be used in multiple threads and each thread has its own instance of CurlClient, we are now able to assign one curl_easy handle for each of the client. So curl would be able to reuse the existing connections to Mononoke API Server.
Reviewed By: chadaustin
Differential Revision: D14334486
fbshipit-source-id: aacd210773c44dea6f1d9dd7f9b09d765a9c7a0a
Summary: Put `CurlClient` into its own thread pool (like how `HgBackingStore` uses `HgImporter`)
Reviewed By: chadaustin
Differential Revision: D14334483
fbshipit-source-id: 3ce2a8245761c11bfa0f66691d10a66004b46fe6
Summary: This diff enables `MononokeCurlBackingStore` in HgBackingStore. It also adds a new config value `mononoke:connection-type` (values: `http`, `curl` and `thrift`).
Reviewed By: chadaustin
Differential Revision: D14135028
fbshipit-source-id: cc6fb95d94b3d98996d872d22ad1a95a97d562ab
Summary: This diff add a MononokeBackingStore implemented based on libcurl so we could add Mononoke support on macOS.
Reviewed By: chadaustin
Differential Revision: D14096313
fbshipit-source-id: 2bcb0c49002dcbb0a0190ba505a6e814e74db747
Summary:
This diff added a configurable value "mononoke:connector" to EdenConfig, so that we can specify a particular type of connection we need when using Mononoke as backing store.
This diff also moved the logic of initializing the existing `MononokeHttpBackingStore` to `initializeHttpMononokeBackingStore`.
Reviewed By: wez, strager
Differential Revision: D14141725
fbshipit-source-id: c7208295d7b3853740d7b0e5166f8854457fcf8e
Summary:
The feature was completed by Phil in D9816270. It's handy and can probably
reduce user support burden like: https://fb.intern.facebook.com/groups/scm/permalink/2039619916087618/
Therefore let's enable it.
Reviewed By: DurhamG
Differential Revision: D14293405
fbshipit-source-id: 54e934e0bf495c090109462e4f743d427df39380
Summary:
Update `eden start` to honor the `--if-necessary` flag even when starting
with systemd.
Reviewed By: chadaustin, strager
Differential Revision: D14356541
fbshipit-source-id: b7c12b11a2fd63eeee84026bf4418702f794cbf0
Summary:
Update most of the `eden/cli/config.py` to use `Path` instead of `str` where
appropriate. This also updates several of the APIs in `util.py` that were
affected as well.
Reviewed By: chadaustin
Differential Revision: D14356543
fbshipit-source-id: a8f6d15b8870bf689eeb78f9fc0e9a0c65c97218
Summary:
Not sure why gflags is automatically included on Linux and not in
mode/mac, but add the includes nonetheless!
Reviewed By: strager
Differential Revision: D14389458
fbshipit-source-id: 27811ec4bb65b03e73b15bb51de1264dbfe053dc
Summary:
Remove the `CheckoutConfig.hooks_path` attribute which is no longer used
anywhere.
Reviewed By: strager
Differential Revision: D14356542
fbshipit-source-id: 28b7e7aeb197aeb5d04edf8adfdcfe7946ffa0c7
Summary:
This file has not been used D13447272, which moved creation of the `.hg`
directory from a separate hook into the CLI itself. Remove it from the
packaging.
Reviewed By: strager
Differential Revision: D14356544
fbshipit-source-id: 18232c0e7e80e8cb255dd66f70b2a21b4abe6ca7
Summary:
If no mounts are configured `eden fsck` previously threw an exception when
trying to compute the return value. It called `max(return_codes)` on an empty
return codes list, which would fail. This changes the code to handle that
code specially and report a warning that there was nothing to check.
Reviewed By: chadaustin
Differential Revision: D14352112
fbshipit-source-id: 3815ef34a12834d642f3eee867dda6dc1117c2ef
Summary:
Update the fsck `check_all()` function to use the newer
`EdenInstance.get_checkouts()` API instead of the older, internal
`_get_directory_map()` method.
Reviewed By: chadaustin
Differential Revision: D14352113
fbshipit-source-id: f1ceae72747833d89d5afd11da585ba96b5615a4
Summary:
Previously the `eden doctor` tests were all in a single `doctor_test.py` file
and it was getting quite large and unwieldy (3000+ lines).
This moves the doctor tests from `eden/cli/test/` to `eden/cli/doctor/test/`
and splits up the various test case classes into their own modules, and the
various helper classes into modules in `eden/cli/doctor/test/lib/`
This doesn't have any functional code changes other than moving things around.
Reviewed By: chadaustin
Differential Revision: D14338814
fbshipit-source-id: 5373c7454567b4417241b4c85ec53d3ded29a8c9
Summary:
Update the "eden doctor" code to directly examine `/proc` when searching for
edenfs processes rather than calling `pgrep`.
This is a bit more efficient, it lets us accurately detect the command line
arguments. In the deadlock scenario in T38417129 this likely would have
prevented `eden doctor` from hanging since it no longer attempts to read
`/proc/<pid>/cmdline` for any process other than `edenfs` itself.
This also fixes a minor bug where the code incorrectly warns about `pgrep`
failing when there are no `edenfs` processes running.
Reviewed By: strager
Differential Revision: D14255535
fbshipit-source-id: f5f40566b46b86201aa59f4554e6acf058337c5b
Summary: Internal Facebook infrastructure is nagging me about some files not having a Facebook copyright notice. Add a notice to these files to make the nagging stop.
Reviewed By: simpkins
Differential Revision: D14173944
fbshipit-source-id: 7234431224fcf4f86ea56ca2f9108f47ef959d87
Summary:
I was thinking about teaching EdenFS' `globFiles` to follow symlinks. To help me think, I documented my understanding of how EdenFS currently handles glob patterns. Publish my notes, even if I decide to not change `globFiles`' behavior.
This document isn't meant to be thorough, so many definitions and nuances are omitted.
Reviewed By: simpkins
Differential Revision: D14288417
fbshipit-source-id: e78749e114224926c2673868854d068672e38cef
Summary:
Add a locale translation file that we can install to modify the error message
that most programs print for `ENOTCONN` errors.
Any I/O operations inside an unmounted FUSE mount will fail with `ENOTCONN`.
Many users don't really know what to do when they encounter this error. This
alternate error message should hopefully help point them in the right
direction.
Reviewed By: wez, pixelb
Differential Revision: D14360858
fbshipit-source-id: 9d538ee675815f017160c3382e2919136367464f
Summary:
This updates `edenfs` to automatically create the mount point directory
if it does not exist.
Previously the `eden mount` CLI command would automatically create the mount
directory in the Python logic. This adds similar logic to the C++ code, which
handles more situations. In particular this makes it so that `eden start`
will now automatically create missing mount point directories.
Note that the C++ code does not create the `README_EDEN.txt` symlink inside
the mount point if it is missing. We could move that functionality into the
C++ code in the future if needed.
Reviewed By: strager
Differential Revision: D14254699
fbshipit-source-id: bad5634f57fba6e7af3b6a3830eb51ac099b435e
Summary:
Update `Overlay::initialization()` to perform the bulk of the initialization
logic in the GC thread. We re-use the GC thread simply for convenience, since
it already exists and won't have any work to do until initialization has
completed anyway.
After an unclean shutdown this allows edenfs to start and perform the overlay
scans for all of its checkouts in parallel rather than serially.
Reviewed By: wez
Differential Revision: D14154868
fbshipit-source-id: c177cb9f950a6317fd7ea06888bd5b326a55ace7
Summary:
Add an Overlay::initialize() function, and consolidate all Overlay
initializtion logic in this function. Previously some initialization was done
by the Overlay constructor, and some was done in `EdenMount::initialize()`
Overlay::initialize() returns a folly::SemiFuture as it may perform a
non-trivial amount of disk I/O. However, at the moment it currently performs
all I/O in the current thread before it returns. I plan to move this work to
a separate thread in a subsequent diff.
Reviewed By: strager
Differential Revision: D13981140
fbshipit-source-id: b59eaef88012a8e74fcb770a9c93ca3f9bde32a0
Summary:
This updates the `EdenServer` class so that the existing `getMount()` and
`getMountPoints()` APIs only return mounts that have finished initializing.
These APIs are primarily used by the thrift interfaces. In most cases the
callers did not intend to operate on mounts that were still initializing, and
doing so was unsafe. The code could potentially dereference a null pointer if
it tried to access the mount's root inode before the root inode object had
been created.
New `getMountUnsafe()` and `getAllMountPoints()` APIs have been added for call
sites that explicitly want to be able to access mounts that may still be
initializing. Currently the `listMounts()` thrift API is the only location
that needs this.
Reviewed By: strager
Differential Revision: D13981139
fbshipit-source-id: e6168d7a15694c79ca2bcc129dda46f82382e8e9
Summary:
Add a flag to tell edenfs to report successful start-up as soon as the thrift
server is running, without waiting for all mount points to finish being
remounted.
In the future I plan to have edenfs automatically perform an fsck scan of the
overlay for checkouts that were not shut down cleanly. This may cause the
remount to take a significant amount of extra start-up time in some cases.
(This is already true today in some cases even with the simpler scan we do to
re-compute the max inode number.)
I think we will probably want to have systemd invoke edenfs with this option,
so that we do not time out during system start up if some mount points need to
be rescanned.
Reviewed By: strager
Differential Revision: D13522040
fbshipit-source-id: 6f183770c25efee34c4805c9bad42a9cce51039e
Summary:
In the future, I want some tests to mock PrivHelper::fuseUnmount. The existing classes for mocking PrivHelper::fuseMount, MountPromiseDelegate and FailingMountDelegate, work poorly for fuseUnmount.
Combine MountPromiseDelegate and FailingMountDelegate to create MockMountDelegate, which is more flexible and allows a mocking fuseUnmount to be implemented easily.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14319712
fbshipit-source-id: 6eecf121e3b44f39cd5dffbeafaf0f3aab75cb76
Summary: Switch to using cpptoml from third-party on all platforms.
Reviewed By: igorsugak
Differential Revision: D14351179
fbshipit-source-id: 183bedc7d27e9c0f9216f3d0cca58ada75ac74e7
Summary: Ensure that an EdenMount is RUNNING after it takes over an EdenMount from another process.
Reviewed By: simpkins
Differential Revision: D14178500
fbshipit-source-id: 2813b024b6815dc5d31f3e3bf89cffa98ad0f1c3
Summary:
I want manipulate a FakeFuseMountDelegate in a test, but FakeFuseMountDelegate is private to FakePrivHelper.cpp. Make FakeFuseMountDelegate usable outside FakePrivHelper.cpp by putting its declaration in FakePrivHelper.h.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14319711
fbshipit-source-id: d7af18c316f63febe1d60f6e5aadcd4f4827cca5
Summary:
Pyre complains about this cast being redundant. It was needed at one point
(in D9144139) to make either pyre or mypy happy, but now neither seem to need
it and pyre actually seems to complain about it.
Reviewed By: chadaustin
Differential Revision: D14319772
fbshipit-source-id: ed252763acfe9c3c6326c6712e30d2840080454c
Summary:
Update `edenfs` to automatically create the bind mount source directories if
they are missing. Previously Eden would report an error and would not be able
to mount the checkout if some of the bind mount source directories were
missing.
Reviewed By: strager
Differential Revision: D14253771
fbshipit-source-id: 87ad091ccf2c0f0f72aebb50437fd7680ddbfd1c
Summary:
Update the `eden rage` code to catch any unexpected exceptions from the
`eden doctor` logic so that it can still generate the final rage report.
Having a report that is missing the doctor information is better than failing
to generate a rage report at all. The traceback from the `eden doctor`
exception may also help diagnose whatever issue is causing problems.
Reviewed By: chadaustin
Differential Revision: D14255648
fbshipit-source-id: 365826f22ed933f80a3cd43ddd1d66009d7032fc
Summary:
Make "eden doctor" check if the process's current working directory matches
the "$PWD" environment variable, and suggest that the user run "cd / && cd -"
if they do not match.
The most common cause of this is users that cd'ed into an Eden checkout
directory before Eden was running, and then started Eden but haven't updated
their working directory since. In this shell they will still see the contents
of the underlying mount point directory itself and not the Eden checkout
contents that have been mounted on top of it.
Reviewed By: chadaustin
Differential Revision: D14249909
fbshipit-source-id: ecfc2a56180e3a6161940e9e1ec6d94663bddbc0
Summary:
[Folly] Stop checking `EventBase::runInEventBaseThread` result, as the function will soon be changed not to return any result.
It returned `false` when failing to enqueue a task. But it cannot really fail anyway besides allocation failure, unless in the `EventBase` destructor and while draining and the `AlwaysEnqueue` variant is called.
Reviewed By: andriigrynenko
Differential Revision: D14254969
fbshipit-source-id: a6a9199cbafa18b61488a240e4318ce946953f51
Summary:
We're not that far from building privhelper on mode/mac but it does
require figuring out how to depend on osxfuse from the Buck build, so
bypass that by breaking the inodes target's dependency on privhelper's
<fuse_ioctl.h> include.
Reviewed By: strager
Differential Revision: D14218709
fbshipit-source-id: edbb2a21df06d6f2a4f860ef13718ad05d445e98
Summary: Fix some missing includes that showed up in my mode/mac builds of parts of Eden.
Reviewed By: simpkins, strager
Differential Revision: D14213572
fbshipit-source-id: 54e070e89afdfb8479abdaa122ba76ca1d2d9ba9
Summary:
Throw a slightly nicer error message if the hg import helper exits before
returning a `CMD_STARTED` response or an error response.
When switching from the `hg_import_helper.py` script to the
`hg debugedenimporthelper` subcommand we are slightly more likely to encounter
errors during startup. For instance, if the repository path was not a valid
repository the `hg_import_helper.py` code would send an error message back
that the `HgImporter.cpp` code could parse. However, we unfortunately can't
catch errors loading the repository the same way when using an `hg`
subcommand. If it fails to load the repository it will simply print an error
message to stderr and then bail out before it invokes the
`debugedenimporthelper` code.
Reviewed By: chadaustin
Differential Revision: D14222266
fbshipit-source-id: cd60e5a61725d45a816b74f63b9969b29ade2160
Summary:
EdenMount calls fuseMount, but EdenServer calls fuseUnmount. I think only one of EdenMount or EdenServer should call fuseMount/fuseUnmount. Splitting the responsibility is confusing.
Move the call to fuseUnmount into EdenMount by creating a member function called unmount.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14115385
fbshipit-source-id: 845c4a87cdba9f270c0eacaf01916ad91c3b7c0e
Summary:
Update `EdenMount::initialize()` to perform a fault injection check. This
allows test code to inject delays and errors into the mount initialization
flow.
Reviewed By: strager
Differential Revision: D14079491
fbshipit-source-id: be80135b0833c8f0300104524473cc3e949fec34
Summary:
This reverts the change in D14188312 that moved `hg_import_helper.py` from
`bin/` to `libexec/eden/`, and instead updates edenfs to look for
`hg_import_helper.py` at `../../bin/hg_import_helper.py`
We cannot remove `hg_import_helper.py` from `/usr/local/bin` without breaking
existing running `edenfs` daemons as they will continue to look for
`hg_import_helper.py` in this location.
Rather than keeping a copy in both `bin/` and `libexec/eden` it seems better
to simply update edenfs so that it can correctly find `hg_import_helper.py`
even when edenfs itself has been installed in `libexec/eden/`
Reviewed By: strager
Differential Revision: D14197390
fbshipit-source-id: 2845bd79343bc29af9d3acbacecac4501cdb9032
Summary: We should only attempt to include selinux on Linux, not macOS.
Reviewed By: strager
Differential Revision: D14181109
fbshipit-source-id: be47b7bdadc3409577fa114559e905214848ebd8
Summary:
The Eden CLI tool is really a control program for `edenfs`. Rename it to
`edenfsctl` to free up the `eden` name for future use.
The Eden daemon shouldn't really be on the user's path, and instead belongs in
`libexec`.
For transition compatibility, `eden` is symlinked to `edenfsctl`.
Reviewed By: simpkins
Differential Revision: D13888875
fbshipit-source-id: 435cc63e92b85b1f28b8691e4846fbcb05bc450e
.pyre_configuration.local is a file specific to Facebook's
infrastructure. Committing this file to the public Eden repo was
unintentional. Delete this file to reduce confusion.
Summary:
futures::sleep returning a Future leads to continuations easily being run on
the Timekeeper's callback. The goal is to change the return type so that
futures::sleep returns a folly::SemiFuture.
This codemod is part of the first stage:
* Migrate all call sites off of futures::sleep and onto futures::sleepUnsafe.
This will be followed by:
* Changing the return type of futures::sleep to folly::SemiFuture.
* Migrating callers, where clearly safe such as when they follow with .via or
.semi() from sleepUnsafe to sleep.
* Migrating remaining callers.
Reviewed By: yfeldblum
Differential Revision: D14152623
fbshipit-source-id: bc6874e4320e4a7352ac61b20d9750458e2cbbff
Summary:
If EdenMount::shutdown is in progress and EdenMount::destroy is called [1], the EdenMount object won't ever be deleted. A comment in EdenMount::destroy claims that EdenMount::shutdown will delete the EdenMount object, but the comment is wrong:
case State::SHUTTING_DOWN:
// Nothing else to do. shutdown() will destroy us when it completes.
return;
This leak is probably benign, but the leak checker can cause tests to fail spuriously. I think the leak currently can't happen for EdenServer's mounts, but in the future the leak might occur and cause problems.
Make EdenMount::shutdown delete the EdenMount if EdenMount::destroy was called. This fixes the leak.
[1] EdenMount::destroy is called by EdenMountDeleter when the last std::shared_ptr<EdenMount> is destroyed.
Reviewed By: simpkins
Differential Revision: D14015024
fbshipit-source-id: 39a0ba78bb00afb5f0363fe5c74dbec00f9d68e3
