Summary:
Tl;DR Reduces costs of fuse request mertics by reducing lock contention.
D20922194 adds tracking for FUSE request metrics, this makes tracking those
metrics more efficient. Since every user request goes through the FUSE channel,
we want to reduce the cost of these metrics as much as possible (originally
mentioned in a comment D20922194).
The synchronization used to track the metrics is costly especially
when the lock is contended.
To reduce the cost, each FuseChannel will have a thread local copy of metrics.
Each will still be synchronized to allow for reading the metrics and for
Requests moved to other threads that will need to access the metrics. However,
the lock should be contended less often since only requests from a single fuse
channel thread will access it.
Reviewed By: chadaustin
Differential Revision: D21043792
fbshipit-source-id: ce58a0cbce334095976233bfac7578d39c81bb55
Summary:
This adds a new `EdenTestCaseBase` class to serve as the base class across a
number of our integration tests and some of our CLI tests.
The main goal of this is to allow eliminating many of the annoying `*Mixin`
classes used in a lot of our integration tests. These mixin classes are
annoying since they result in complicated multiple inheritance, and it can be
tricky to ensure that the method resolution order behaves the way you want.
The systemd tests in particular use a lot of mixins, which gets complicated.
These mixin classes are also awkward from a Python typing perspective, and the
systemd tests end up resorting to just declaring different APIs in several
places when `typing.TYPE_CHECKING` is True.
The fact that `EdenTestCaseBase` has a `contextlib.ExitStack` member variable
should make it easier for us to eliminate these mixins moving forward: rather
than using mixins that use inheritance and assume a `self.cleanUp()` method
exists, we can transition this code to standalone functions or context
managers, and they can take the `ExitStack` variable as an argument if
necessary.
Reviewed By: wez
Differential Revision: D21084097
fbshipit-source-id: 77ee457b7debe6f584f630e3e30f79fe634a2026
Summary:
Update most locations in edenfsctl to report the version number that was built
into the edenfsctl binary at build time, rather than querying the RPM database
for the installed RPM version. The RPM behavior only works on to RedHat-based
Linux systems, and the currently running process doesn't necessarily have to
have come from the RPM.
The one place where we do still attempt to print the RPM version is in the
`edenfsctl rage` report, when running on Linux.
Reviewed By: wez
Differential Revision: D21000168
fbshipit-source-id: 0fb747e71b6950d74f22c458efa0dfcbd45270bd
Summary:
This updates the top-level CMakeLists.txt file to compute package version
information, and expose this to C++ code in `eden-config.h`, and to Python
code in a new `eden/config.py` module.
Previously we exposed an `EDEN_VERSION` macro for the C++ code in
`eden-config.h`, but this was not initialized or used anywhere. Now the
top-level CMakeLists.txt file computes appropriate version information and
exposes the package name, version, release, commit ID, and build time in these
configuration files.
The version selection logic in CMakeLists.txt based largely on the code that
wez wrote for watchman in D20636833.
Reviewed By: wez
Differential Revision: D21000164
fbshipit-source-id: db1a1035f1eefec058bbad558d35e113005e454e
Summary:
This exposes metrics for the live FUSE requests (the duration
of the longest outstanding request and the number of outstanding
requests).
Because FUSE is the interface through which the user mostly interacts
with the file system they provide good metrics to judge if the perfomance
of eden is normal, or there may be an issue.
Exposing these counters this way will send them to ods, so it will not only
allow for debuging current issues, but can be used to look back at previous
problems. This data could also be used for alerting or more proactive
remediation.
Metrics are exposed per checkout to allow seeing which checkout was
having issues. This data will aggregated in `eden top` to be used as
an overall health indicator, but should more information be needed it
will be logged in ods.
Reviewed By: chadaustin
Differential Revision: D20922194
fbshipit-source-id: 16208883417acb77b62bf712cfdd9068c5420303
Summary:
Build and run the fuse privhelper unit tests, as well as the `drop_privs`
helper program used by some of the integration tests.
Reviewed By: wez
Differential Revision: D21004425
fbshipit-source-id: 650e0729909f4753095e19fba4f01c02d516713b
Summary:
The listIgnored argument is not used anywhere, so this won't fix anything,
but this caught my eye while looking at the code.
Reviewed By: fanzeyi
Differential Revision: D21147432
fbshipit-source-id: e696d926fe0999da14c8c5748a0951b229c20efb
Summary: These appear to be passing on my laptop, let's enable them.
Reviewed By: fanzeyi
Differential Revision: D21145712
fbshipit-source-id: cbe6578c2206192ec2fd8c68aba07d8ea860682b
Summary:
On Windows, paths components are usually separated by '\', and since the
repository path is stored in a toml file, whatever character is after a '\',
will be escaped. In my case, this is followed by U (for C:\Users), and thus
toml expects the next characters to be an escaped unicode. That's obviously
not the case and thus EdenFS fails to parse the config, preventing me from
cloning fbsource.
Since Windows is perfectly fine with '/' as path separator, let's just
replace '\' with '/'.
The underlying bug appears to be in the toml Python code: https://github.com/uiri/toml/issues/280
Manually trying some random path is pretty conclusive:
(Pdb) toml.dumps({'foo': 'c:\\Users\\wez'})
'foo = "c:\\\\Users\\\\wez"\n'
(Pdb) toml.dumps({'foo': 'c:\\Users\\xavier'})
'foo = "c:\\Users\\xavier"\n'
Reviewed By: chadaustin
Differential Revision: D21143545
fbshipit-source-id: 448471da12c253dd37680f6a28251a1e69850920
Summary:
The `eden_fuse_privhelper` library depends on CoreFoundation and IOKit.
Previously this dependency was specified for the `eden_service` library, which
doesn't use these frameworks directly, only indirectly through the
`eden_fuse_privhelper` library.
The fact that `eden_fuse_privhelper` was missing this dependency causes
problems when trying to build unit tests and tools that depend on
`eden_fuse_privhelper` without using the `eden_service` library.
Reviewed By: genevievehelsel
Differential Revision: D21057925
fbshipit-source-id: b846ebde0de158b70be462067c4412a655ad8036
Summary:
As a top layer of reliability in graceful restart, we'd like the CLI to enforce that something has started after a graceful restart call. There are a few things that can happen
1) The restart short circuts before attempting to shutdown (version mismatch etc). The old process will continue (`success == False`)
2) The restart is successful, the new process will continue (`success == True`)
3) The restart is not successful (failed ping), the old process will recover (`success ==
False`)
(note here that the `success` field is the return code of the `StartupLogger` process, not the edenfs process itself).
In case any of these fail, we'd like a final line of defence with the CLI.
First, in the case of a successful start from the `StartupLogger` report, we consider this successful and exit. If we do not have a successful (`success == False`) graceful restart, we will wait for the new process to recover (since the old process will go from `ALIVE` to `STOPPED` to `ALIVE`). I do not expect recovery to take long since we are only remounting, not waiting on RocksDB. Here though, we give a fairly long timeout (1hr) to ensure we're not stuck on an in process thrift call, since we would not like to force kill the process if we're writing to the overlay.
If we time out waiting, we will just kick off a force restart and assume the old process is stuck.
Reviewed By: simpkins
Differential Revision: D20673460
fbshipit-source-id: 215e06fa8aa76dbe5dae6602f82ce5f05968e538
Summary:
This restructures the `ThriftServer server_` stopping logic in the graceful restart case. Instead of stopping the server, we stop listening then explicitly stop the server. This refactor should exhibit the same behavior as today since we block on `stopListening()`, but this will allow for simpler refactoring of thrift call queueing logic in the future (by removing the `stopListening()` call and replacing it with a `startQueueingAndWaitForOutstandingCallsToFinish()` type of call).
In terms of layout, this consolidates all the stopping code into one function `startTakeoverShutdown()`. This function now returns the `TakeoverData` itself instead returning a future that is fulfilled after stopping the Thrift server. The TakeoverServer still communicates via `takeoverComplete`, but now that future is stored during `startTakeoverShutdown()` instead of being the return value of `performTakeoverShutdown()`. This also generally eliminates `TakeoverPromise`.
Reviewed By: simpkins
Differential Revision: D20744151
fbshipit-source-id: 60f0c273b4f3889b53586d79efd95bfb27256e1b
Summary:
D20846826 added a dependency from the fuse library to the telemetry code,
so the CMakeLists.txt file needs to be updated.
Reviewed By: genevievehelsel
Differential Revision: D21083715
fbshipit-source-id: 823cc2eb128808d0e807c6b91cacc9fd91cdad48
Summary:
We got a user request to add this output after a restart. This (` run "cd / && cd -" to
update your shell's working directory.`) is the output from eden doctor when it detects stale mount points. I don't think it hurts to add, especially since new customers may not make the connection/have the context to deduce that "programs" in this message also apply to the user's shell.
Reviewed By: kmancini
Differential Revision: D20971070
fbshipit-source-id: 3448651cae90b2b5c8c4d2432b946521cdc5a292
Summary: RSRoutingHandler is now added by default
Reviewed By: bithree
Differential Revision: D21058633
fbshipit-source-id: 7d7607a5c5c7c3a5ed70259496c5f036c6fa8e4c
Summary:
This refactors the `edenfsctl start` command so that we more clearly split the
functionality into two pieces:
* Starting EdenFS as a service
* Running EdenFS in the foreground
In most normal situations in production the `start`, `restart`, and `stop`
commands are used to manage running EdenFS as a service. In the future I
believe our service management logic will start to diverge a bit more on Linux
vs Mac vs Windows, and this should help isolate the service-management code a
bit more cleanly.
The foreground behavior is mainly only used by developers during testing and
during the integration tests. Several options like `--gdb` and `--strace` are
only allowed in foreground mode, and this refactoring makes that clearer. In
the future we may also want to further restrict this, to allow only
specifying additional custom arguments and a custom binary path when running
in foreground mode. However, for now I have not updated that as I believe
some of our integration tests may be exercising this behavior today.
This change also cleans up some of the platform-specific code, and lets them
share more of the logic to construct arguments for edenfs. With this change
`edenfsctl start --foreground` now works on Windows.
Reviewed By: pkaush
Differential Revision: D20833244
fbshipit-source-id: 0f09d59702d8b64ca8f4fedccbc30da1c858afb4
Summary:
Update the `edenfsctl` and `edenfs` code so that `edenfsctl stop` works on
Windows.
Reviewed By: pkaush
Differential Revision: D20854616
fbshipit-source-id: 9a51cdc1f1d1838c5089b945df1fb624a3eaedb7
Summary:
- Catch the Windows-specific exceptions types thrown when attempting to open
the socket if EdenFS is not running, and convert this to an EdenNotRunning
exception.
- Update EdenFS to write its pid to a separate file, in addition to the normal
lock file, and have `edenfsctl` read this file instead of the lock file on
Windows. Other processes cannot read the lock file while EdenFS is running
and holding the lock, so it is nice to have the pid written in an alternate
location.
Reviewed By: pkaush
Differential Revision: D20854615
fbshipit-source-id: 1c8e8f402eb17dd012d03e11fbb662f493d9362d
Summary:
Move the `start_systemd_service()` function from `daemon.py` to a separate
module. This allows `daemon.py` to be imported on Windows without causing any
of the systemd logic to be imported.
I kept this separate from the existing `systemd.py` module for now just to
help keep the dependencies a little bit cleaner. The `systemd.py` module
currently does not depend on any other logic from other edenfsctl modules.
Reviewed By: pkaush
Differential Revision: D20854612
fbshipit-source-id: dbdbf0fec90682e3b7d16a7532aefdd3f186bf80
Summary: I believe it would be nice to include the output of `eden redirect list` in `eden rage` since I've run into a few times this week where that was something that was asked for.
Reviewed By: chadaustin
Differential Revision: D20955632
fbshipit-source-id: 0d28688b3764a1bec8a9e92b72c166dddd18f591
Summary:
This sets up the counters that will allow us to expose metrics for
live FUSE requests in Eden top.
This will allow us to track the number of live FUSE requests and the duration
of the longest FUSE request. If the duration of the longest FUSE request has
become very long, the number of FUSE requests is stuck at some value or
increasingly growing these can indicate there is a problem with the Fuse Channel
or something effecting this.
This provides more insight beyond the metrics for imports since many of the
FUSE requests will not cause imports, so this monitors more functionality.
Further because all user interaction with the filesystem goes through FUSE,
these metrics will give a good overall indication weather Eden is performing
normally.
Reviewed By: chadaustin
Differential Revision: D20846826
fbshipit-source-id: 34d834d193833a3c91d95fbb87361ddd863f64ca
Summary: It's a little weird to have get batch returns a pair of `RelativePath` and `Hash`.
Reviewed By: chadaustin
Differential Revision: D20844545
fbshipit-source-id: c0c823e0aca6ea2326cb637226dc0de43fd4f3cd
Summary: This diff makes `HgProxyHash` moveable & copyable so we can return it from `getBatch` method instead returning a pair of `RelativePath` & `Hash`. See next diff for usage.
Reviewed By: chadaustin
Differential Revision: D20841434
fbshipit-source-id: c082c217f75d53bcbfb4853e2055d1458a040188
Summary:
Some of our types were vulnerable to the issue described in
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1008r0.pdf so
make all deleted default constructors explicit.
Reviewed By: simpkins
Differential Revision: D21008976
fbshipit-source-id: 5b21923f25121dabf4bb0ea55f94536fb3532e6b
Summary:
Clean up the last two remaining pyre-fixme comments in this file.
I believe these casts were needed at one point to make mypy happy, but it
looks like they are no longer necessary.
Reviewed By: genevievehelsel
Differential Revision: D21007030
fbshipit-source-id: 6933425d9fde8b86272c68b955faf21348453f68
Summary: gcc warned that these functions did not return with a value on all code paths.
Reviewed By: genevievehelsel
Differential Revision: D21004224
fbshipit-source-id: 3ccd38b19bb4f4c0dbe0b75b4d11f351b6690591
Summary:
Instead of manually enumerating every method defined in eden.thrift to
enable histogram statistics, use Thrift's own metadata.
Reviewed By: genevievehelsel
Differential Revision: D20929642
fbshipit-source-id: 5a54eb03e75fcd08cf653e2c53a78c8137ba9553
Summary:
Update the `edenfsctl clone` command to require that an explicit repository
path be passed in. Using the name of a repository listed in an EdenFS config
file is no longer supported.
Reviewed By: wez
Differential Revision: D20876459
fbshipit-source-id: 5e9fef11a2afa4cc48cb8a9bb5b874d2e7923f25
Summary:
We no longer use repository configs, so remove the `repository` subcommand
that supported adding and listing these configurations.
The main information that used to be included in the repository configuration
was the bind mount settings. This has since been replaced with the
`.eden-redirections` file that is placed directly in each repository.
Reviewed By: wez
Differential Revision: D20876462
fbshipit-source-id: cc7d8e6f0a6a2e04fbf3159417af41a44908b3a8
Summary:
Add a command line argument to `edenfsctl doctor` to cause it to only report
problems with the current EdenFS instance, and to skip checks for system-wide
problems or other running EdenFS processes.
This does refactor a fair amount of the top-level doctor logic to encapsulate
most of the state into an `EdenDoctor` class.
This also updates the integration tests to use this flag when invoking
`edenfsctl doctor`. Previously the integration tests could spuriously fail
due to other pre-existing problems on the system, or due to other EdenFS
instances that are currently being started or shut down by other tests running
in parallel.
Reviewed By: wez
Differential Revision: D20357521
fbshipit-source-id: 36640cc21e7bd79fbd300c4d2c7dbba127ec9170
Summary:
This code isn't really working and is giving out corrupted file content. Let's remove it for reading blobs.
We still cannot remove the C++ mercurial code entirely as we rely on it to import manifests. I tested on switching to the Rust manifest implementation but it wasn't working due to new pack file scanning.
Reviewed By: chadaustin
Differential Revision: D20938001
fbshipit-source-id: 9cc7b1ed9cd8228afd6fc5374cda001d1309a379
Summary:
The bind-mounts configuration has been ignored by EdenFS since D17236366.
This removes all CLI code for dealing with this config section.
Reviewed By: wez
Differential Revision: D20876460
fbshipit-source-id: 6b3f3552de25ee28fc0418a6aaec14446520203c
Summary:
As discussed in D20629833 metrics for live imports
like those for pending imports can help differentiate
where imports are in the process.
This can help give users more insight into what is
happening inside eden. Meaning that they can see
progress being made, and that eden is not stuck. But
also when eden is stuck these metics can indicate
that it is stuck fetching data from the mercurial servers.
This displays those metrics in `eden top` similar to those
for pending imports
Reviewed By: chadaustin
Differential Revision: D20630826
fbshipit-source-id: ab7368927d2832acbea89a82585fda5195ce03d1
Summary:
Since imports are now queued in `HgQueuedBackingStore`
the metrics for queued imports were broken
the queueing use to happen in `HgBackingStore`, but
now the queue happens before the imports get here,
in `HgQueuedBackingStore`, thus these metrics were
only measuring live imports
=> this moves the metrics for queued imports into
`HgQueuedBackingStore` so that they capture both
live and queued imports
**note**: since prefetching in `HgQueuedBackingStore`
is in progress, these metrics are only set-up for when
prefetching this is finished
Reviewed By: chadaustin
Differential Revision: D20714494
fbshipit-source-id: 392337aaa60b44b75184889e6c92cb5d2c2577cc
Summary:
As mentioned in D20629833, adding metrics for live
imports in `eden top` gives more transparency to the
imports process and makes identifying import related
issues easier. This is set up to expose metrics for live
imports like those for pending imports in `eden top`.
Similar to D20611728 exposing this via these counters
will log this data. Having this data persisted will allow
tracking the performance of imports, and does the set
up for more pro-active fixing of issues. Further we can
look back to see issues that are no longer occurring, but
still of interest.
This also refactors the registration code so that it requires
no copy pasting to add a new counter. Avoiding copy paste
errors when adding more counters and making it easier to
maintain.
Reviewed By: chadaustin
Differential Revision: D20630813
fbshipit-source-id: 8a7a2a0135c7b7a5cde960b84dcb434c6c99eaeb
Summary: Now that fb303's Thrift function histogram support is open sourced, remove the _WIN32 ifdefs.
Reviewed By: simpkins
Differential Revision: D20907360
fbshipit-source-id: bb72dda0c801ca1bba4866d26645148881630596
Summary:
When cloning a Mercurial repository, default to checking out the `master`
bookmark, if it exists. Continue using `.` in case the repository does not
have a `master` bookmark.
Reviewed By: pkaush
Differential Revision: D20876461
fbshipit-source-id: 57fa12e4c713bd50c15f59eb9281e0511c3cfe88
Summary:
The `tm_mon` field returned by `localtime_r()` has a range of 0 to 11.
We want to show human-readable month numbers of 1 to 12 in the fsck directory
name and log timestamps. Fix the formatting by adding 1 to the `tm_mon`
value.
Reviewed By: fanzeyi
Differential Revision: D20909591
fbshipit-source-id: 8625d09306b625e4e71dab9e0679fed3abc7bcf6
Summary:
Do not put a `README_EDEN.txt` file in the checkout root on Windows. On Linux
& Mac the EdenFS mount hides this directory, so the README file is not visible
while the checkout is mounted and running normally. However on Windows
anything present in this directory is visible to the user, so this results in
`README_EDEN.txt` incorrectly showing up in the checkout root.
Reviewed By: genevievehelsel
Differential Revision: D20929408
fbshipit-source-id: 9994524041f22fd8922c531f0185186b04c54821
Summary:
In the Windows main.cpp file, print an exception if one is thrown while
running EdenFS. This doesn't report any backtrace information, but at least
prints the exception message itself.
Previously if an exception was thrown EdenFS would exit with a non-zero status
code but the actual exception message wasn't printed anywhere.
Reviewed By: fanzeyi
Differential Revision: D20928827
fbshipit-source-id: f9397f9688ef25b38f23421213058c417ddefaf9
Summary:
Make sure the function that decides if we should use systemd always returns
False on non-Linux platforms, even if it is explicitly enabled via the config
file or environment variable. systemd is Linux-specific, and it doesn't make
sense to try and use it on other platforms.
Reviewed By: fanzeyi
Differential Revision: D20925344
fbshipit-source-id: cee67f607809da15f584de1eb12a2c4a243b0c91
Summary:
On Windows, skip the check that tests if our lock file is still valid.
This check did not work properly, as the `st_dev` and `st_ino` fields reported
by stat() are not reliable. The `st_dev` value reported could sometimes
change, causing EdenFS to incorrectly think the lock file was no longer valid
and then quit.
This check isn't really necessary on Windows, since Windows prevents the lock
file from being removed while we have it open.
Reviewed By: pkaush
Differential Revision: D20875410
fbshipit-source-id: d607a9b89bd018bb6aacfd4b58cb9ba7e9bf94b8
Summary: In order to speed up the starting of the new process in a graceful restart, flush and compact the local store. This is running while we are still serving mounts, so the user should not be greatly effected on the old process.
Reviewed By: simpkins
Differential Revision: D20625721
fbshipit-source-id: ef141f76fb7311492d97b120f762f36141f8967d
Summary:
* This adds a `EdenServer::recover()` method to start back up on unsuccessful takeover data send.
* On an unsuccessful ping, filfill the `shutdownPromise` with a `TakeoverSendError` continaing the constructed `TakeoverData`. After this `recover` function is called, `takeoverPromise_` is reset, `takeoverShutdown` is set to `false`, and the `runningState_` is set to `RUNNING`.
With taking over from the returned `TakeoverData`, the user will not encounter `Transport not connected` errors on recovery.
* This adds a `EdenServer::closeStorage()` method to defer closing the `backingStore_` and `localStore_` until after our ready handshake is successful.
* This defers the shutdown of the `PrivHelper` until a successful ready handshake.
I also update the takeover documentation here with the new logic (and fix some formatting issues)
Reviewed By: simpkins
Differential Revision: D20433433
fbshipit-source-id: f59e660922674d281957e80aee5049735b901a2c
Summary: This allows for fault injection into the TakeoverServer, which will be helpful when trying to mock a failed "ready" handshake (this will be used by injecting an error, not a timeout).
Reviewed By: simpkins
Differential Revision: D20527046
fbshipit-source-id: 24d493fdac620c759c98b050b6cec6b88587789a
Summary:
For graceful restart takeovers, we would like to implement an additional handshake. This handshake will occur right after the takeover data is ready to be sent to the client, but before actually sending it. This is to make sure the old daemon can recover in case of the client not being responsive (the client replies back to the server, and if no response is recieved in 5 seconds, the server will recover).
There are a few cases here:
* **Server sends ping (two cases discussed below)**
I introduced a new ProtocolVersion. Daemons with this change will now have ProtocolVersion4. The Server checks the max version of the client, and if this version is ProtocolVersion4, we know the client can listen for pings. So we will send the ping. Otherwise, we don't send a ping. With this, we will only send pings if we know the client will be listening for one. The case in which a client isn't listening is if we adopt this change and we downgrade past the change.
* **Server does not send ping and Client knows to listen for ping**
This will be a common case immediately after this change. The client will parse the sent data and check if it matches the "ready" ping, and if it doesn't, the client assumes the server simply sent the Takeover Data.
* **Server does not sends ping and Client doesn't know to listen for ping**
This is the case before this change.
Reviewed By: simpkins
Differential Revision: D20290271
fbshipit-source-id: b68e4df6264fb071d770671a80e28c90ddb0d3f2
Summary: This is actually missing from `HgQueuedBackingStore`. This diff fixes this and assign low priority to these prefetch requests.
Reviewed By: chadaustin
Differential Revision: D20655681
fbshipit-source-id: f3c92b358e16e980390ac7adcae27d41ae5a7277
Summary: Ergonomics improvements. This allows us to have type safety instead of dynamically check for types at runtime.
Reviewed By: chadaustin
Differential Revision: D20740318
fbshipit-source-id: d0a3255105bef22dd236695be9fc9c4220547b83
