Summary: Nothing should be running with this version, so this check is not needed anymore!
Reviewed By: chadaustin, wez
Differential Revision: D21513269
fbshipit-source-id: 698c9d84e87461cd0484ef997f44dc9169e87a25
Summary: add two methods calls as typehints (no real code change).
Reviewed By: zeroxoneb
Differential Revision: D21337646
fbshipit-source-id: 8079883f7f989251965d3308c5374f097023b57a
Summary:
- Added uptime field to DaemonInfo thrift struct
- Created startTime member variable in EdenServer
- Made appropriate refactoring changes to EdenMain and EdenServer
- Changed main.py and util.py to use the new uptime value
Reviewed By: genevievehelsel
Differential Revision: D21471140
fbshipit-source-id: 8868de667dfb95de93e3e71b90c0412fb3825388
Summary:
Migrating buck integration to use configurations, which is more performant and is the primary supported way to run Pyre.
Mypy is no longer supported at Facebook, so mypy buck integration is covered in this migration as well.
Configuration target automatically expanded to include all subtargets, expanding type coverage while introducing no more than 2 fixmes per file.
Reviewed By: dkgi
Differential Revision: D21474723
fbshipit-source-id: 2501af8250b4998fe331350b80c9dc8b4ba2dd1a
Summary:
Sometimes, in shell one-liners, it's convenient to insert the current
eden pid. We print the pid as part of `eden status` (like nginx and
many other daemonizing programs), but that is not useful for shell
substitution. So a pid subcommand that succeeds and prints the pid to
stdout if running and fails if edenfs is not running.
An example use would be `strace -fyp $(eden pid)`
Reviewed By: genevievehelsel
Differential Revision: D21397118
fbshipit-source-id: 06010150557ec882b452180665b84452fa06ebbd
Summary:
The information available in `eden top` is growing and as we add more and
more to eden top, we want to make sure that eden top users can understand
what `eden top` is telling them. Particularly if we want eden end users to be
able to debug themselves, providing them enough context on the
metrics eden top provides is important.
The help page explains what data is included, why this data is helpful and
when this data indicates a concern.
Reviewed By: wez
Differential Revision: D20974262
fbshipit-source-id: caaef70e76d283968bee8533c7c34d63a95c5490
Summary:
this is preperation for adding a help page to eden top (D20974262)
This allows adding a scrollable section to `eden top`. This is particuarly needed
so that the help page is readable when it does not fit all in one screen. This
also allows scrolling the process table on the main page to see all the
processes listed (this is added here).
Reviewed By: chadaustin
Differential Revision: D20974245
fbshipit-source-id: 402409efdc451db32e7a267b0750204f2322dc4f
Summary:
this is preparation for adding a help page to eden top (D20974262)
This moves all code for writing to the screen to its own class so that
it is not scattered through out the rest of the eden top logic. This is
primarlity so that it is easier to make changes the screen writing logic to
allow scrolling, which is needed so that the full help screen can be read
when it takes up more space than is avaiable.
Reviewed By: chadaustin
Differential Revision: D20965954
fbshipit-source-id: c2690a293579706605d26be9d3bbe491104e6847
Summary:
Our use of the term "client" to refer to a checkout is
deprecated. Rename some internal functions that use the term client.
Reviewed By: simpkins
Differential Revision: D21395159
fbshipit-source-id: fa96ba593f53b493e5ae816fa686f333a132c232
Summary:
The name of a checkout is not a public concept, so refer to the
checkout's path.
Reviewed By: genevievehelsel
Differential Revision: D21393208
fbshipit-source-id: 4c014a6f65515f4632f2dffe5d563d0ee859dda0
Summary:
When I want FUSE stats, I never run `eden stats io`. To unify the
command language, rename `io` to `fuse` and `latency` to
`fuse-latency`, to match `thrift` and `thrift-latency`.
Leave aliases in place in the off chance someone cares.
Reviewed By: genevievehelsel
Differential Revision: D21392931
fbshipit-source-id: 843c1c85ea0aa162ba167f251f0f2cde5a389e72
Summary:
`eden stats memory` was never useful because it shows system
statistics which you can get better from `top`, `htop`, or `atop`.
Reviewed By: genevievehelsel
Differential Revision: D21392737
fbshipit-source-id: 010021b8a97bd8ba8ac289d906acc3c3ecd10768
Summary:
Addressing issues simpkins brought up on D21207287 when we upgraded and introduced some pyre bugs.
Temporarily upgrading just this project, once we resolve some sandcastle capacity issues we'll release this via another global upgrade in fbcode.
Reviewed By: simpkins
Differential Revision: D21316793
fbshipit-source-id: f0c79f53d97f7182e7d8fe6e081c58ef53ce0c9a
Summary:
D21316793 is blocked from landing because there are a few targets in eden that are running pyre via buck targets integration.
We can't do custom version overrides for projects that are using a mix of local configurations and buck integration, because buck doesn't provide an interface for setting the equivalent pyre version override.
We're moving away from buck targets integration for pyre across the board, and I've run a codemod over the project to clean up all of the buck typing integration (including some residual mypy) as well as updated type ignores / fixmes accordingly.
Let me know if you have any concerns; upon skimming it looks like most changes are either converting `type: ignore`s into fixmes, or removing `type: ignores`.
Reviewed By: dkgi
Differential Revision: D21343093
fbshipit-source-id: 5ee1436377eb526c0a679fb821c42e07cbca52a5
Summary: I thought it would be helpful to introduce and `eden uptime` command, especially with automated graceful restart on the way. This prints it in human readable format, later on if for some reason automation would like to use this, a flag could be added that allows for custom formatting. Also, this can be added to `eden rage` output later.
Reviewed By: chadaustin
Differential Revision: D21260800
fbshipit-source-id: 3f9a4f8d6264dfc38bd15c024a0209f7eeb912fa
Summary:
D21074489 adds metrics for pending FUSE requests, this cleans up the display for
pending requests.
This removes the max duration of pending requests for FUSE requests since this
data is not available (it is not measured by the FUSE library).
Reviewed By: chadaustin
Differential Revision: D21074746
fbshipit-source-id: e5585ec091aa5fd5499deee2d8be89f47f769a6a
Summary:
This adds the number and duration of FUSE requests to eden top.
The purpose of this metric is to provide a highlevel indicator of
the health of eden and the fuse interface. Because FUSE is the
interface through which the user mostly interacts with the file
system it these metrics are reflective of edens overall
responsiveness.
The metrics are collected per mount, but we only display the
aggregate since this serves as an overall summary metric.
These metrics are displayed similarly to the import metrics,
for consistency. There is not a metric for the max duration
of pending FUSE requests D21074746 cleans this up so that
the NA is removed from the display.
Reviewed By: chadaustin
Differential Revision: D20928603
fbshipit-source-id: ce78f4274203c2302e7595ae2ef601e18893db38
Summary:
This makes it a bit easier to write integration tests
that run an isolated EdenFS instance complete with its own
unique scratch path environment.
Reviewed By: genevievehelsel
Differential Revision: D21258455
fbshipit-source-id: 3ea13b6337bb8bb58a237e67488b116c423bbe6c
Summary:
Now that we have globFiles available via thrift on windows,
we can use it to implement eden prefetch on windows, rather than the
significantly slower `hg prefetch`.
Reviewed By: pkaush
Differential Revision: D20781187
fbshipit-source-id: f6cd37a2a504d07f03be2695a69a72ddf7a62593
Summary:
This adds a new `EdenTestCaseBase` class to serve as the base class across a
number of our integration tests and some of our CLI tests.
The main goal of this is to allow eliminating many of the annoying `*Mixin`
classes used in a lot of our integration tests. These mixin classes are
annoying since they result in complicated multiple inheritance, and it can be
tricky to ensure that the method resolution order behaves the way you want.
The systemd tests in particular use a lot of mixins, which gets complicated.
These mixin classes are also awkward from a Python typing perspective, and the
systemd tests end up resorting to just declaring different APIs in several
places when `typing.TYPE_CHECKING` is True.
The fact that `EdenTestCaseBase` has a `contextlib.ExitStack` member variable
should make it easier for us to eliminate these mixins moving forward: rather
than using mixins that use inheritance and assume a `self.cleanUp()` method
exists, we can transition this code to standalone functions or context
managers, and they can take the `ExitStack` variable as an argument if
necessary.
Reviewed By: wez
Differential Revision: D21084097
fbshipit-source-id: 77ee457b7debe6f584f630e3e30f79fe634a2026
Summary:
Update most locations in edenfsctl to report the version number that was built
into the edenfsctl binary at build time, rather than querying the RPM database
for the installed RPM version. The RPM behavior only works on to RedHat-based
Linux systems, and the currently running process doesn't necessarily have to
have come from the RPM.
The one place where we do still attempt to print the RPM version is in the
`edenfsctl rage` report, when running on Linux.
Reviewed By: wez
Differential Revision: D21000168
fbshipit-source-id: 0fb747e71b6950d74f22c458efa0dfcbd45270bd
Summary:
On Windows, paths components are usually separated by '\', and since the
repository path is stored in a toml file, whatever character is after a '\',
will be escaped. In my case, this is followed by U (for C:\Users), and thus
toml expects the next characters to be an escaped unicode. That's obviously
not the case and thus EdenFS fails to parse the config, preventing me from
cloning fbsource.
Since Windows is perfectly fine with '/' as path separator, let's just
replace '\' with '/'.
The underlying bug appears to be in the toml Python code: https://github.com/uiri/toml/issues/280
Manually trying some random path is pretty conclusive:
(Pdb) toml.dumps({'foo': 'c:\\Users\\wez'})
'foo = "c:\\\\Users\\\\wez"\n'
(Pdb) toml.dumps({'foo': 'c:\\Users\\xavier'})
'foo = "c:\\Users\\xavier"\n'
Reviewed By: chadaustin
Differential Revision: D21143545
fbshipit-source-id: 448471da12c253dd37680f6a28251a1e69850920
Summary:
As a top layer of reliability in graceful restart, we'd like the CLI to enforce that something has started after a graceful restart call. There are a few things that can happen
1) The restart short circuts before attempting to shutdown (version mismatch etc). The old process will continue (`success == False`)
2) The restart is successful, the new process will continue (`success == True`)
3) The restart is not successful (failed ping), the old process will recover (`success ==
False`)
(note here that the `success` field is the return code of the `StartupLogger` process, not the edenfs process itself).
In case any of these fail, we'd like a final line of defence with the CLI.
First, in the case of a successful start from the `StartupLogger` report, we consider this successful and exit. If we do not have a successful (`success == False`) graceful restart, we will wait for the new process to recover (since the old process will go from `ALIVE` to `STOPPED` to `ALIVE`). I do not expect recovery to take long since we are only remounting, not waiting on RocksDB. Here though, we give a fairly long timeout (1hr) to ensure we're not stuck on an in process thrift call, since we would not like to force kill the process if we're writing to the overlay.
If we time out waiting, we will just kick off a force restart and assume the old process is stuck.
Reviewed By: simpkins
Differential Revision: D20673460
fbshipit-source-id: 215e06fa8aa76dbe5dae6602f82ce5f05968e538
Summary:
We got a user request to add this output after a restart. This (` run "cd / && cd -" to
update your shell's working directory.`) is the output from eden doctor when it detects stale mount points. I don't think it hurts to add, especially since new customers may not make the connection/have the context to deduce that "programs" in this message also apply to the user's shell.
Reviewed By: kmancini
Differential Revision: D20971070
fbshipit-source-id: 3448651cae90b2b5c8c4d2432b946521cdc5a292
Summary:
This refactors the `edenfsctl start` command so that we more clearly split the
functionality into two pieces:
* Starting EdenFS as a service
* Running EdenFS in the foreground
In most normal situations in production the `start`, `restart`, and `stop`
commands are used to manage running EdenFS as a service. In the future I
believe our service management logic will start to diverge a bit more on Linux
vs Mac vs Windows, and this should help isolate the service-management code a
bit more cleanly.
The foreground behavior is mainly only used by developers during testing and
during the integration tests. Several options like `--gdb` and `--strace` are
only allowed in foreground mode, and this refactoring makes that clearer. In
the future we may also want to further restrict this, to allow only
specifying additional custom arguments and a custom binary path when running
in foreground mode. However, for now I have not updated that as I believe
some of our integration tests may be exercising this behavior today.
This change also cleans up some of the platform-specific code, and lets them
share more of the logic to construct arguments for edenfs. With this change
`edenfsctl start --foreground` now works on Windows.
Reviewed By: pkaush
Differential Revision: D20833244
fbshipit-source-id: 0f09d59702d8b64ca8f4fedccbc30da1c858afb4
Summary:
Update the `edenfsctl` and `edenfs` code so that `edenfsctl stop` works on
Windows.
Reviewed By: pkaush
Differential Revision: D20854616
fbshipit-source-id: 9a51cdc1f1d1838c5089b945df1fb624a3eaedb7
Summary:
- Catch the Windows-specific exceptions types thrown when attempting to open
the socket if EdenFS is not running, and convert this to an EdenNotRunning
exception.
- Update EdenFS to write its pid to a separate file, in addition to the normal
lock file, and have `edenfsctl` read this file instead of the lock file on
Windows. Other processes cannot read the lock file while EdenFS is running
and holding the lock, so it is nice to have the pid written in an alternate
location.
Reviewed By: pkaush
Differential Revision: D20854615
fbshipit-source-id: 1c8e8f402eb17dd012d03e11fbb662f493d9362d
Summary:
Move the `start_systemd_service()` function from `daemon.py` to a separate
module. This allows `daemon.py` to be imported on Windows without causing any
of the systemd logic to be imported.
I kept this separate from the existing `systemd.py` module for now just to
help keep the dependencies a little bit cleaner. The `systemd.py` module
currently does not depend on any other logic from other edenfsctl modules.
Reviewed By: pkaush
Differential Revision: D20854612
fbshipit-source-id: dbdbf0fec90682e3b7d16a7532aefdd3f186bf80
Summary: I believe it would be nice to include the output of `eden redirect list` in `eden rage` since I've run into a few times this week where that was something that was asked for.
Reviewed By: chadaustin
Differential Revision: D20955632
fbshipit-source-id: 0d28688b3764a1bec8a9e92b72c166dddd18f591
Summary:
Clean up the last two remaining pyre-fixme comments in this file.
I believe these casts were needed at one point to make mypy happy, but it
looks like they are no longer necessary.
Reviewed By: genevievehelsel
Differential Revision: D21007030
fbshipit-source-id: 6933425d9fde8b86272c68b955faf21348453f68
Summary:
Update the `edenfsctl clone` command to require that an explicit repository
path be passed in. Using the name of a repository listed in an EdenFS config
file is no longer supported.
Reviewed By: wez
Differential Revision: D20876459
fbshipit-source-id: 5e9fef11a2afa4cc48cb8a9bb5b874d2e7923f25
Summary:
We no longer use repository configs, so remove the `repository` subcommand
that supported adding and listing these configurations.
The main information that used to be included in the repository configuration
was the bind mount settings. This has since been replaced with the
`.eden-redirections` file that is placed directly in each repository.
Reviewed By: wez
Differential Revision: D20876462
fbshipit-source-id: cc7d8e6f0a6a2e04fbf3159417af41a44908b3a8
Summary:
Add a command line argument to `edenfsctl doctor` to cause it to only report
problems with the current EdenFS instance, and to skip checks for system-wide
problems or other running EdenFS processes.
This does refactor a fair amount of the top-level doctor logic to encapsulate
most of the state into an `EdenDoctor` class.
This also updates the integration tests to use this flag when invoking
`edenfsctl doctor`. Previously the integration tests could spuriously fail
due to other pre-existing problems on the system, or due to other EdenFS
instances that are currently being started or shut down by other tests running
in parallel.
Reviewed By: wez
Differential Revision: D20357521
fbshipit-source-id: 36640cc21e7bd79fbd300c4d2c7dbba127ec9170
Summary:
The bind-mounts configuration has been ignored by EdenFS since D17236366.
This removes all CLI code for dealing with this config section.
Reviewed By: wez
Differential Revision: D20876460
fbshipit-source-id: 6b3f3552de25ee28fc0418a6aaec14446520203c
Summary:
As discussed in D20629833 metrics for live imports
like those for pending imports can help differentiate
where imports are in the process.
This can help give users more insight into what is
happening inside eden. Meaning that they can see
progress being made, and that eden is not stuck. But
also when eden is stuck these metics can indicate
that it is stuck fetching data from the mercurial servers.
This displays those metrics in `eden top` similar to those
for pending imports
Reviewed By: chadaustin
Differential Revision: D20630826
fbshipit-source-id: ab7368927d2832acbea89a82585fda5195ce03d1
Summary:
When cloning a Mercurial repository, default to checking out the `master`
bookmark, if it exists. Continue using `.` in case the repository does not
have a `master` bookmark.
Reviewed By: pkaush
Differential Revision: D20876461
fbshipit-source-id: 57fa12e4c713bd50c15f59eb9281e0511c3cfe88
Summary:
Do not put a `README_EDEN.txt` file in the checkout root on Windows. On Linux
& Mac the EdenFS mount hides this directory, so the README file is not visible
while the checkout is mounted and running normally. However on Windows
anything present in this directory is visible to the user, so this results in
`README_EDEN.txt` incorrectly showing up in the checkout root.
Reviewed By: genevievehelsel
Differential Revision: D20929408
fbshipit-source-id: 9994524041f22fd8922c531f0185186b04c54821
Summary:
Make sure the function that decides if we should use systemd always returns
False on non-Linux platforms, even if it is explicitly enabled via the config
file or environment variable. systemd is Linux-specific, and it doesn't make
sense to try and use it on other platforms.
Reviewed By: fanzeyi
Differential Revision: D20925344
fbshipit-source-id: cee67f607809da15f584de1eb12a2c4a243b0c91
Summary: This has been deprecated for a long time in favor of `eden debug logging`
Reviewed By: genevievehelsel
Differential Revision: D20876084
fbshipit-source-id: 0d9bf71e55aaffde7ae9f061def0adf8312c2411
Summary:
Add methods to check if a process ID is alive, and if it looks like an EdenFS
process.
This also adds an initial version of ProcUtils for Windows, and implements
these two methods on Windows. I have moved parts of the `winproc.py` module
to the new `proc_utils_win.py` module, to help better manage dependencies
between our modules. This keeps all of the Windows-specific `ctypes` code
together in `proc_utils_win.py`. The functionality that is still left on
`winproc.py` depends on `config.py`, and the `proc_utils` code should not
depend on `config.py` to help avoid circular dependencies.
Reviewed By: wez
Differential Revision: D20833245
fbshipit-source-id: 43e9b6dd1b520dcb6b2da7701de885058f0f7ea2
Summary:
This renames the `ProcessFinder` class to `ProcUtils`, in preparation for
adding some more generic process management utilities into this class.
This class was originally just used for finding stale processes in
`edenfsctl doctor`. However, it seems like a good place to consolidate all
platform-specific process querying logic. We are currently need
platform-specific querying to tell if specific process IDs are still running
for `edenfsctl status` and `edenfsctl stop`, and we also already have some
Windows-specific spawning logic in `edenfsctl start`. I plan to move all of
that logic into this `proc_utils.py` module.
This renames `ProcessFinder` to `ProcUtils` just to better reflect its
expanded purpose.
Reviewed By: genevievehelsel
Differential Revision: D20833243
fbshipit-source-id: 8d33460b7468d877bc327e855af5c620ac5b80a7
Summary:
Update out platform checks to use `sys.platform` instead of `os.name`
Python type checkers (mypy and Pyre) currently understand checks against
`sys.platform`, and know that code guarded by these checks is platform
specific. They don't understand the `os.name` checks.
This code should behave the same, but allows type checkers to do a better job
checking this code.
Note that there is also `platform.system()`, but this has drawbacks that it
apparently does runtime-checks to determine the platform, whereas `sys.platform`
is baked-in at compile time. Additionally, the typecheckers do not appear to
support checking based on `platform.system()` for now. There is an open
feature request for this for mypy, but it is not implemented yet.
One caveat for `sys.platform` is that the results on Linux are not consistent
across Python versions: older versions of Python used to report `linux2` while
new versions report simply `linux`.
Reviewed By: chadaustin
Differential Revision: D20830149
fbshipit-source-id: d173e41f1ae84951a84b87e2dc05787fe8b01407
Summary:
Update the `edenfsctl.exe` binary to find `edenfs.exe` relative to itself.
This ensures that when you run `edenfsctl.exe` from a development build that it
finds the associated development build of `edenfs.exe`, rather than the current
system-installed version.
Reviewed By: wez
Differential Revision: D20821419
fbshipit-source-id: 29ff0b587800f1b76c0b729512bc19f5aef648a7
Summary:
As discussed in D20611704, the duration of time that imports are queued for is a strong indicator of the health of the import process. Adding it to `eden top` will help give users insight into what eden is doing when it hangs for a long time or when there is an issue, indicate whether that issue is due to the import process.
Because it is a strong indicator (specifically stronger than the number of imports) for when there is an issue effecting imports, we use it as a warning signal to the user, meaning we display warning colors when the duration of the longest import is long enough to warrant concern.
We have 2 levels of concern to differentiate when there may be something strange happening and when there is almost definitely something strange happening. These correspond to displaying the import metrics in yellow and red respectively. The current threshold for something strange possibly happing is imports taking longer than 10s and something strange definitely happening is imports taking longer than 30s.
Reviewed By: chadaustin
Differential Revision: D20627754
fbshipit-source-id: 7931d2c0ab53952c1291fd1d411dad14d142734b
Summary:
When there is an issue in the import process, there is not an easy way to detect this currently. As a first step towards making it possible to detect when there is an issue here, we expose the number if imports pending to `eden top`.
If there is a problem blocking imports from happening, then they will start to queue up and the number of pending imports will keep growing. When there are a lot of pending imports it indicates that there may be a problem in the import process.
note: there are cases when there are many pending imports under normal operation. It is common to batch fetch a bunch of files at once, and submitting all these prefetches at once will mean a lot of pending imports though there is no problem. Thus number of pending imports only indicates there *may* be an issue in the import process. (Thus we couple it with the duration of the longest import D20627754 for stronger signal)
The other value in showing the number of pending imports is giving users more transparency into eden. When there is a big batch of imports this may leave the user waiting a while, and this gives a way for them to know why. They can see that eden is making progress rather than being stuck.
Reviewed By: fanzeyi
Differential Revision: D20527273
fbshipit-source-id: 59e0fdff6e7b154b409dfab385b4fdb02062b066
Summary:
Update the LinuxProcessFinder code to ignore the command line arguments when
looking for the state directory, and instead only look at the open file
descriptors.
The path found from the command line arguments could be incorrect: if the
EdenFS state directory had been deleted or renamed the information extracted
from the command line would be incorrect.
The information found by examining the open file descriptors will point to the
new directory location if the state directory has been renamed. We can also
tell if the lock file had been deleted in this case as well. This adds
another field to the `EdenFSProcess` class to indicate if the lock file is no
longer valid.
Reviewed By: wez
Differential Revision: D20638621
fbshipit-source-id: e9db1390f97baf50ad3246de4a964eaaf92622e7
Summary: Enable the CLI tests on macOS when run with Buck.
Reviewed By: wez
Differential Revision: D20524347
fbshipit-source-id: cf3e302256b6b0e6958999cf83c5be5d48f65907
