Summary:
Sometimes you only want to trace writes into the EdenFS checkout, so
add the ability to run `eden strace` with `--reads`, `--writes`, or
both in order to see only those events.
Reviewed By: wez
Differential Revision: D24468539
fbshipit-source-id: a1b3c730987cf86ce3d39952c6a5e9c5edaddfc2
Summary:
Now that Linux/macOS startup no longer uses fork it becomes trivial to share
the same code with Windows. This improves Windows startup in several different ways:
- `edenfsctl start` now displays the status of the start process in the console
- `edenfsctl start` no longer has an arbitrary timeout after which it reports
a timeout even though EdenFS is still starting up.
This also kills a bunch of Windows specific code that is no longer needed.
Reviewed By: fanzeyi
Differential Revision: D24393690
fbshipit-source-id: 28100aec96da81c92d5b592353edceed332e2364
Summary:
Add an `eden strace` command that subscribes to all FUSE events for
the specified mount (or current directory), and streams them to stdout
with response latency measurements and the requesting process.
If there are any pending requests at the time `eden strace` is run,
they are printed first.
Reviewed By: wez
Differential Revision: D24038978
fbshipit-source-id: 59a2112dbdb1708571d12b04bdccaf9eca725cf7
Summary:
Buck-out can contain some files that we wont have the ability to read on mac os
(like .Spotlight-V100 and .Trashes). This means `eden du` will error out on
reading them.
We handle FileNotFound but not permission errors, lets treat them the same.
Reviewed By: genevievehelsel
Differential Revision: D24396252
fbshipit-source-id: 31e2ecfeff1783ae9c3479f59335d95a0d9b6de7
Summary: Add option to redirect the eden rage output to stderr
Reviewed By: chadaustin
Differential Revision: D24210031
fbshipit-source-id: 736af81de6a41e49c2459a1cd8124123d6928323
Summary:
There's a bug in Thrift-py3 streaming support, where interrupting
iterating over an async stream leaves Thrift objects in a broken
state. Futures get dropped (and warnings are printed to the console)
but the destructors hang. Don't even try to garbage collect the heap
in that case.
Reviewed By: genevievehelsel
Differential Revision: D24032229
fbshipit-source-id: 5f16667fe6cfd27de1b39cc2974028729e18b214
Summary: To be more Windows compatable, we should move away from using `sh -c`. We don't use shell=True because that is susceptible to shell injection. As long as we don't close stdin until we're done, using Popen raw should be fine.
Reviewed By: xavierd
Differential Revision: D24151425
fbshipit-source-id: c0bcc883af948491862e8ce0cee56bcbe98e04f1
Summary:
On Windows, the du command line application doesn't exist, thus we cannot use
it. Instead, we can simply re-implement the du functionality in Python and use
that on all platforms.
Reviewed By: chadaustin
Differential Revision: D24030269
fbshipit-source-id: e86c1bcdeac7eeca70201f6fde48c20ef7e305a6
Summary:
Prefetch profile can now be used in automation to register profiles and to fetch
them. Since it will be used out side of the team it seems like this should be
moved out from under debug.
Reviewed By: genevievehelsel
Differential Revision: D23773337
fbshipit-source-id: b7f646ec202653233ed82528380d91f583a8341e
Summary:
Assuming the format for active prefetch profiles as discussed here:
https://fb.quip.com/v8yFAzeGx8TU
This allows the cli to read in and write out active profiles, allowing the cli
to manipulate them in the following changes. This will allow tools to activate
profiles useful for that workflow. Additionally, it will allow us (and maybe
users) to see/remove a users active profiles easily when investigating issues.
Reviewed By: genevievehelsel
Differential Revision: D23517411
fbshipit-source-id: 8ab18a1aeccf8dcdd04aca826d61070fa0aedd33
Summary:
Rather than switch all of Eden at once to the thrift-py3 client,
rename get_thrift_client to get_thrift_client_legacy so uses of the
new client can be introduced piecemeal.
(I did try migrating everything at once but it's been quite painful.)
Reviewed By: fanzeyi
Differential Revision: D22423399
fbshipit-source-id: 9e6d938b90fff9fc3266ba20bc77e880e7f5b1aa
Summary:
added `record-profile` and `finish-profile` subcommands to `eden prefetch`
`eden prefetch record-profile` triggers `startRecordingFetch()` in `HgQueuedBackingStore`. `eden prefetch finish-profile` triggers `stopRecordingFetch()` in `HgQueuedBackingStore` and write the returned fetched file names to a text file. We can use the output file to predict what files are fetched for frequently used commands.
Reviewed By: chadaustin
Differential Revision: D22797896
fbshipit-source-id: ff240829b4fb3c47093279a1f2bd453009ff5f73
Summary: While testing something for another change, I came across this overlooked typo.
Reviewed By: wez
Differential Revision: D22894060
fbshipit-source-id: 8aa48ef5da714650c974adcf8a34a542fdd4ed9e
Summary:
Rather than dynamically allocating an event loop in the systemd async
code, make all the corresponding functions async, so the caller is
responsible for threading an event loop down.
Reviewed By: genevievehelsel
Differential Revision: D21894106
fbshipit-source-id: 398c769c30c85a3bb210dbc209f34f9f7336996c
Summary: In advance of peppering async everywhere, allow subcmd's run method to be async.
Reviewed By: genevievehelsel
Differential Revision: D21892187
fbshipit-source-id: f611faacf95649d8bb5588aeefc4546bd5f63984
Summary:
This diff updated the `eden du` command. Instead of printing
```Reclaim space from the LFS cache directory by running:
hg -R /data/users/akushner/eden-repos/opsfiles gc
Reclaim space from the LFS cache directory by running:
hg -R /data/users/akushner/eden-repos/megarepo_test gc
Reclaim space from the LFS cache directory by running:
hg -R /data/users/akushner/eden-repos/instagram-server_test gc
```
We now print "Reclaim.." only once and put available commands together
```
LFS cache detected in backing repo. To reclaim space from the LFS cache directory, run:
hg -R /data/users/akushner/eden-repos/opsfiles gc
hg -R /data/users/akushner/eden-repos/megarepo_test gc
hg -R /data/users/akushner/eden-repos/instagram-server_test gc
```
and did the same for working copy under backing repositories.
Reviewed By: fanzeyi
Differential Revision: D22038555
fbshipit-source-id: c19869db6f91e90a0627b3bb4b4f7e95142ae198
Summary: This diff fixed small problems in previous `eden du` summary printing and added a reminder of option `--clean` to users.
Reviewed By: fanzeyi
Differential Revision: D22038552
fbshipit-source-id: 7dbac858b74833e59d8d3f1c6fb1f5d8944935d0
Summary: In its current state, FORCE is a restart mode that cannot be combined with other modes (specifically `graceful`). Removing this coupling from `force` will allow us to at first attempt a graceful restart, but if the daemon is not healthy, then do a force restart, all within one call.
Reviewed By: wez
Differential Revision: D21873166
fbshipit-source-id: 079a777ff6e7b219198cbef15335244b9865f5f3
Summary:
This diff fixes `eden rage` command on Windows.
`eden doctor` and `eden redirect` related rage items are disabled as they are not implemented on Windows yet.
Reviewed By: chadaustin
Differential Revision: D21962673
fbshipit-source-id: 14399a89c26eead8fe5114e0a3764644ee86a1ed
Summary:
The Python 2-and-3 Thrift API is sort of deprecated and does not
handle binary data in `binary` fields. In advance of migrating to the
modern Python 3 API, remane eden.thrift to eden.thrift.legacy.
Reviewed By: fanzeyi
Differential Revision: D21889697
fbshipit-source-id: a745ee8977999acbfb383a4edebe81d8deb1734e
Summary: This diff updated `eden du` to display a summary and display warnings and cleaning information with colors.
Reviewed By: kmancini
Differential Revision: D21885051
fbshipit-source-id: be127b81c92bea1051a80715682cdbccf22f22e3
Summary: This diff updated `eden du` to display an aggregated result of all mounts instead of showing all details for each mount, as users generally just want to reduce disk usage but don't really care about details.
Reviewed By: genevievehelsel
Differential Revision: D21877178
fbshipit-source-id: dde43e51e96a5c2569c9fe21ab06cc7ea4295866
Summary:
I am planning to start migrating Eden's CLI to the new Python 3 Thrift
implementation. In preparation, slightly clean up the interface and
implementation of our Python 2 Thrift wrapper.
Reviewed By: genevievehelsel
Differential Revision: D21854539
fbshipit-source-id: d398dd3f324c12288871cf0c9db41e64ed4cf7ed
Summary: This diff made `eden du` able to run under any directory and gives statistics for every EdenFS mount on disk by default.
Reviewed By: fanzeyi
Differential Revision: D21846894
fbshipit-source-id: 2d421db8a4a0202419aa7a41e620d92f6ebdea2e
Summary:
On Windows, if edenfsctl is started by pythonw, sys.stdout will be None,
check if it is before calling isatty on it.
Reviewed By: fanzeyi
Differential Revision: D21732282
fbshipit-source-id: f0a63094f2c53b026c388f4e5b58433be4397ba9
Summary:
1. Add directory existence checking logic before checking "legacy_bind_mounts_dir"
2. Still print the disk consumption returned by the du command when it fails with an error invoking du -skxc
Reviewed By: chadaustin
Differential Revision: D21683421
fbshipit-source-id: 531b6b289e9ffa445ed691611c9cbf22f393e742
Summary:
This diff is auto-generated to upgrade the Pyre version and suppress errors in eden. The upgrade will affect Pyre local configurations in the following directories:
```
eden
```
Differential Revision: D21687853
fbshipit-source-id: baf0d9bc33f86da63ea289690faca6cf4d566588
Summary:
- Added uptime field to DaemonInfo thrift struct
- Created startTime member variable in EdenServer
- Made appropriate refactoring changes to EdenMain and EdenServer
- Changed main.py and util.py to use the new uptime value
Reviewed By: genevievehelsel
Differential Revision: D21471140
fbshipit-source-id: 8868de667dfb95de93e3e71b90c0412fb3825388
Summary:
Sometimes, in shell one-liners, it's convenient to insert the current
eden pid. We print the pid as part of `eden status` (like nginx and
many other daemonizing programs), but that is not useful for shell
substitution. So a pid subcommand that succeeds and prints the pid to
stdout if running and fails if edenfs is not running.
An example use would be `strace -fyp $(eden pid)`
Reviewed By: genevievehelsel
Differential Revision: D21397118
fbshipit-source-id: 06010150557ec882b452180665b84452fa06ebbd
Summary:
Our use of the term "client" to refer to a checkout is
deprecated. Rename some internal functions that use the term client.
Reviewed By: simpkins
Differential Revision: D21395159
fbshipit-source-id: fa96ba593f53b493e5ae816fa686f333a132c232
Summary:
The name of a checkout is not a public concept, so refer to the
checkout's path.
Reviewed By: genevievehelsel
Differential Revision: D21393208
fbshipit-source-id: 4c014a6f65515f4632f2dffe5d563d0ee859dda0
Summary: I thought it would be helpful to introduce and `eden uptime` command, especially with automated graceful restart on the way. This prints it in human readable format, later on if for some reason automation would like to use this, a flag could be added that allows for custom formatting. Also, this can be added to `eden rage` output later.
Reviewed By: chadaustin
Differential Revision: D21260800
fbshipit-source-id: 3f9a4f8d6264dfc38bd15c024a0209f7eeb912fa
Summary:
Now that we have globFiles available via thrift on windows,
we can use it to implement eden prefetch on windows, rather than the
significantly slower `hg prefetch`.
Reviewed By: pkaush
Differential Revision: D20781187
fbshipit-source-id: f6cd37a2a504d07f03be2695a69a72ddf7a62593
Summary:
Update most locations in edenfsctl to report the version number that was built
into the edenfsctl binary at build time, rather than querying the RPM database
for the installed RPM version. The RPM behavior only works on to RedHat-based
Linux systems, and the currently running process doesn't necessarily have to
have come from the RPM.
The one place where we do still attempt to print the RPM version is in the
`edenfsctl rage` report, when running on Linux.
Reviewed By: wez
Differential Revision: D21000168
fbshipit-source-id: 0fb747e71b6950d74f22c458efa0dfcbd45270bd
Summary:
As a top layer of reliability in graceful restart, we'd like the CLI to enforce that something has started after a graceful restart call. There are a few things that can happen
1) The restart short circuts before attempting to shutdown (version mismatch etc). The old process will continue (`success == False`)
2) The restart is successful, the new process will continue (`success == True`)
3) The restart is not successful (failed ping), the old process will recover (`success ==
False`)
(note here that the `success` field is the return code of the `StartupLogger` process, not the edenfs process itself).
In case any of these fail, we'd like a final line of defence with the CLI.
First, in the case of a successful start from the `StartupLogger` report, we consider this successful and exit. If we do not have a successful (`success == False`) graceful restart, we will wait for the new process to recover (since the old process will go from `ALIVE` to `STOPPED` to `ALIVE`). I do not expect recovery to take long since we are only remounting, not waiting on RocksDB. Here though, we give a fairly long timeout (1hr) to ensure we're not stuck on an in process thrift call, since we would not like to force kill the process if we're writing to the overlay.
If we time out waiting, we will just kick off a force restart and assume the old process is stuck.
Reviewed By: simpkins
Differential Revision: D20673460
fbshipit-source-id: 215e06fa8aa76dbe5dae6602f82ce5f05968e538
Summary:
We got a user request to add this output after a restart. This (` run "cd / && cd -" to
update your shell's working directory.`) is the output from eden doctor when it detects stale mount points. I don't think it hurts to add, especially since new customers may not make the connection/have the context to deduce that "programs" in this message also apply to the user's shell.
Reviewed By: kmancini
Differential Revision: D20971070
fbshipit-source-id: 3448651cae90b2b5c8c4d2432b946521cdc5a292
Summary:
This refactors the `edenfsctl start` command so that we more clearly split the
functionality into two pieces:
* Starting EdenFS as a service
* Running EdenFS in the foreground
In most normal situations in production the `start`, `restart`, and `stop`
commands are used to manage running EdenFS as a service. In the future I
believe our service management logic will start to diverge a bit more on Linux
vs Mac vs Windows, and this should help isolate the service-management code a
bit more cleanly.
The foreground behavior is mainly only used by developers during testing and
during the integration tests. Several options like `--gdb` and `--strace` are
only allowed in foreground mode, and this refactoring makes that clearer. In
the future we may also want to further restrict this, to allow only
specifying additional custom arguments and a custom binary path when running
in foreground mode. However, for now I have not updated that as I believe
some of our integration tests may be exercising this behavior today.
This change also cleans up some of the platform-specific code, and lets them
share more of the logic to construct arguments for edenfs. With this change
`edenfsctl start --foreground` now works on Windows.
Reviewed By: pkaush
Differential Revision: D20833244
fbshipit-source-id: 0f09d59702d8b64ca8f4fedccbc30da1c858afb4
Summary:
Update the `edenfsctl` and `edenfs` code so that `edenfsctl stop` works on
Windows.
Reviewed By: pkaush
Differential Revision: D20854616
fbshipit-source-id: 9a51cdc1f1d1838c5089b945df1fb624a3eaedb7
Summary:
Move the `start_systemd_service()` function from `daemon.py` to a separate
module. This allows `daemon.py` to be imported on Windows without causing any
of the systemd logic to be imported.
I kept this separate from the existing `systemd.py` module for now just to
help keep the dependencies a little bit cleaner. The `systemd.py` module
currently does not depend on any other logic from other edenfsctl modules.
Reviewed By: pkaush
Differential Revision: D20854612
fbshipit-source-id: dbdbf0fec90682e3b7d16a7532aefdd3f186bf80
Summary:
Update the `edenfsctl clone` command to require that an explicit repository
path be passed in. Using the name of a repository listed in an EdenFS config
file is no longer supported.
Reviewed By: wez
Differential Revision: D20876459
fbshipit-source-id: 5e9fef11a2afa4cc48cb8a9bb5b874d2e7923f25
Summary:
We no longer use repository configs, so remove the `repository` subcommand
that supported adding and listing these configurations.
The main information that used to be included in the repository configuration
was the bind mount settings. This has since been replaced with the
`.eden-redirections` file that is placed directly in each repository.
Reviewed By: wez
Differential Revision: D20876462
fbshipit-source-id: cc7d8e6f0a6a2e04fbf3159417af41a44908b3a8
Summary:
Add a command line argument to `edenfsctl doctor` to cause it to only report
problems with the current EdenFS instance, and to skip checks for system-wide
problems or other running EdenFS processes.
This does refactor a fair amount of the top-level doctor logic to encapsulate
most of the state into an `EdenDoctor` class.
This also updates the integration tests to use this flag when invoking
`edenfsctl doctor`. Previously the integration tests could spuriously fail
due to other pre-existing problems on the system, or due to other EdenFS
instances that are currently being started or shut down by other tests running
in parallel.
Reviewed By: wez
Differential Revision: D20357521
fbshipit-source-id: 36640cc21e7bd79fbd300c4d2c7dbba127ec9170
Summary:
The bind-mounts configuration has been ignored by EdenFS since D17236366.
This removes all CLI code for dealing with this config section.
Reviewed By: wez
Differential Revision: D20876460
fbshipit-source-id: 6b3f3552de25ee28fc0418a6aaec14446520203c
Summary:
This renames the `ProcessFinder` class to `ProcUtils`, in preparation for
adding some more generic process management utilities into this class.
This class was originally just used for finding stale processes in
`edenfsctl doctor`. However, it seems like a good place to consolidate all
platform-specific process querying logic. We are currently need
platform-specific querying to tell if specific process IDs are still running
for `edenfsctl status` and `edenfsctl stop`, and we also already have some
Windows-specific spawning logic in `edenfsctl start`. I plan to move all of
that logic into this `proc_utils.py` module.
This renames `ProcessFinder` to `ProcUtils` just to better reflect its
expanded purpose.
Reviewed By: genevievehelsel
Differential Revision: D20833243
fbshipit-source-id: 8d33460b7468d877bc327e855af5c620ac5b80a7
Summary:
Update out platform checks to use `sys.platform` instead of `os.name`
Python type checkers (mypy and Pyre) currently understand checks against
`sys.platform`, and know that code guarded by these checks is platform
specific. They don't understand the `os.name` checks.
This code should behave the same, but allows type checkers to do a better job
checking this code.
Note that there is also `platform.system()`, but this has drawbacks that it
apparently does runtime-checks to determine the platform, whereas `sys.platform`
is baked-in at compile time. Additionally, the typecheckers do not appear to
support checking based on `platform.system()` for now. There is an open
feature request for this for mypy, but it is not implemented yet.
One caveat for `sys.platform` is that the results on Linux are not consistent
across Python versions: older versions of Python used to report `linux2` while
new versions report simply `linux`.
Reviewed By: chadaustin
Differential Revision: D20830149
fbshipit-source-id: d173e41f1ae84951a84b87e2dc05787fe8b01407
Summary:
Update the `edenfsctl.exe` binary to find `edenfs.exe` relative to itself.
This ensures that when you run `edenfsctl.exe` from a development build that it
finds the associated development build of `edenfs.exe`, rather than the current
system-installed version.
Reviewed By: wez
Differential Revision: D20821419
fbshipit-source-id: 29ff0b587800f1b76c0b729512bc19f5aef648a7
Summary:
I noticed that the output was `{backing_repo}` in a couple
of user reports, rather than the path to the backing repo.
Reviewed By: genevievehelsel
Differential Revision: D20669756
fbshipit-source-id: c9f1dbb4f4b2ad3de03c54f1e3cb5f688ee433ac
