Summary:
Sometimes on Windows, we receive user reports that `edenfsctl` hangs indefinitely. We would like to see where it hangs in these cases. Adding this will allow us to see the traceback when ctrl-c edenfsctl with the environment variable set.
Note `--debug` is shared with Rust edenfsctl (where it enables verbose logging).
Reviewed By: xavierd
Differential Revision: D34731381
fbshipit-source-id: 24477d05c01d353f9f7a45fc3c5d28f29b62e3be
Summary:
Let's be consistent and use EdenFS everywhere in our user facing documentation
and messages.
Reviewed By: fanzeyi
Differential Revision: D34692175
fbshipit-source-id: 6346bd9ac3516199fe47123d81833d2c457efbbc
Summary:
On Windows, the default asyncio event loop would throw an exception on tear down.
This diff fixes that by opting to the older event loop. The older event loop doesn't support subprocess but it doesn't look like we use it anyway.
https://stackoverflow.com/a/62413786/694963
Reviewed By: xavierd
Differential Revision: D34486221
fbshipit-source-id: 765056f1e82c9a53d89482527725154445e59494
Summary:
macFUSE support on apple is going away soon. If macFUSE breaks for folks,
they will need to migrate to NFS. This makes the migration a one command step.
In case folks see any issues with NFS during our migration we also have
an option to migrate back to fuse. This will temporarily unblock them
while we fix any bugs that come up.
Reviewed By: xavierd
Differential Revision: D34081417
fbshipit-source-id: ea89345b5ccc019cbfe6387a1dfa366ae9dd6345
Summary:
With Facebook having been renamed Meta Platforms, we need to change the license
headers.
Reviewed By: fanzeyi
Differential Revision: D33407812
fbshipit-source-id: b11bfbbf13a48873f0cea75f212cc7b07a68fb2e
Summary:
On clone, the CLI tries to open an .arcconfig file at the given revision for
the solve purpose of printing a slightly different log message. Since this log
isn't necessary to be present, let's avoid the `hg cat` and simplify the code a
bit on top of making it faster.
This should make cloning a couple of seconds faster on Windows due to Mercurial
taking roughly a second to start.
Reviewed By: chadaustin
Differential Revision: D33070099
fbshipit-source-id: 81c5cba8348d0c8d82c57a6656e6eb79c2b1f878
Summary:
By default, we do not want to support nested checkouts. Their behavior is not well defined and can cause issues.
This diff disallows nested checkouts unless specifically requested by users. They can use the --allow-nested-checkout or -n flags to force nested checkout creation. If these flags are not used, the clone is aborted and an error message is printed.
Reviewed By: chadaustin
Differential Revision: D32514269
fbshipit-source-id: d04a7ef82c6d31993801648da42e4182167f37e6
Summary:
On Windows, printing paths by default use backward slashes, but forward
slashses are also acceptable. For the sake of making tests pass, let's always
use forward slashes.
Reviewed By: chadaustin
Differential Revision: D32960607
fbshipit-source-id: c1424c4a090d805bab337fc6d540563e8205e5dc
Summary:
This config has been enabled for many months now, let's simply assume it is set
to true on Windows.
Reviewed By: genevievehelsel
Differential Revision: D32192585
fbshipit-source-id: 76829a031f859594a5139dd09a6479cabeda2089
Summary:
As a first step, as I ramp up on Rust and the edenfsctl codebase and in order to more quickly support basic top functionality in windows, we're adding a minitop subcommand that will have a subset of the `eden top` output.
This diff adds scaffolding for the minitop subcommand.
Reviewed By: fanzeyi
Differential Revision: D31841478
fbshipit-source-id: 7020e81a0c8021bad45de835f2e279c049b76200
Summary:
When eden is not running on windows `eden rm` will spit out some error
messages and exits with error code 1. But it does actually successfully
remove the repo.
On linux removing a repo while eden is not running behaves just like if
eden were running.
Let's make the removal more graceful on windows.
Reviewed By: xavierd
Differential Revision: D31519805
fbshipit-source-id: d393922a9474e64251142207ae38a602766f17bf
Summary:
Now that we might have multiple kernel protocols per mount (i.e. both fuse and
nfs on macOS) let's include them in eden rage.
Reviewed By: xavierd
Differential Revision: D31154042
fbshipit-source-id: 38e7630829d70fe9dd6dbeabacc3b538ee798e0d
Summary:
During an `eden chown`, EdenFS will try to chown both the repository, and the
redirections. In some cases, chowning the redirection can both take a long time
and be unecessary. Consider the case where some automation temporarily chown a
repository to a service user that needs to access the repository, and then
chown it back to the owner of the repository. In that case, changing the
ownership of the redirection is superfluous and unecessary.
Reviewed By: mrkmndz
Differential Revision: D31010912
fbshipit-source-id: a882948005ac4fe29ff465088f196e0fc2bc10be
Summary: the rage summary is getting hard to quickly parse, so this underlines each section header, as well as unifies underline looks (with `eden stats`). This adopts the underline code from `eden du` and makes it a util function for shareability.
Differential Revision: D30857773
fbshipit-source-id: 66b5b06f5b0125304d45d3465a8bc2248693b791
Summary: it would be helpful to see a user's or sandcastle job's eden config, especially in the case of a gated feature rollout / staged feature rollout.
Differential Revision: D30857763
fbshipit-source-id: ee2a311ee643fc9db5acef1b02017564c51d2362
Summary:
Added eden prefetch-profile activate-predictive/deactivate-predictive subcommands to activate and deactivate predictive prefetch profiles. This will update the checkout config to indicate if predictive prefetch profiles are currently active or not, and stores the overridden num_dirs if specified on activate (--num-dirs N). If activate is called twice with different num_dirs, the value is updated (only one is stored). Unless --skip-prefetch is specified, a predictive prefetch with num_dirs globs (or the default inferred in the daemon) is run.
Also added fetch-predictive [--num-dirs N], which will:
1. if num_dirs is specified: fetch num_dirs globs predictively
2. if num_dirs is not specified, and predictive fetch is active: get the active num_dirs from the checkout config and fetch globs predictively
3. if num_dirs is not specified, and predictive fetch is not active: fetch the default num_dirs (inferred in the daemon)
Added --if-active to fetch-predictive. If set, fetch will not run if predictive prefetch profiles have not been activated (predictive-prefetch-active in checkout-config). Used for post pull hook.
Reviewed By: genevievehelsel
Differential Revision: D30306235
fbshipit-source-id: ba02c2bc976128704c8ab0c3d567637265b7c95d
Summary:
This adds the `eden redirect cleanup-apfs` command for deleting stale APFS volumes.
* An APFS is considered as stale if it's not currently mounted and not considered as under any of the checkouts managed by the eden instance.
* The command prints the list of such volumes and uses the APFS util to delete them if the user confirms.
Note: as the command is local to an eden instance, it will list not-mounted APFS volumes of a checkout managed by another eden instance as stale. This should rarely happen as in production we expect there to be a single eden instance. The prompt would also let the user abort if something is wrong.
Reviewed By: chadaustin
Differential Revision: D29940980
fbshipit-source-id: e784cb54d20198bb1f74cd5f15cee0e7546b227c
Summary:
This newly added config controls the default behavior of `eden clone` and
whether a new clone will use NFS or FUSE. This is intended to facilitate the
transition to NFS from FUSE on macOS.
Differential Revision: D30110056
fbshipit-source-id: ea6b493aa803297952b46434f6d11d8edf58e40b
Summary: As title. This was added as `thrift-py3` doesn't support CMake well but we can workaround by importing from py.
Reviewed By: xavierd
Differential Revision: D30078300
fbshipit-source-id: 277866f8b226f164b5e30231aa10b59c0aba5807
Summary: `eden mount` currently swallows all but one type of exceptions, which resulted in cli reporting false positive mount success when the thrift endpoint throws. The try-catch was added in D21934538 (51df752a46) to provide clear error message for a particular windows error and we can just re-raise so that the other exceptions are propagated.
Reviewed By: chadaustin
Differential Revision: D30017261
fbshipit-source-id: 6a8a44330a90275b3c044301ff644dce0d6dee13
Summary: Combine `eden debug log` and `eden logs`. The logic from `eden logs` is moved to `eden debug log upload`.
Reviewed By: genevievehelsel
Differential Revision: D29801785
fbshipit-source-id: 6283a33a3180fec6934ac52fc8d5eed4a0a483b0
Summary: This change let Eden cli can ```clone``` and ```info``` on a RE Digest backed store
Reviewed By: chadaustin
Differential Revision: D28855458
fbshipit-source-id: 5582992acc5b3b3acb05b0b53d59a6768cc02491
Summary: This can be used on windows since it uses `shutil.rmtree` instead of `fm -rf`
Differential Revision: D29723916
fbshipit-source-id: 7d12ce8d265661698c1f5ecd17271d1c2e950a55
Summary: Added a --deep-clean option to eden du that removes .edeb/clients/x/fsck directories.
Reviewed By: genevievehelsel
Differential Revision: D29501641
fbshipit-source-id: 9c01dc76b54e151ada977c0ee0c28baafe761824
Summary: There are a lot of places in user visible text such as command line help where EdenFS is mentioned as eden/Eden/edenfs/EdenFS. This will make it consistent to 'EdenFS' in most cases. In the places where it is referring to the process/daemon, 'edenfs' will be used.
Reviewed By: chadaustin
Differential Revision: D29419151
fbshipit-source-id: 7b8296f0a0c84fdcb566ff811f7fcedbe7079189
Summary:
When buck kill fails, eden rm will also fail. This has caused some checkouts
to not be removed when they could be. Stopping aux processes is a nice thing
to do before we unmount. It ensures these processes close file handles in the
mount, but we force unmount anyways so open file handles should not be able to
block the umount call.
Reviewed By: xavierd
Differential Revision: D29205962
fbshipit-source-id: a899940efa5cc1d960cd14a775b7053c34f5d6f2
Summary:
Making it the default should make cloning using EdenFS easier and will remove
the need to manually pass the `--nfs` command line to `eden clone`.
Reviewed By: kmancini
Differential Revision: D28913818
fbshipit-source-id: 742c35c950cb5edf34fd9769fb78bf26095af0d5
Summary:
Currently running `edenfsctl top` will crash on Windows:
```
Traceback (most recent call last):
File "C:\Python38\Lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Python38\Lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\tools\eden\bin\edenfsctl.exe\__main__.py", line 3, in <module>
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2253, in zipapp_main
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2236, in main
File "C:\Python38\Lib\asyncio\runners.py", line 43, in run
return loop.run_until_complete(main)
File "C:\Python38\Lib\asyncio\base_events.py", line 616, in run_until_complete
return future.result()
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2212, in async_main
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 1059, in run
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\top.py", line 395, in __init__
File "C:\Python38\Lib\curses\__init__.py", line 13, in <module>
from _curses import *
ModuleNotFoundError: No module named '_curses'
```
This diff will let it prints an error message.
Reviewed By: xavierd
Differential Revision: D28207330
fbshipit-source-id: a465fe5941b469f4a1ef964f1d4dc8a593639e7c
Summary:
It looks like argparse's exit was not able to handle asyncio event loop well,
causing edenfsctl to generate a long ugly stack trace when the command line
flag does not parse.
Let's just move the arguments parsing outside the asyncio runloop to avoid this
problem as a whole. In theory it should improve our `--help` time a little bit.
Reviewed By: chadaustin
Differential Revision: D28206622
fbshipit-source-id: 881eefaea73b244eadff0165965085e64dad935f
Summary:
Some user reported to see `edenfsctl restart` crashes due to this call to
os.getuid() since it does not available on Windows. P410914264
https://docs.python.org/3.9/library/os.html#os.getuid
Reviewed By: chadaustin
Differential Revision: D28204262
fbshipit-source-id: 077bf207d8b1b6c014fface63ea93e66057629cd
Summary:
Context: https://fb.workplace.com/groups/edenfswindows/permalink/828914994691047/
Even with D27872753 it doesn't really make sense to have `eden fsck` running on Windows since it requires EdenFS repository to **be unmounted**.
This diff changes it to generate a warning to redirect users to run `eden doctor` instead (which is likely what they need).
Reviewed By: kmancini
Differential Revision: D28203778
fbshipit-source-id: ae105678876903bcf6514252bf07189775f9b187
Summary: This just makes it more obvious _where_ `--force` should be passed.
Reviewed By: genevievehelsel
Differential Revision: D28119590
fbshipit-source-id: 1fbdb4428e9b89e7b66c959f874067485a91d534
Summary: This diff makes treeoverlay the default overlay type for Windows users.
Reviewed By: kmancini
Differential Revision: D27247658
fbshipit-source-id: 866eafc794eff1c262eab3061f14eb597bea0a66
Summary:
Some of our subprocess calls are running into dylib errors. The cause looks to
be related to our environment variables. We already have environment hygenics
for buck, so lets borrow this to use elsewhere.
This is to fix prefetch profile fetching on mac, but I ran into another error
when testing `eden du --clean`.
Reviewed By: genevievehelsel
Differential Revision: D27135268
fbshipit-source-id: 3955ddefc5e9ff60e966f63f7dc65ef737186464
Summary:
It's silly to use `eden prefetch --no-prefetch` to efficiently glob
for filenames. Introduce an `eden glob` command which resolves a glob
relative to the current working directory.
Reviewed By: genevievehelsel
Differential Revision: D25450358
fbshipit-source-id: 45d6dc870d21510e51d5662c75e80385886899fc
Summary:
The world has moved on utf-8 as the default encoding for files and data, but
EdenFS still accepts non utf-8 filenames to be written to it. In fact, most of
the time when a non utf-8 file is written to the working copy, and even though
EdenFS handles it properly, Mercurial ends up freaking out and crash. In all of
these cases, non-utf8 files were not intentional, and thus refusing to create
them wouldn't be a loss of functionality.
Note that this diff makes the asumption that Mercurial's manifest only accept
utf8 path, and thus we only have to protect against files being created in the
working copy that aren't utf8.
The unfortunate part of this diff is that it makes importing trees a bit more
expensive as testing that a path is utf8 valid is not free.
Reviewed By: chadaustin
Differential Revision: D25442975
fbshipit-source-id: 89341a004272736a61639751da43c2e9c673d5b3
Summary:
The default filesystem on macOS (APFS) is case-insensitive, but EdenFS has so
far been case-sensitive except on Windows. Some of the native tools (Unity for
instance), expect macOS to always be case-insensitive, and is thus breaking due
to that.
The safe behavior would be to have EdenFS behave exactly the same as APFS: be
case insensitive. For now, to avoid breaking users this will be done on new
mounts only, and once fully validated, this will be made the default and forced
on.
Reviewed By: chadaustin
Differential Revision: D26356269
fbshipit-source-id: 96ca331d8c9726213520dff3e3563019d0400a95
Summary:
This diff adds a Rust wrapper that will forward any command it fails to parse
to the companion binary `edenfsctl.real` at the same directory. This will allow
us to rewrite some of the subcommands in Rust while keeps the original
functionality.
With `EDENFSCTL_SKIP_RUST` environment variable set, it will skip the argument
parsing process in case of any emergency situation.
Reviewed By: xavierd
Differential Revision: D25758959
fbshipit-source-id: 0fbc69d0e0733ad2608622417be93dc7db2d6fa6
Summary: This enables the repository to be mounted via NFS, and not FUSE.
Reviewed By: chadaustin
Differential Revision: D26229827
fbshipit-source-id: 5af5a47ebe5f1dd54df7707bf57d9b7476921f29
Summary: `hg gc` is deprecated and does nothing anymore. Lets just remove this section talking about running `hg gc`
Reviewed By: chadaustin
Differential Revision: D26178391
fbshipit-source-id: 2bb16bbac38465a524ed50454d548564324d35bf
Summary:
As described in D26033704, it would be really nice if we could have a version of `eden rm` that did everything except remove the mount point directory. This creates a nice symmetry between the tolerated start and end points of a clone/rm cycle, in that `eden clone` already tolerates being targeted against and empty directory.
NOTE: I made no effort to make this work on Windows. The implementation was different enough that I wasn't exactly sure how you all wanted to go about it. We don't run on Windows at all, so this is all we need, but I'd understand if you all want to make it work, or at least throw an explanatory error.
Reviewed By: chadaustin
Differential Revision: D26033912
fbshipit-source-id: 36d63ad510393d3d09b1912a73efaebede534f61
Summary: Like `strace`, show the return codes from FUSE requests.
Reviewed By: kmancini
Differential Revision: D26033195
fbshipit-source-id: 2347129ce480e50a3b0f588937e535e0df45dfbd
Summary:
A lot of users have been having trouble getting their eden repos recloned
recently, to make this process simplier I add the reclone process to the clone
script.
In this script I check if a user has multiple repos depending on the same
backing store. This allows me to warn them that they might lose changes from
these other repos. This diff threads along the backing store to the
`eden list` result for that check.
Reviewed By: chadaustin
Differential Revision: D25078423
fbshipit-source-id: 9ceb1f9acc4ec170cbb12d4b0b3b7d51987f88e3
Summary:
On Windows, the GUID of the mount point identifies the virtualization instance,
that GUID is then propagated automatically to the created placeholders when
these are created as a response to a getPlaceholderInfo callback.
When the placeholders are created by EdenFS when invalidating directories we
have to pass GUID. The documentation isn't clear about whether that GUID needs
to be identical to the mount point GUID, but for a very long time these have
been mismatching due to the mount point GUID being generated at startup time
and not re-used.
One of the most common issue that users have reported is that sometimes
operations on the repository start failing with the error "The provider that
supports file system virtualization is temporarily unavailable". Looking at the
output of `fsutil reparsepoint query` for all the directories from the file
that triggers the error to the root of the repositories, shows that one of the
folder and its descendant don't share the same GUID, removing it solves the
issue.
It's not clear to me why this issue doesn't always reproduce when restarting
EdenFS, but a simple step that we can take to solve this is to always re-use
the GUID, and that hopefully will lead to the GUID always being the same and
the error to go away.
Reviewed By: fanzeyi
Differential Revision: D25513122
fbshipit-source-id: 0058dedbd7fd8ccae1c9527612ac220bc6775c69
