Summary:
On Windows, paths components are usually separated by '\', and since the
repository path is stored in a toml file, whatever character is after a '\',
will be escaped. In my case, this is followed by U (for C:\Users), and thus
toml expects the next characters to be an escaped unicode. That's obviously
not the case and thus EdenFS fails to parse the config, preventing me from
cloning fbsource.
Since Windows is perfectly fine with '/' as path separator, let's just
replace '\' with '/'.
The underlying bug appears to be in the toml Python code: https://github.com/uiri/toml/issues/280
Manually trying some random path is pretty conclusive:
(Pdb) toml.dumps({'foo': 'c:\\Users\\wez'})
'foo = "c:\\\\Users\\\\wez"\n'
(Pdb) toml.dumps({'foo': 'c:\\Users\\xavier'})
'foo = "c:\\Users\\xavier"\n'
Reviewed By: chadaustin
Differential Revision: D21143545
fbshipit-source-id: 448471da12c253dd37680f6a28251a1e69850920
Summary:
As a top layer of reliability in graceful restart, we'd like the CLI to enforce that something has started after a graceful restart call. There are a few things that can happen
1) The restart short circuts before attempting to shutdown (version mismatch etc). The old process will continue (`success == False`)
2) The restart is successful, the new process will continue (`success == True`)
3) The restart is not successful (failed ping), the old process will recover (`success ==
False`)
(note here that the `success` field is the return code of the `StartupLogger` process, not the edenfs process itself).
In case any of these fail, we'd like a final line of defence with the CLI.
First, in the case of a successful start from the `StartupLogger` report, we consider this successful and exit. If we do not have a successful (`success == False`) graceful restart, we will wait for the new process to recover (since the old process will go from `ALIVE` to `STOPPED` to `ALIVE`). I do not expect recovery to take long since we are only remounting, not waiting on RocksDB. Here though, we give a fairly long timeout (1hr) to ensure we're not stuck on an in process thrift call, since we would not like to force kill the process if we're writing to the overlay.
If we time out waiting, we will just kick off a force restart and assume the old process is stuck.
Reviewed By: simpkins
Differential Revision: D20673460
fbshipit-source-id: 215e06fa8aa76dbe5dae6602f82ce5f05968e538
Summary:
We got a user request to add this output after a restart. This (` run "cd / && cd -" to
update your shell's working directory.`) is the output from eden doctor when it detects stale mount points. I don't think it hurts to add, especially since new customers may not make the connection/have the context to deduce that "programs" in this message also apply to the user's shell.
Reviewed By: kmancini
Differential Revision: D20971070
fbshipit-source-id: 3448651cae90b2b5c8c4d2432b946521cdc5a292
Summary:
This refactors the `edenfsctl start` command so that we more clearly split the
functionality into two pieces:
* Starting EdenFS as a service
* Running EdenFS in the foreground
In most normal situations in production the `start`, `restart`, and `stop`
commands are used to manage running EdenFS as a service. In the future I
believe our service management logic will start to diverge a bit more on Linux
vs Mac vs Windows, and this should help isolate the service-management code a
bit more cleanly.
The foreground behavior is mainly only used by developers during testing and
during the integration tests. Several options like `--gdb` and `--strace` are
only allowed in foreground mode, and this refactoring makes that clearer. In
the future we may also want to further restrict this, to allow only
specifying additional custom arguments and a custom binary path when running
in foreground mode. However, for now I have not updated that as I believe
some of our integration tests may be exercising this behavior today.
This change also cleans up some of the platform-specific code, and lets them
share more of the logic to construct arguments for edenfs. With this change
`edenfsctl start --foreground` now works on Windows.
Reviewed By: pkaush
Differential Revision: D20833244
fbshipit-source-id: 0f09d59702d8b64ca8f4fedccbc30da1c858afb4
Summary:
Update the `edenfsctl` and `edenfs` code so that `edenfsctl stop` works on
Windows.
Reviewed By: pkaush
Differential Revision: D20854616
fbshipit-source-id: 9a51cdc1f1d1838c5089b945df1fb624a3eaedb7
Summary:
- Catch the Windows-specific exceptions types thrown when attempting to open
the socket if EdenFS is not running, and convert this to an EdenNotRunning
exception.
- Update EdenFS to write its pid to a separate file, in addition to the normal
lock file, and have `edenfsctl` read this file instead of the lock file on
Windows. Other processes cannot read the lock file while EdenFS is running
and holding the lock, so it is nice to have the pid written in an alternate
location.
Reviewed By: pkaush
Differential Revision: D20854615
fbshipit-source-id: 1c8e8f402eb17dd012d03e11fbb662f493d9362d
Summary:
Move the `start_systemd_service()` function from `daemon.py` to a separate
module. This allows `daemon.py` to be imported on Windows without causing any
of the systemd logic to be imported.
I kept this separate from the existing `systemd.py` module for now just to
help keep the dependencies a little bit cleaner. The `systemd.py` module
currently does not depend on any other logic from other edenfsctl modules.
Reviewed By: pkaush
Differential Revision: D20854612
fbshipit-source-id: dbdbf0fec90682e3b7d16a7532aefdd3f186bf80
Summary: I believe it would be nice to include the output of `eden redirect list` in `eden rage` since I've run into a few times this week where that was something that was asked for.
Reviewed By: chadaustin
Differential Revision: D20955632
fbshipit-source-id: 0d28688b3764a1bec8a9e92b72c166dddd18f591
Summary:
Clean up the last two remaining pyre-fixme comments in this file.
I believe these casts were needed at one point to make mypy happy, but it
looks like they are no longer necessary.
Reviewed By: genevievehelsel
Differential Revision: D21007030
fbshipit-source-id: 6933425d9fde8b86272c68b955faf21348453f68
Summary:
Update the `edenfsctl clone` command to require that an explicit repository
path be passed in. Using the name of a repository listed in an EdenFS config
file is no longer supported.
Reviewed By: wez
Differential Revision: D20876459
fbshipit-source-id: 5e9fef11a2afa4cc48cb8a9bb5b874d2e7923f25
Summary:
We no longer use repository configs, so remove the `repository` subcommand
that supported adding and listing these configurations.
The main information that used to be included in the repository configuration
was the bind mount settings. This has since been replaced with the
`.eden-redirections` file that is placed directly in each repository.
Reviewed By: wez
Differential Revision: D20876462
fbshipit-source-id: cc7d8e6f0a6a2e04fbf3159417af41a44908b3a8
Summary:
Add a command line argument to `edenfsctl doctor` to cause it to only report
problems with the current EdenFS instance, and to skip checks for system-wide
problems or other running EdenFS processes.
This does refactor a fair amount of the top-level doctor logic to encapsulate
most of the state into an `EdenDoctor` class.
This also updates the integration tests to use this flag when invoking
`edenfsctl doctor`. Previously the integration tests could spuriously fail
due to other pre-existing problems on the system, or due to other EdenFS
instances that are currently being started or shut down by other tests running
in parallel.
Reviewed By: wez
Differential Revision: D20357521
fbshipit-source-id: 36640cc21e7bd79fbd300c4d2c7dbba127ec9170
Summary:
The bind-mounts configuration has been ignored by EdenFS since D17236366.
This removes all CLI code for dealing with this config section.
Reviewed By: wez
Differential Revision: D20876460
fbshipit-source-id: 6b3f3552de25ee28fc0418a6aaec14446520203c
Summary:
As discussed in D20629833 metrics for live imports
like those for pending imports can help differentiate
where imports are in the process.
This can help give users more insight into what is
happening inside eden. Meaning that they can see
progress being made, and that eden is not stuck. But
also when eden is stuck these metics can indicate
that it is stuck fetching data from the mercurial servers.
This displays those metrics in `eden top` similar to those
for pending imports
Reviewed By: chadaustin
Differential Revision: D20630826
fbshipit-source-id: ab7368927d2832acbea89a82585fda5195ce03d1
Summary:
When cloning a Mercurial repository, default to checking out the `master`
bookmark, if it exists. Continue using `.` in case the repository does not
have a `master` bookmark.
Reviewed By: pkaush
Differential Revision: D20876461
fbshipit-source-id: 57fa12e4c713bd50c15f59eb9281e0511c3cfe88
Summary:
Do not put a `README_EDEN.txt` file in the checkout root on Windows. On Linux
& Mac the EdenFS mount hides this directory, so the README file is not visible
while the checkout is mounted and running normally. However on Windows
anything present in this directory is visible to the user, so this results in
`README_EDEN.txt` incorrectly showing up in the checkout root.
Reviewed By: genevievehelsel
Differential Revision: D20929408
fbshipit-source-id: 9994524041f22fd8922c531f0185186b04c54821
Summary:
Make sure the function that decides if we should use systemd always returns
False on non-Linux platforms, even if it is explicitly enabled via the config
file or environment variable. systemd is Linux-specific, and it doesn't make
sense to try and use it on other platforms.
Reviewed By: fanzeyi
Differential Revision: D20925344
fbshipit-source-id: cee67f607809da15f584de1eb12a2c4a243b0c91
Summary: This has been deprecated for a long time in favor of `eden debug logging`
Reviewed By: genevievehelsel
Differential Revision: D20876084
fbshipit-source-id: 0d9bf71e55aaffde7ae9f061def0adf8312c2411
Summary:
Add methods to check if a process ID is alive, and if it looks like an EdenFS
process.
This also adds an initial version of ProcUtils for Windows, and implements
these two methods on Windows. I have moved parts of the `winproc.py` module
to the new `proc_utils_win.py` module, to help better manage dependencies
between our modules. This keeps all of the Windows-specific `ctypes` code
together in `proc_utils_win.py`. The functionality that is still left on
`winproc.py` depends on `config.py`, and the `proc_utils` code should not
depend on `config.py` to help avoid circular dependencies.
Reviewed By: wez
Differential Revision: D20833245
fbshipit-source-id: 43e9b6dd1b520dcb6b2da7701de885058f0f7ea2
Summary:
This renames the `ProcessFinder` class to `ProcUtils`, in preparation for
adding some more generic process management utilities into this class.
This class was originally just used for finding stale processes in
`edenfsctl doctor`. However, it seems like a good place to consolidate all
platform-specific process querying logic. We are currently need
platform-specific querying to tell if specific process IDs are still running
for `edenfsctl status` and `edenfsctl stop`, and we also already have some
Windows-specific spawning logic in `edenfsctl start`. I plan to move all of
that logic into this `proc_utils.py` module.
This renames `ProcessFinder` to `ProcUtils` just to better reflect its
expanded purpose.
Reviewed By: genevievehelsel
Differential Revision: D20833243
fbshipit-source-id: 8d33460b7468d877bc327e855af5c620ac5b80a7
Summary:
Update out platform checks to use `sys.platform` instead of `os.name`
Python type checkers (mypy and Pyre) currently understand checks against
`sys.platform`, and know that code guarded by these checks is platform
specific. They don't understand the `os.name` checks.
This code should behave the same, but allows type checkers to do a better job
checking this code.
Note that there is also `platform.system()`, but this has drawbacks that it
apparently does runtime-checks to determine the platform, whereas `sys.platform`
is baked-in at compile time. Additionally, the typecheckers do not appear to
support checking based on `platform.system()` for now. There is an open
feature request for this for mypy, but it is not implemented yet.
One caveat for `sys.platform` is that the results on Linux are not consistent
across Python versions: older versions of Python used to report `linux2` while
new versions report simply `linux`.
Reviewed By: chadaustin
Differential Revision: D20830149
fbshipit-source-id: d173e41f1ae84951a84b87e2dc05787fe8b01407
Summary:
Update the `edenfsctl.exe` binary to find `edenfs.exe` relative to itself.
This ensures that when you run `edenfsctl.exe` from a development build that it
finds the associated development build of `edenfs.exe`, rather than the current
system-installed version.
Reviewed By: wez
Differential Revision: D20821419
fbshipit-source-id: 29ff0b587800f1b76c0b729512bc19f5aef648a7
Summary:
As discussed in D20611704, the duration of time that imports are queued for is a strong indicator of the health of the import process. Adding it to `eden top` will help give users insight into what eden is doing when it hangs for a long time or when there is an issue, indicate whether that issue is due to the import process.
Because it is a strong indicator (specifically stronger than the number of imports) for when there is an issue effecting imports, we use it as a warning signal to the user, meaning we display warning colors when the duration of the longest import is long enough to warrant concern.
We have 2 levels of concern to differentiate when there may be something strange happening and when there is almost definitely something strange happening. These correspond to displaying the import metrics in yellow and red respectively. The current threshold for something strange possibly happing is imports taking longer than 10s and something strange definitely happening is imports taking longer than 30s.
Reviewed By: chadaustin
Differential Revision: D20627754
fbshipit-source-id: 7931d2c0ab53952c1291fd1d411dad14d142734b
Summary:
When there is an issue in the import process, there is not an easy way to detect this currently. As a first step towards making it possible to detect when there is an issue here, we expose the number if imports pending to `eden top`.
If there is a problem blocking imports from happening, then they will start to queue up and the number of pending imports will keep growing. When there are a lot of pending imports it indicates that there may be a problem in the import process.
note: there are cases when there are many pending imports under normal operation. It is common to batch fetch a bunch of files at once, and submitting all these prefetches at once will mean a lot of pending imports though there is no problem. Thus number of pending imports only indicates there *may* be an issue in the import process. (Thus we couple it with the duration of the longest import D20627754 for stronger signal)
The other value in showing the number of pending imports is giving users more transparency into eden. When there is a big batch of imports this may leave the user waiting a while, and this gives a way for them to know why. They can see that eden is making progress rather than being stuck.
Reviewed By: fanzeyi
Differential Revision: D20527273
fbshipit-source-id: 59e0fdff6e7b154b409dfab385b4fdb02062b066
Summary:
Update the LinuxProcessFinder code to ignore the command line arguments when
looking for the state directory, and instead only look at the open file
descriptors.
The path found from the command line arguments could be incorrect: if the
EdenFS state directory had been deleted or renamed the information extracted
from the command line would be incorrect.
The information found by examining the open file descriptors will point to the
new directory location if the state directory has been renamed. We can also
tell if the lock file had been deleted in this case as well. This adds
another field to the `EdenFSProcess` class to indicate if the lock file is no
longer valid.
Reviewed By: wez
Differential Revision: D20638621
fbshipit-source-id: e9db1390f97baf50ad3246de4a964eaaf92622e7
Summary: Enable the CLI tests on macOS when run with Buck.
Reviewed By: wez
Differential Revision: D20524347
fbshipit-source-id: cf3e302256b6b0e6958999cf83c5be5d48f65907
Summary:
I noticed that the output was `{backing_repo}` in a couple
of user reports, rather than the path to the backing repo.
Reviewed By: genevievehelsel
Differential Revision: D20669756
fbshipit-source-id: c9f1dbb4f4b2ad3de03c54f1e3cb5f688ee433ac
Summary:
Move the code for the command-line tool up one directory, out of eden/fs.
This better separates the code so that eden/fs contains code for the edenfs
daemon, while eden/cli contains code for the command line tool.
Reviewed By: bolinfest
Differential Revision: D4888633
fbshipit-source-id: 5041e292c5353d05122eefe5db3257289e31239a
Summary:
This diff adds a new "eden debug" command, with a few subcommands:
- "eden debug inode": Report information about currently loaded inodes
- "eden debug tree": Report data about source control Tree objects
- "eden debug blobmeta": Report metadata about source control Blob objects
- "eden debug blob": Report the contents of source control Blob objects
This diff also includes the thrift APIs in edenfs to support these commands.
Reviewed By: bolinfest
Differential Revision: D4760528
fbshipit-source-id: 7dc2bd6e0e952ba6fb61702c672fb9417d603ffa
Summary:
When the "eden daemon" CLI command has to run edenfs via sudo, also explicitly
pass through the KRB5CCNAME environment variable. This will allow edenfs and
it's hg import helper to find the correct Kerberos credentials if it needs to
connect to a remote server.
Reviewed By: wez, bolinfest
Differential Revision: D4584132
fbshipit-source-id: 6c72e4eca76d81b3d4b53a9e72a0e8388da30a7d
Summary:
Refactor the Overlay code to store data using inode numbers rather than the
affected file's path in the repository. This simplifies the TreeInode code a
bit, as we no longer have to rename overlay files to stay in sync with the file
paths. This also eliminates some crashes when trying to update overlay files
for inodes that have been unlinked (and hence no longer have a path). This
also includes a few fixes to avoid writing journal entries for unlinked files
too. Additionally this contains a few fixes to how mode bits are stored in the
overlay, and fixes a bug where create() was ignoring the mode argument.
Reviewed By: wez
Differential Revision: D4517578
fbshipit-source-id: c1e31497dcf62c322b0deff72b0a02675b0509ab
Summary:
This mirrors the capability added in D4444058.
I don't think the `getenv` calls are awesome, but also don't think that we're
likely to want to override these outside of the environment either, so it
doesn't warrant plumbing these all the way through the eden cli, edenfs command
line flags and the layers to get to instantiate this object.
Reviewed By: simpkins
Differential Revision: D4446320
fbshipit-source-id: d5661e4f3e8dee82617eb6edddbcb9da5f4296d2
Summary:
While testing with the fb-eden rpm installed, I hit some integration
test failures. These were caused by the integration tests picking up the
default post-clone hook configuration.
This diff changes our existing `systemConfigDir` option (which defaults to
`/etc/eden/config.d`) to `etcEdenDir` (which defaults to `/etc/eden`) and
adjusts the code that consumed `systemConfigDir` to construct the effective
value by computing `etcEdenDir + "config.d"`.
Doing this allows us to also default the `repoHooks` path to be
`etcEdenDir + "hooks"` rather than just hard coding `/etc/eden/hooks`.
The result of this is that our integration tests will now pass when `fb-eden`
is installed, because they override the `etcEdenDir` option and isolate their
`edenfs` processes from the globally installed configuration.
Reviewed By: bolinfest
Differential Revision: D4446321
fbshipit-source-id: 524fdb2f386fdf16dce42dce7661d07e13c0f0e7
Summary:
I'm going to follow up with some changes to substitute
some core env vars; this is just some prep.
Reviewed By: bolinfest
Differential Revision: D4444030
fbshipit-source-id: ab5d0e39aba14cdba39bb4867ec24955665723c6
Summary:
`eden config` dumps all parsed config values.
`eden config --get SECTION.OPTION` prints the value of the corresponding key.
Reviewed By: bolinfest
Differential Revision: D4434084
fbshipit-source-id: 63d9f72aa8794371e89a6d9e527bacaab17540de
Summary:
This makes it easier to reason about where these things can be found,
and in particular, allows passing the path through to the post-clone hook
script. Although post-clone accepts some command line configuration options,
our hooks scheme doesn't seem to allow setting that in the eden configuration
file.
Reviewed By: bolinfest
Differential Revision: D4443872
fbshipit-source-id: ed92dbac3b91b7f7dc86a39a85b465907f290b47
Summary:
We implement some sanity checking in wrappers, preventing some unsafe
consequences of using raw Buck UI in fbcode.
Reviewed By: Gownta
Differential Revision: D4453147
fbshipit-source-id: 3069898069c7e89223b133224f7c87d1a6b5886a
Summary:
Fix issues flagged by running "arc lint" on all eden files. There are still 6
warnings outstanding, but these are either false positives or advice that we
intentionally are ignoring for legitimate reasons.
Reviewed By: bolinfest
Differential Revision: D4446615
fbshipit-source-id: 992f3c146f99d63935f849aa775dd6d611a04acf
Summary:
Update copyright statements to "2016-present". This makes our updated lint
rules happy and complies with the recommended license header statement.
Reviewed By: wez, bolinfest
Differential Revision: D4433594
fbshipit-source-id: e9ecb1c1fc66e4ec49c1f046c6a98d425b13bc27
Summary:
Previously we'd emit an AttributeError. This diff
fixes things up so that we show the help when no arguments are provided.
Reviewed By: simpkins
Differential Revision: D4434060
fbshipit-source-id: 838b1cab47118d8517e4dc0c5fde17fc69752b13
Summary:
We can use `//` exclusively because we always build Eden with Buck and never
fbbuild, our legacy build system for fbcode.
This revision was initially created by running:
```
find eden -name TARGETS | xargs sed -i -e 's#@/#//#g'
```
And then manually updating the `DEFS` file now that we no longer need
some normalization code for an outdated pattern.
But then I got annoyed by other inconsistencies, so I went through and
alpha-sorted some lists, replaced all double quotes with single quotes,
and fixed indents to be two spaces.
Reviewed By: simpkins
Differential Revision: D4356724
fbshipit-source-id: ab07a48f12fa937c257213d12331efdf09e42da6
Summary:
The keys in the config directory map are normalized, absolute paths to the
mount point. When trying to look up a mount point make sure we also always use
a normalized absolute path.
Reviewed By: bolinfest
Differential Revision: D3942565
fbshipit-source-id: 63db838ffc7139d779925adf07c50f849d73bcc5
Summary:
This design is inspired by that of Git hooks:
https://git-scm.com/docs/githooks
By default, `/etc/eden/hooks` should be the place where Eden looks for
hooks; however, this can be overridden in `~/.edenrc` on a per-`repository` basis.
This directory should be installed as part of installing Eden.
There is information in `eden/hooks/README.md` about this.
The first hook that is supported is for post-clone logic for a repository.
This change demonstrates the need for an `eden config --get <value>`
analogous to what Git has, as hooks should be able to leverage this in their
own scripts. There introduces a `TODO` in `post-clone.py` where such a
feature would be useful, so that I could add the following to my `~/.edenrc`
to develop the Eden extension for Hg:
```
[hooks]
hg.edenextension = /data/users/mbolin/fbsource/fbcode/eden/hg/eden
[repository fbsource]
path = /data/users/mbolin/fbsource
type = hg
hooks = /data/users/mbolin/eden-hooks
```
Note that this revision also introduces a `generate-hooks-dir` script that can be
used to generate the standard `/etc/eden/hooks` directory that we intend to
distribute with Eden. This is also useful in creating the basis for a custom `hooks`
directory that can be specified as shown above in an `~/.edenrc` file.
Reviewed By: simpkins
Differential Revision: D3858635
fbshipit-source-id: 215ca26379a4b3b0a07d50845fd645b4d9ccf0f2
Summary:
This did not work because the dict-like object read from
a `ConfigParser` was not JSON-serializable by Python.
I had to add some methods to the `Repository` that we use in our
integration test harness in order to verify everything I wanted to
in my new integration test. I implemented these methods in both
`HgRepository` and `GitRepository`.
Reviewed By: simpkins
Differential Revision: D3837879
fbshipit-source-id: e0bfb5f1bd3add192ef9bdf561591ac8e52bc002
Summary: I did what the linter told me to do.
Reviewed By: wez
Differential Revision: D3836659
fbshipit-source-id: a5d3fc8974cf6cb7c7e2d88a6215ac5c54479780
