Summary:
Define an `add_mount()` method to simplify the process of defining mounts in
the FakeMountTable. Also simplify the `lstat()` code to only try lookups using
strings. The keys in the dictionary are always strings, so there is no need to
ever try looking up entries using bytes.
Reviewed By: chadaustin
Differential Revision: D7704335
fbshipit-source-id: 9c81b43c0a196287254bcd749d096f9b148af8ac
Summary:
Add a pretty basic `restart` command to the Eden CLI.
This is relatively simple at the moment. It has `--graceful` and `--full`
options, and defaults to a full restart at the moment if neither of these
arguments is specified. We probably could add most of the arguments from the
`daemon` command to `restart`, but I'm not sure if we really want to support
that complexity in this command.
In the future we can make this command smarter and have it do a better job of
reporting the status as Eden restarts.
Reviewed By: chadaustin
Differential Revision: D7682838
fbshipit-source-id: 80c8d6604c3ca120fb71d6b8a388309218b5500c
Summary:
Refactor the Eden CLI command so that all subcommands are implemented as
subclasses. This helps keep the command line argument definitions together
with the logic for the command.
This is primarily just a code refactoring change, but I did include a few minor
behavioral changes to the help output:
- The command list is now always sorted alphabetically in the help output.
- The "help" subcommand can now show help for more than just one subcommand
deep. (e.g., `eden help stats io` now works correctly)
- I made some minor improvements to a few of the help strings.
Reviewed By: chadaustin
Differential Revision: D7673021
fbshipit-source-id: dc4c6db20a0fe7452d38bdafc6273e234dba8e4e
Summary:
Temporary fix to bump the takeover timeout to five minutes in the cli
code. I messed up and forgot to ship these changes with D7660642.
Reviewed By: simpkins
Differential Revision: D7676402
fbshipit-source-id: a22963fbf89663e44537e7946b3be5eb870f930c
Summary:
In practice, we have received a number of user reports where Nuclide had lost
track of its Watchman subscriptions and `eden doctor` failed to report it.
When the `NuclideHasExpectedWatchmanSubscriptions` check was first added to
`eden doctor`, in the repro case we had, it seemed that Nuclide would lose its
`filewatcher-<root-folder>` subscription, but would retain its
`hg-repository-watchman-subscription-primary` subscription. As such, we encoded
a heuristic where the presence of the `hg-repository-watchman-subscription-primary`
subscription without the accompanying `filewatcher-<root-folder>` subscription
was taken as a sign that (1) the user was connected to the directory via
Nuclide and (2) Nuclide's Watchman subscriptions had gotten corrupted.
Unfortunately, this was not a sufficient heuristic because it would not detect
the error case where Nuclide had lost //all// of its Watchman subscriptions!
Because `eden doctor` did not flag this, users kept using Nuclide even though
things were in a bad state, sometimes losing data as a result.
To fix that, we introduced a new tool, `nuclide-connections`, in D7598900. This
executable prints the list of local folders used as root folders in Nuclide
clients connected to the local Nuclide server. Now we no longer have to rely on
the presence of the `hg-repository-watchman-subscription-primary` Watchman
subscription as a proxy for a Nuclide connection.
This commit updates the `NuclideHasExpectedWatchmanSubscriptions` check in
`eden doctor` to use `nuclide-connections`. For every Nuclide connection that is
also an Eden mount, it verifies that the expected Watchman subscriptions exist.
As noted in the TODO in `NuclideHasExpectedWatchmanSubscriptions`, this
check is a bit brittle, but it is intended as a stopgap until we find the root cause
of the missing Watchman subscriptions and fix it.
Reviewed By: simpkins
Differential Revision: D7663214
fbshipit-source-id: 200b78ce25010222e59f3296f2412161086aa7ce
Summary: Materialized entries are stored with an unset hash. Fix the CLI.
Reviewed By: simpkins
Differential Revision: D7465192
fbshipit-source-id: 41e50c7214feb8afb1f3bf47252e4538945a5737
Summary:
My muscle memory doesn't quite know where the edenfs log file
is, so add a shortcut command to view it.
Reviewed By: simpkins
Differential Revision: D7418743
fbshipit-source-id: 290b51b63fc3573663ad53f44e50dfd073c211c8
Summary:
Update the Eden CLI to preserve the `ASAN_OPTIONS` environment variable when
starting edenfs.
Reviewed By: wez
Differential Revision: D7440148
fbshipit-source-id: c9932c5c14b5a2e040309268f95229c04715bb28
Summary:
Update the `eden clone` command to accept extra command line flags telling it
where to find the edenfs executable, as well as additional arguments to pass
to edenfs.
This enables the `test_clone_should_start_daemon()` integration test to pass in
flags to tell it how to start edenfs correctly.
Reviewed By: chadaustin
Differential Revision: D7433367
fbshipit-source-id: 0eddd93d1332e113fb85fa4b8fc87ba51d7eab2c
Summary:
Update most of the Eden CLI code to include python type annotations.
I believe the stats.py and stats_print.py are the only CLI files that do not
have complete typing information now.
Reviewed By: chadaustin
Differential Revision: D7433368
fbshipit-source-id: dfd6a064cacffeeed9147739da7064f3303de789
Summary:
Request data was coming back as null because of a race between ctx->getContextData() in FuseChannel::getOutstandingRequests() and the it initialization.
Also, added a check to verify rdata is valid before dereferencing it.
Plus, fixed debug.py where output gave an error.
Reviewed By: chadaustin
Differential Revision: D7377993
fbshipit-source-id: 8343119983c74185fd5d8cc05c2f5af63dcff99e
Summary:
Added a thrift call to return the outstanding FUSE requests.
Cli will call the thrift and print the output.
Added a unit test to test getOutstandingRequests().
Reviewed By: simpkins
Differential Revision: D7314584
fbshipit-source-id: 420790405babdb734f598e19719b487096ec53ca
Summary:
If you run `eden daemon` on a machine where sudo needs input, sudo's
output would get redirected to edenfs.log and eden daemon would appear
to hang. Worse, if you pressed ctrl-C, sudo would remain in the
background and continue to swallow keypresses from the tty, appearing
to somewhat break your shell until it gave up. The fix is to stop
redirecting stdout and stderr from Python and instead have edenfs
redirect itself into the log if given.
Reviewed By: simpkins
Differential Revision: D7223399
fbshipit-source-id: bae17b150b6594a3dd87bb88741c8dcefd1c5213
Summary:
This removes the TARGETS files from the eden github repository. The
open source buck build has been failing for several months, since buck
removed support for the thrift_library() rule.
I will potentially take a stab at adding CMake build support for Eden
at some point in the future.
Reviewed By: chadaustin
Differential Revision: D6893233
fbshipit-source-id: e6023094a807cf481ac49998c6f21b213be6c288
Summary:
We would immediately satisfy the health check and
tell the user that the system is healthy and show the pid of
the prior incarnation rather than the one for the instance
that we just launched.
This diff refactors the health checking code so that we can
share the implementation between the cli and the integration
tests; the integration tests already had code to do the right
thing for this.
Reviewed By: simpkins
Differential Revision: D6944989
fbshipit-source-id: 7c0f02c875b1b81f8f1b7521add67928200b27ed
Summary:
If we fail to lstat a mount because the FUSE daemon is
unconnected, go ahead and unmount it.
Reviewed By: simpkins
Differential Revision: D6848115
fbshipit-source-id: df797b15956db24c4ac87dc6d4cd0a7177dd20ef
Summary:
This is a fairly minimal attempt at asking sandcastle to
run our tests as root. This is inspired by D6608746 but is for
the moment pretty coarse; I just want to see how successful this
is before finishing this off nicely, which means splitting our
contbuild config into two pieces along similar lines as D6608746;
one piece would run the tests as usual without root privileges
whilst the other would run just the integration tests with root
privs.
Reviewed By: chadaustin
Differential Revision: D6612669
fbshipit-source-id: 9a730b02424e939dd3509310b6265d50d1102e77
Summary:
in our sandcastle environment, eden is not currently
installed and the tests that look at the version are throwing
an error when the rpm query fails.
Don't do that!
Reviewed By: simpkins
Differential Revision: D6853305
fbshipit-source-id: d17cb1fb0253ef5ae5000695b6ec5bcda4a6a448
Summary: A minor convenience for mass mounting and unmounting.
Reviewed By: simpkins
Differential Revision: D6803003
fbshipit-source-id: 16c0d6982ba0ce2dba9900ee15013fcec8bc5ad5
Summary:
My stale mounts check regressed eden doctor's handling of
unconfigured mounts. For now, just skip them, and add a test that
eden doctor doesn't crash.
Reviewed By: simpkins
Differential Revision: D6801445
fbshipit-source-id: 65d2f9028af651ef487a7ce6c334f387b541492d
Summary:
The prior implementation of StaleMountsCheck filtered by path
and did not correctly handle seeing the same FUSE mount multiple times
in the mount table. This occurred when an Eden mount was created
underneath a bind mount.
Now it only unmounts mounts where st_dev does not match the st_dev of
any active mounts, and where st_uid matches the current user.
Reviewed By: simpkins
Differential Revision: D6787618
fbshipit-source-id: 24e0f156cb74822500d91205349c0e6638c0340c
Summary:
It's interesting to see the total number of loaded files
vs. trees when the loaded inode count is high.
Reviewed By: wez
Differential Revision: D6765874
fbshipit-source-id: 178b30184428bd5cf5e005eb475e4f5a1476c385
Summary:
I'm not sure what was wrong with the old code, but I
simplified and clarified all of the time math and now `eden debug
unload` behaves as I'd expect it should.
Reviewed By: simpkins
Differential Revision: D6764962
fbshipit-source-id: 3ed359d4ab4652e95d1538a0982c24185999351c
Summary:
bolinfest has observed `eden doctor` unmounting active mounts, so disable
that check until this is resolved.
Reviewed By: bolinfest
Differential Revision: D6768284
fbshipit-source-id: 21add1fa283f8631055019dae99819d04e179d5e
Summary:
On closer investigation, I realized that the typechecker is
complaining about a real problem here, that was apparently hidden by
the stubs. The problem is explained in
https://our.intern.facebook.com/intern/dex/qa/1596320410425583/
It's not safe to `import thrift` followed by accessing
`thrift.Thrift.TException`; it only happens to work at runtime if some
intervening code has explicitly imported `thrift.Thrift` submodule for
you.
This diff fixes the code to be safer and also make mypy happy.
Reviewed By: chadaustin
Differential Revision: D6761089
fbshipit-source-id: 5c6fdfa0bef893c4aea17ade8e7bfacaf7c0c8cc
Summary:
As noted by the TODO, enough time has passed since the migration to the new
directory layout has been available that we should feel comfortable deleting the
migration code.
Reviewed By: wez
Differential Revision: D6762457
fbshipit-source-id: 9b4a9ad1a0204878a419362c8cfbdcbb500b1aee
Summary:
If the Eden process dies or malfunctions, it's possible to end up with stale
edenfs mounts on the system. Change `eden doctor` to correct them.
Reviewed By: simpkins
Differential Revision: D6659193
fbshipit-source-id: d9fcf2e68663295e4f43b2c11fd4503a1dfac038
Summary:
Added --version to main.py, including -v and version.
It prints both installed and running version (as per Wez's suggestion), and accounts for the possibilities that eden may not be running at all, or dev version may be running.
Reviewed By: wez
Differential Revision: D6724204
fbshipit-source-id: 5085f53a00a557f759a23fe41fb57189c9ad6a7e
Summary:
D6738525 and D6731821 (https://github.com/python/typeshed/pull/1827)
broke type checking on Eden's Python. This works around removing the thrift
stubs.
Reviewed By: carljm
Differential Revision: D6745233
fbshipit-source-id: 760f7ee199698a0a570a993f16b45c833910aade
Summary:
Changed `eden clone` to check out master commit in both git and hg.
Previously, it checked out the current commit for the repo.
Reviewed By: simpkins
Differential Revision: D6663754
fbshipit-source-id: 92b185ccca5d082dc2bde9c8b191c82a2a4f06b4
Summary:
`eden list` enumerates all of the mounts in the Eden config,
whether or not the daemon is running and whether or not the mount is
currently active or not. This diff adds an (active) suffix on a mount
if the daemon is running and the mount is currently active.
Reviewed By: simpkins
Differential Revision: D6661303
fbshipit-source-id: c098e90fc9a77f16c723c707cc4da3ee3d4c5abb
Summary: eden rage crashes sometimes due to too much inode info in the info dump. Replacing per-inode info dump with the stats summary from the stats module.
Reviewed By: chadaustin
Differential Revision: D6641312
fbshipit-source-id: 16910aa21306db4e5533217bd2ffb7b37440a807
Summary:
This is a codemod to change from using @/ to // in basic cases.
- TARGETS files with lines starting with @/ (but excluding @/third-party:
- autodeps lines in source and TARGETS files ( (dep|manual)=@/ ), excluding @/third-party
- Targets in string macros
The only thing left of the old format should be @/third-party:foo:bar
drop-conflicts
Reviewed By: ttsugriy
Differential Revision: D6605465
fbshipit-source-id: ae50de2e1edb3f97c0b839d4021f38d77b7ab64c
Summary: Previously, this would cause a crash when running `eden doctor`. Now fixed.
Reviewed By: simpkins
Differential Revision: D6607211
fbshipit-source-id: a0e077beaf6b7031d57efd72b947e90884369852
Summary:
In making this change, I was hoping to remove `.encode()` calls, but the net
number is zero.
Reviewed By: chadaustin
Differential Revision: D6606997
fbshipit-source-id: ea1851d719aff0bae424f15c94a7cc48b8f516f0
Summary:
This additional debugging info should help when investigating user issues.
While here, I also fixed things up so that `eden rage` does not crash if Eden
is not running (it just prints less information and tells the user).
Reviewed By: chadaustin
Differential Revision: D6599309
fbshipit-source-id: cf5c63dcc6f4dbf224122579c3bd458730629a66
Summary:
`eden stats` used to show system memory usage which was not very
interesting (and can be gleaned from top). Instead read the contents
of /proc/self/smaps and sum the Private_Dirty fields to get a number
that more accurately reflects impact on the rest of the system.
Reviewed By: wez
Differential Revision: D6575595
fbshipit-source-id: 9badc5cd5a1b56d3ccb27edd1a2d20ee74ec34ae
Summary:
When I updated D6577031 in response to code review feedback,
I manually tested with `--qf` but at the last minute changed to the
suggested long-form `--queryfmt` without verifying it was the
appropriate name.
Reviewed By: simpkins
Differential Revision: D6579002
fbshipit-source-id: ac2a96e7340099eb83336cb11d16dd711b0d4f9f
Summary:
I forgot that `subprocess.check_output()` returns a `bytes` instead of a `str`.
This caused `eden doctor` to return false positives because the installed and
running versions of `edenfs` would be the same, but the check would fail because
it was comparing a `bytes` to a `str`.
Reviewed By: simpkins
Differential Revision: D6577031
fbshipit-source-id: 681ea22ef79604a3dfb278d713e5c68c54d8ecd5
Summary:
This reports the difference between what is returned by the `aliveSince()`
Thrift method and the current time.
Reviewed By: chadaustin
Differential Revision: D6566127
fbshipit-source-id: 449103bcb31a87e4efd780299131eb38a45a6bd7
Summary:
`eden debug buildinfo` lists the exported values from the Thrift daemon that
start with `build_`.
Reviewed By: simpkins
Differential Revision: D6565668
fbshipit-source-id: 62009d7a23211765039a5045a517113043b5d8a9
Summary:
In the initial implementation of `eden doctor`, if you had a mount point like:
/data/users/mbolin/eden-fbsource
then it looked for a pair of Watchman subscriptions for
`/data/users/mbolin/eden-fbsource` with the names:
hg-repository-watchman-subscription-primary
filewatcher-/data/users/mbolin/eden-fbsource
as a heuristic that Nuclide was being used to edit files in that Eden mount.
However, if the user is using Nuclide to edit files in a subdirectory of that
Eden mount (as determined by the Nuclide connection profile), then the
subscriptions look like this:
hg-repository-watchman-subscription-primary
filewatcher-/data/users/mbolin/eden-fbsource/<path-to-subdirectory>
In this case, `eden doctor` was returning a false positive because it thought
that the `filewatcher-/data/users/mbolin/eden-fbsource` subscription was missing
when in fact everything was just fine.
This revision changes the logic to look for a subscription to the root or one of
its subdirectories.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D6467113
fbshipit-source-id: bfe5fafaa388405f2e59e61315c5a2084c8adc56
Summary:
This makes the `expected_output` in `stats_test.py` cleaner to declare.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D6465282
fbshipit-source-id: 80b697c84f28a0fece77eb1050c4364b14685c0b
Summary:
Presumably for historical reasons, we still had some tests hanging around with
`from __future__ import absolute_import` in the header even though all of our
tests should be Python 3. I converted these stragglers and added a smattering of
type annotations to enforce their Python 3-ness.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D6464805
fbshipit-source-id: 6177d7663ab428a0dbbecb287f340d770679551d
Summary: This is not material to the test, so it's distracting.
Reviewed By: wez
Differential Revision: D6462338
fbshipit-source-id: f289516c7e67891f1fed74df721fa1eb24e5f368
Summary:
`eden doctor` is a new subcommand that attempts to diagnose
issues with Eden and autofix them, as appropriate. There is also
a `--dry-run` flag that will tell the user about issues without attempting
to fix them.
Initially, `eden doctor` checks for the following:
- If Watchman has an `inotify` watcher, `eden doctor` identifies this and
will attempt to replace it with an `eden` watcher.
- If there are some active Watchman subscriptions that appear to come
from Nuclide, warn the user if the mission-critical `filewatcher-` subscription
that watches the entire repo is missing.
- If p1 in `SNAPSHOT` and `.hg/dirstate` do not match, warn the user.
The code for `eden doctor` is organized so that it should be relatively
easy to add new conditions to check for going forward. Admittedly, the
UX could be better by formatting the output (color, boldness, etc.) to
make certain information stand out, but we can improve that in
subsequent revisions.
Note that I had to do a bit of cleanup in `eden/cli/TARGETS` as part of
this revision and I created `eden/cli/test/TARGETS` so the tests have
their own build file.
Reviewed By: chadaustin
Differential Revision: D6446057
fbshipit-source-id: ae23c5996dba4f7f70f118179e5556efc29c31c3
Summary:
Previously, we used the Mercurial code `g` when faced with an `UNTRACKED_ADDED`
file conflict, but that was allowing merges to silently succeed that should not
have. This revision changes our logic to use the code `m` for merge, which
unearthed that we were not honoring the user's `update.check` setting properly.
Because we use `update.check=noconflict` internally at Facebook, we changed the
Eden integration tests to default to verifying Hg running with this setting. To
support it properly, we had to port this code from `update.py` in Mercurial to
our own `_determine_actions_for_conflicts()` function:
```
if updatecheck == 'noconflict':
for f, (m, args, msg) in actionbyfile.iteritems():
if m not in ('g', 'k', 'e', 'r', 'pr'):
msg = _("conflicting changes")
hint = _("commit or update --clean to discard changes")
raise error.Abort(msg, hint=hint)
```
However, this introduced an interesting issue where the `checkOutRevision()`
Thrift call from Hg would update the `SNAPSHOT` file on the server, but
`.hg/dirstate` would not get updated with the new parents until the update
completed on the client. With the new call to `raise error.Abort` on the client,
we could get in a state where the `SNAPSHOT` file had the hash of the commit
assuming the update succeeded, but `.hg/dirstate` reflected the reality where it
failed.
To that end, we changed `checkOutRevision()` to take a new parameter,
`checkoutMode`, which can take on one of three values: `NORMAL`, `DRY_RUN`, and
`FORCE`. Now if the user tries to do an ordinary `hg update` with
`update.check=noconflict`, we first do a `DRY_RUN` and examine the potential
conflicts. Only if the conflicts should not block the update do we proceed with
a call to `checkOutRevision()` in `NORMAL` mode.
To make this work, we had to make a number of changes to `CheckoutAction`,
`CheckoutContext`, `EdenMount`, and `TreeInode` to keep track of the
`checkoutMode` and ensure that no changes are made to the working copy when a
`DRY_RUN` is in effect.
One minor issue (for which there is a `TODO`) is that a `DRY_RUN` will not
report any `DIRECTORY_NOT_EMPTY` conflicts that may exist. As `TreeInode` is
implemented today, it is a bit messy to report this type of conflict without
modifying the working copy along the way.
Finally, any `UNTRACKED_ADDED` conflict should cause an update to
abort to match the behavior in stock Mercurial if the user has the following
config setting:
```
[commands]
update.check = noconflict
```
Though the original name for this setting was:
```
[experimental]
updatecheck = noconflict
```
Although I am on Mercurial 4.4.1, the `update.check` setting does not seem to
take effect when I run the integration tests, but the `updatecheck` setting
does, so for now, I set both in `hg_extension_test_base.py` with a `TODO` to
remove `updatecheck` once I can get `update.check` to do its job.
Reviewed By: simpkins
Differential Revision: D6366007
fbshipit-source-id: bb3ecb1270e77d59d7d9e7baa36ada61971bbc49
Summary:
Increase the eden daemon start timeout from 5 to 60 seconds.
With the 5 second timeout I sometimes see `buck test eden/...` time out in the
clone integration test which waits for eden to start. This likely takes longer
than normal when lots of other integration tests are also running in parallel.
Even when simply manually starting eden I have seen it take nearly 60 seconds
to open the RocksDB database. (I suspect RocksDB performs some extra checking
when opening a DB created by an older version of the code.) Eden requires the
RocksDB be open before it starts responding successfully to health check
requests.
Reviewed By: wez
Differential Revision: D6434357
fbshipit-source-id: 5f62ff821ecd64f7a1345e611f2299177513e411
Summary: Update the functions in cli/util.py with type annotations.
Reviewed By: bolinfest
Differential Revision: D6434356
fbshipit-source-id: 054d81427132229a390b8a133d5180be5f70bd20
Summary:
In onboarding users, we usually tell them to run `eden clone fbsource`,
but that fails because users generally have not run `eden daemon` yet.
The simplest thing is to do it for them when they run `eden clone` when
the daemon is not running.
Reviewed By: wez
Differential Revision: D6357249
fbshipit-source-id: dc112c1efe214485e3c5c8e06522d299a100d3a0
Summary:
This begins implementing the "client-side" portion of graceful takeover in
edenfs. When the --takeover flag is specified, if edenfs finds that another
edenfs process is already running it will attempt to gracefully take over its
mount points rather than exiting with an error.
This does not yet actually take over the mount points themselves--it still
sends dummy mount information during shutdown, and does not use this data
during startup. However, we do perform takeover of the eden lock file and the
thrift server socket.
Reviewed By: bolinfest
Differential Revision: D6038944
fbshipit-source-id: 42406a0559367cec79af088b4ca84c22de3f3ef3
Summary:
This fixes up a minor but sadly fatal oversight from D6314115; a call to
`get_repo_config` didn't get renamed to `find_config_for_alias`
(Note: this ignores all push blocking failures!)
Reviewed By: bolinfest
Differential Revision: D6354474
fbshipit-source-id: 13c665882419feebf0b8c0596eb3e0220ee2cf13
Summary:
It seemed unnecessary to include a `\n` in the template only to strip it.
Similarly, we should be strict when parsing the output since we expect it to be
a string of hex characters.
Reviewed By: wez
Differential Revision: D6322129
fbshipit-source-id: 61c483badfd7b68ed012310360aa582d6bdf5181
Summary:
When cloning an existing Eden mount, we should be smart and inherit its
underlying config so that we inherit properties such as its bind mounts.
Reviewed By: wez
Differential Revision: D6322002
fbshipit-source-id: 3f5ba135b12ad7dcecef6676d27495cfbf0ce97b
Summary:
Previously, a user had to define a config for a repo in a file like `~/.edenrc`
in order to create a new Eden mount via `eden clone`. In practice, the
information that is hardcoded in the config can generally be inferred from an
existing repo, so this expands `eden clone` to support both modes of operation.
Note this made it possible to finally unify the `RepoConfig` and `ClientConfig`
types. This revision removes `RepoConfig`, so I dutifully renamed every
local variable named `repo_config` to `client_config`.
Reviewed By: wez
Differential Revision: D6314115
fbshipit-source-id: 9625a5fbe35b30f76b6099180580c64435a4cf72
Summary:
As explained by the comment in the file, the alias for the config that was used
to create the client should not be necessary to mount it. This builds on top of
D6310325 to rewrite some internal functions that still relied on this alias
being present.
Reviewed By: wez
Differential Revision: D6313778
fbshipit-source-id: 867300e9ec844731993376b8957935f175a242f5
Summary:
The headline changes of this revision are:
- Changes the format of the config file from INI to TOML
(the `edenrc` file under `~/local/.eden` has been replaced
with `config.toml`). This revision includes logic for automatically
performing the migration when Eden is restarted.
- Inlines data from `/etc/eden/config.d` into the TOML file.
Historically, the `edenrc` file for a client would contain the
name of the "configuration alias" defined in a config file like
`~/.edenrc` or `/etc/eden/config.d/00-defaults`. When Eden
loaded a client, it would have to first read the `edenrc` and
then reconstitute the rest of the client configuration by
looking up the alias in the set of config files that were used to
create the client in the first place.
This changes things so that all of the data that was being
cross-referenced is now inlined in the client's config file.
This makes loading a config considerably simpler at the cost
of no longer being able to change the config for multiple clients
that were cloned from the same configuration alias in one place.
It was questionable whether being able to modify a client from
a foreign config after it was created was a safe thing to do, anyway.
Eliminating the need for a historic link to the configuration alias
will make it easier to support running `eden clone` on an arbitrary
local Hg or Git repo. So long as `eden clone` can extract enough
information from the local repo to create an appropriate config file
for the new Eden client, there is no need for a configuration alias
to exist a priori.
Since we were already changing the data in the config file, this
seemed like an appropriate time to make the switch from INI to
TOML, as this was something we wanted to do, anyway.
In testing, I discovered a discrepancy between how boost's
`boost::property_tree::ptree` and Python's `ConfigParser` handled
the following section heading:
```
[repository ZtmpZsillyZeden-clone.LIkh32]
```
Apparently `hasSection("repository ZtmpZsillyZeden-clone.LIkh32")`
in boost would fail to find this section. Because
[[https://stackoverflow.com/questions/13109506/are-hyphens-allowed-in-section-definitions-in-ini-files | there is no spec for INI]],
it is not that surprising that boost and `ConfigParser` do not 100% agree
on what they accept. Moving to TOML means we have a configuration
language with the following desirable properties:
- It has a formal spec, unlike INI. This is important because there are parsers
in a wide range of programming languages that, in theory, accept a consistent
input language.
- It is reasonable for humans to write, as it supports comments, unlike JSON.
- It supports nested structures, like maps and arrays, without going crazy
on the input language it supports, unlike YAML.
Eden now depends on the following third-party TOML parsers:
* C++ https://github.com/skystrife/cpptoml
* Python https://github.com/uiri/toml
This revision also changes the organization of `~/local/.eden` slightly. For now,
there is still a `config.json` file, but the values are no longer hashes of the realpath
of the mount. Instead, we take the basename of the realpath and use that as the
name of the directory under `~/local/.eden/clients`. If there is a naming collision, we
add the first available integral suffix. Using the basename makes it easier to
navigate the `~/local/.eden/clients` directory.
Although the `edenrc` file under `~/local/.eden/clients` has been switched from INI
to TOML, the other Eden config files (`~/.edenrc` and `/etc/eden/config.d/*`) still use
INI. Migrating those to TOML will be done in a future revision.
Note this revision allowed us to eliminate `facebook::eden::InterpolatedPropertyTree`
as well as a number of uses of boost due to the elimination of
`ClientConfig::loadConfigData()` in the C++ code. Because `ClientConfig`
no longer does interpolation, a bit of `ClientConfigTest` was deleted as part of
this revision because it is no longer relevant.
Reviewed By: wez
Differential Revision: D6310325
fbshipit-source-id: 2548149c064cdf8e78a3b3ce6fe667ff70f94f84
Summary:
This flips the sense of the unmount command; previously
we would default to destroying the mount and associated state,
but this was a bit of a massive sharp edge to our UX.
Now the default is non-destructive and you have to explicitly
pass in `--destroy` to enable destructive mode.
Reviewed By: chadaustin
Differential Revision: D6346013
fbshipit-source-id: ce612e7d8a70540d63217a97f96bc5760f3951af
Summary:
The messaging implies that something is already mounted, especially
if one is used to the error messages that are emitted by `mount`. In this
situation we're complaining that we already have a repo configured at that
path, so let's help point folks in the right direction in the error message.
Reviewed By: chadaustin
Differential Revision: D6344017
fbshipit-source-id: d48176f5806b3233e191567a840c88af101002cf
Summary:
Moving the post-clone step out of C++ makes it so that
`ClientConfig` in C++ no longer knows about `hooks` and requires
`RepoConfig` to know about `hooks`. This helps us reduce
`ClientConfig`'s dependency on `ConfigData`, as my next step
is to move the remaining information that `ClientConfig` gets from
`ConfigData` directly into the client's `edenrc` file under
`~/local/.eden`.
Reviewed By: chadaustin
Differential Revision: D6310544
fbshipit-source-id: dec7a21281ab49e0416b8872757970a4eff2d943
Summary:
This will facilitate an upcoming change where I include the `hooks` information
as part of the data returned by `get_repo_config()`
(formerly `get_repo_data()`).
Reviewed By: chadaustin
Differential Revision: D6311536
fbshipit-source-id: 2e1bc063c4a12772c9806f6880822fb15e18fc92
Summary:
This seems like something I missed during the original code review in D3554550.
In practice, there probably isn't much contention for this lock, so it's not
surprising that we haven't knowingly been bitten by this yet.
Reviewed By: chadaustin
Differential Revision: D6310326
fbshipit-source-id: c16a07aa654240be77a38c16473a4cc1e06125b6
Summary:
Unless you're going to clean up and then reraise, catching
BaseException rarely makes sense, since it swallows keyboard
interrupts.
Reviewed By: simpkins, bolinfest
Differential Revision: D6262224
fbshipit-source-id: b6b9bc963a4c75fe223c3b787ba822c3cf5da24d
Summary:
This is a major change to how we manage the dirstate in Eden's Hg extension.
Previously, the dirstate information was stored under `$EDEN_CONFIG_DIR`,
which is Eden's private storage. Any time the Mercurial extension wanted to
read or write the dirstate, it had to make a Thrift request to Eden to do so on
its behalf. The upside is that Eden could answer dirstate-related questions
independently of the Python code.
This was sufficiently different than how Mercurial's default dirstate worked
that our subclass, `eden_dirstate`, had to override quite a bit of behavior.
Failing to manage the `.hg/dirstate` file in a way similar to the way Mercurial
does has exposed some "unofficial contracts" that Mercurial has. For example,
tools like Nuclide rely on changes to the `.hg/dirstate` file as a heuristic to
determine when to invalidate its internal caches for Mercurial data.
Today, Mercurial has a well-factored `dirstatemap` abstraction that is primarily
responsible for the transactions with the dirstate's data. With this split, we can
focus on putting most of our customizations in our `eden_dirstate_map` subclass
while our `eden_dirstate` class has to override fewer methods. Because the
data is managed through the `.hg/dirstate` file, transaction logic in Mercurial that
relies on renaming/copying that file will work out-of-the-box. This change
also reduces the number of Thrift calls the Mercurial extension has to make
for operations like `hg status` or `hg add`.
In this revision, we introduce our own binary format for the `.hg/dirstate` file.
The logic to read and write this file is in `eden/py/dirstate.py`. After the first
40 bytes, which are used for the parent hashes, the next four bytes are
reserved for a version number for the file format so we can manage file format
changes going forward.
Admittedly one downside of this change is that it is a breaking change.
Ideally, users should commit all of their local changes in their existing mounts,
shutdown Eden, delete the old mounts, restart Eden, and re-clone.
In the end, this change deletes a number of Mercurial-specific code and Thrift
APIs from Eden. This is a better separation of concerns that makes Eden more
SCM-agnostic. For example, this change removes `Dirstate.cpp` and
`DirstatePersistance.cpp`, replacing them with the much simpler and more
general `Differ.cpp`. The Mercurial-specific logic from `Dirstate.cpp` that turned
a diff into an `hg status` now lives in the Mercurial extension in
`EdenThriftClient.getStatus()`, which is much more appropriate.
Note that this reverts the changes that were recently introduced in D6116105:
we now need to intercept `localrepo.localrepository.dirstate` once again.
Reviewed By: simpkins
Differential Revision: D6179950
fbshipit-source-id: 5b78904909b669c9cc606e2fe1fd118ef6eaab95
Summary:
Apparently `Dict[str, [int, int, int, int]]` is not a valid MyPy type.
Fixed to be `Dict[str, Tuple[int, int, int, int]]`.
This was introduced in a recent change (D6189543).
Reviewed By: danzimm
Differential Revision: D6241810
fbshipit-source-id: a32da67be0c555f985669e7fc2d8058f469bb371
Summary: Adds an `eden stats thrift` command that prints call counts for each Thrift call, similar to `eden stats io` printing FUSE counts.
Reviewed By: bolinfest
Differential Revision: D6189543
fbshipit-source-id: 59c29a4036687e1bf75b1c0ca2a7d311b9d1399f
Summary:
Using this option prints the rage report to stdout rather than to the default
reporter. Sometimes it's easier to get the report on stdout so you can pipe it
to other commands.
Reviewed By: simpkins
Differential Revision: D6195684
fbshipit-source-id: 59e1ae6b19b951eac67377cdd5f8c0560e0d181a
Summary:
This was an error that an end-user ran into. Previously, we did not fail
gracefully and the user was faced with an intimidating stacktrace.
Reviewed By: simpkins
Differential Revision: D6195529
fbshipit-source-id: bde3c2a3e6f49457a4c6ac5c87103cf52cd227c2
Summary:
Previous to this change, the user got an inscrutable error message. It turns out
that it is easy to make this mistake, typing `eden clone fbsource/` instead of
`eden clone fbsource` if you accidentally use tab completion.
Reviewed By: simpkins
Differential Revision: D6153889
fbshipit-source-id: 3642fdd207d6abf896d6a12891d5eb68ad984acc
Summary:
Add an `eden debug hg_dirstate` command to dump the contents of the Hg dirstate.
This data is stored in a binary format, so we need a custom command to view it
easily.
Reviewed By: simpkins
Differential Revision: D6139172
fbshipit-source-id: 622c0b7bcaa471a88483c6c4ddef7e0be95a3dfa
Summary: This diff helps some common pitfalls when using set_log_level.
Reviewed By: simpkins
Differential Revision: D6142849
fbshipit-source-id: 7fa35392dda148af90d0aefdb872b6e8a8b770db
Summary:
We have encountered cases where `eden health` reported
`"edenfs not healthy: edenfs not running"` even though the `edenfs` process is
still running. Because the existing implementation of `eden health` bases its
health check on the output of a `getStatus()` Thrift call, it will erroneously
report `"edenfs not running"` even if Eden is running but its Thrift server is
not running. This type of false negative could occur if `edenfs` has shutdown
the Thrift server, but not the rest of the process (quite possibly, its
shutdown is blocked on calls to `umount2()`).
This is further problematic because `eden daemon` checks `eden health`
before attempting to start the daemon. If it gets a false negative, then
`eden daemon` will forge ahead, trying to launch a new instance of the daemon,
but it will fail with a nasty error like the following:
```
I1017 11:59:25.188414 3064499 main.cpp:81] Starting edenfs. UID=5256, GID=100, PID=3064499
terminate called after throwing an instance of 'std::runtime_error'
what(): another instance of Eden appears to be running for /home/mbolin/local/.eden
*** Aborted at 1508266765 (Unix time, try 'date -d 1508266765') ***
*** Signal 6 (SIGABRT) (0x1488002ec2b3) received by PID 3064499 (pthread TID 0x7fd0d3787d40) (linux TID 3064499) (maybe from PID 30644
99, UID 5256), stack trace: ***
@ 000000000290d3cd folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
@ 00007fd0d133cacf (unknown)
@ 00007fd0d093e7c8 __GI_raise
@ 00007fd0d0940590 __GI_abort
@ 00007fd0d1dfeecc __gnu_cxx::__verbose_terminate_handler()
@ 00007fd0d1dfcdc5 __cxxabiv1::__terminate(void (*)())
@ 00007fd0d1dfce10 std::terminate()
@ 00007fd0d1dfd090 __cxa_throw
@ 00000000015fe8ca facebook::eden::EdenServer::acquireEdenLock()
@ 000000000160f27b facebook::eden::EdenServer::prepare()
@ 00000000016107d5 facebook::eden::EdenServer::run()
@ 000000000042c4ee main
@ 00007fd0d0929857 __libc_start_main
@ 0000000000548ad8 _start
Aborted
```
By providing more accurate information to `eden daemon`, if the user tries to
run it while the daemon is already running, they will get a more polite error
like the following:
```
error: edenfs is already running (pid 274205)
```
This revision addresses this issue by writing the PID of `edenfs` in the
lockfile. It updated the implementation of `eden health` to use the PID in the
lockfile to assess the health of Eden if the call to `getStatus()` fails. It
does this by running:
```
ps -p PID -o comm=
```
and applying some heuristics on the output to assess whether the command
associated with that process is the `edenfs` command. If it is, then
`eden health` reports the status as `STOPPED` whereas previously it would report
it as `DEAD`.
Reviewed By: wez
Differential Revision: D6086473
fbshipit-source-id: 825421a6818b56ddd7deea257a92c070c2232bdd
Summary:
In practice, we have seen cases where `eden shutdown` does not succeed within
15 seconds (the default timeout), and then we see the following message printed
to the console:
error: sent shutdown request, but edenfs did not exit within 15.0 seconds
Sometimes we end up in a state where the Thrift server has successfully shutdown
but the `edenfs` process is still running. Because `eden health` is implemented
using a Thrift call, it reports that `edenfs` is not running even though it
still is. This is confusing because a subsequent `eden daemon` request will
likely fail with something like:
```
terminate called after throwing an instance of 'std::runtime_error'
what(): another instance of Eden appears to be running for /home/mbolin/local/.eden
```
To address this, we make `eden shutdown` more aggressive in killing `edenfs`
by sending `SIGKILL`.
Follow-up work:
- Make Eden's shutdown process more robust so that `SIGKILL` is rarely, if ever,
necessary.
- Store the PID of the `edenfs` process in the lock file so that Eden does not
rely upon the Thrift server to be running in order to check whether the
`edenfs` process is running.
Reviewed By: wez
Differential Revision: D6080929
fbshipit-source-id: 462a6ec506cdb6a51211d2c64b777eadbcee9fc1
Summary:
This adds support for specifying where the output of `eden rage` should be sent
by adding a `reporter` property to the `[rage]` section of `~/.edenrc` like so:
```
[rage]
reporter = arc paste
```
If specified, Eden will take the command as an opaque string and use it as an
argument to `sh -c <reporter_from_edenrc>` when spawning a subprocess that will
receive the output of `eden rage` on its stdin. If no reporter is specified,
then the stream will be written directly to stdout.
Reviewed By: wez
Differential Revision: D6049355
fbshipit-source-id: beafe37eed8c48dac7499a6978cacfb44d8f253e
Summary:
Previously, `$HOME` was used as the location of the home dir when determining
the config dir even if `--home-dir` was specified. Now `--home-dir` affects
both.
Reviewed By: simpkins
Differential Revision: D5997870
fbshipit-source-id: 11e9d4fccc558da0c90fb2e38071ebc2d33491bf
Summary:
Add an `--strace <FILE>` flag to the `eden daemon` CLI command to run eden
under strace, saving the strace output to the specified path.
Reviewed By: wez
Differential Revision: D5771462
fbshipit-source-id: fe4bf18f372f3276400bee624e906ed4f3569735
Summary:
Make the CLI read and write a SNAPSHOT file that is
consistent with the C++ server implementation.
Ideally we'd only ever write this file from the C++ side.
Reviewed By: simpkins
Differential Revision: D5740079
fbshipit-source-id: 2057df0ee2b0b271a4734d58e1b6d1334a28020b
Summary:
Added new tool to report stat information of EdenFs like fuse counters, Memory counters, latencies, Inode status for all the mount points etc.
eden stat : Prints the general information about eden like list of mount points, loaded unloaded and materialized inodes in each mount point. Also this reports how well periodic unload job is doing by reporting the number of unloaded inodes by periodic job.
eden stat io : Prints how many number of calls made to a system call in Edenfs.
eden stat memory : returns the memory stat for edenfs.
eden stat latency : reports the latencies of system calls in Edenfs.
Reviewed By: bolinfest
Differential Revision: D5660345
fbshipit-source-id: 97a1c2b83a6d8df0cd1b82c4d54b52d7ebd126bd
Summary:
This test was supposed to be a part of D5627411 but it was causing strange behaviour so was brought to a separate diff for further investigation.
After investigating, the test didn't pass because the UnloadedInodeData struct only contained the name of the file, not the path to it. The fix for this was to implement a way to get the relative path of the file even after the inode is unloaded.
Reviewed By: simpkins
Differential Revision: D5646929
fbshipit-source-id: f166398a651e8aea49da7e4474a5ad7fde2eaa4e
Summary:
Also modified the `getpath` handler to defer the `os.getcwd()` call.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D5698232
fbshipit-source-id: 9483907771fd1fd2918f62120664bb0d8c431cf3
Summary: This seems a little more user-friendly.
Reviewed By: bradenwatling
Differential Revision: D5686562
fbshipit-source-id: 8142fb9105a3a44823f935fc04187cf0ed2258d7
Summary:
Note that this feature was mostly implemented before this commit, but never
tested. Unsurprisingly, there were bugs.
This change also introduces a new `eden debug hg_copy_map_get_all` subcommand
because that was a straightforward way to verify the internal state of the copy
map on the server side from an integration test.
Adding this test uncovered a key copy/paste bug in `EdenThriftClient.py`
(`hgCopyMapGet` was being invoked instead of `hgCopyMapPut`.)
It also uncovered a bug in `LameThriftClient` because the `compile()` and
`eval()` calls on the output are not appropriate when the return type of the
Thrift endpoint is `string`.
Reviewed By: simpkins
Differential Revision: D5686114
fbshipit-source-id: f0093d2b67062c01982dc5bc1f0db2774b3a9356
Summary:
Added a test case `test_unload_free_inodes_age` to verify the behaviour of unloadChildrenNow with age parameter.
Added new parameter age to `unloadInodeForPath` in eden.thrift, and `EdenServiceHandler`.
Modified `do_unload_inodes` function in `debug.py` to support the new behaviour.
Reviewed By: simpkins
Differential Revision: D5565859
fbshipit-source-id: a35053725be26bc906cf158969cbe21db1cbadde
Summary: Provide a thrift interface to invalidate the cache for an inode denoted by path.
Reviewed By: simpkins
Differential Revision: D5655387
fbshipit-source-id: 887aa4963d216a0d8eed93b6fb8721632cc31d19
Summary:
This is a convenient way to test the `hgGetDirstateTuple()` endpoint in
`eden.thrift`.
Reviewed By: quark-zju, wez
Differential Revision: D5654237
fbshipit-source-id: 0b285e056002d4556733a53293582345f36780b2
Summary: Add a command to lookup the path for an inode given the inode number and eden mount path.
Reviewed By: bolinfest
Differential Revision: D5627411
fbshipit-source-id: 25928f506d3f48d8a6784fe81fb17fa0500d6bc9
Summary:
I'm trying to troubleshoot why a new dir that I added isn't showing up in the overlay and I found that the debug tool doesn't know about the overlay.
This is the dumbest minimal thing I could do to make this basically functional again
Reviewed By: simpkins
Differential Revision: D5602439
fbshipit-source-id: 4ae7fa34136697f9f915ccd95275cb3a7923ae4a
Summary:
This updates the TreeInode code to remove the redundant materialized flag.
A TreeInode should have a Tree Hash if and only if it is dematerialized, so
there is no need for an extra `materialized` boolean.
This diff also fixes an issue in TreeInode::saveOverlayPostCheckout() where it
was not correctly informing it's parent TreeInode of the change if it moved
from one dematerialized state to another (with a different TreeInode hash).
This fixes the code to correctly call `parent->childDematerialized()` when it
needs to inform the parent that it now refers to a different source control
hash.
Reviewed By: wez
Differential Revision: D5336629
fbshipit-source-id: b4d86ecdef2f5faefbc243a09d869c02384ae95c
Summary:
Add a command to deserialize and display information about files in the
overlay. This can be used to help debug the current state of files in the
overlay.
Reviewed By: wez
Differential Revision: D5332014
fbshipit-source-id: 25b01579df33aa9f1926c0144e9f03aa8ece38fc
Summary:
1. Moved read, write, mkdir, rm methods in hg/lib/hg_extension_test_base.py to lib/test_case.py.
2. Added integration test case to test unload free inodes.
Reviewed By: simpkins
Differential Revision: D5277870
fbshipit-source-id: b93b6049a10357cf8c92366e6dca3968f7f30c30
Summary:
1.Added a new thrift method to unload free inodes of a directory and its sub directories.
2.Added a new debug sub command 'eden debug unload <path>' to cli tools to unload free inodes.
Reviewed By: simpkins
Differential Revision: D5261038
fbshipit-source-id: 85b4c5ae18c0ae24c666a44ac9892765e753397f
Summary:
1.Added new rage command to command line tools.
2.Eden rage command currently shows Logs,package version,rpm version,build revision,build upstream revision,list of running eden processes and their info,list of mount points and their info.
Reviewed By: simpkins
Differential Revision: D5220250
fbshipit-source-id: 357f46d8d08d4a1f197b705dfd1a28668dd180f0
Summary: made the relative path return "" when mount point directory is returned "."
Reviewed By: simpkins
Differential Revision: D5250543
fbshipit-source-id: bd0d648db8030cfc6ca952fccfb9972f18290b5e
Summary:
When we say we are building "all" of Eden, we generally want to ensure we have
built the pyremote service that the Hg extension needs, as well.
Reviewed By: wez
Differential Revision: D4982118
fbshipit-source-id: 7282266e82e662a48b5ced6da6b7daedab8b65ea
Summary: We overlooked this in the recent move around of the cli code
Reviewed By: bolinfest
Differential Revision: D4894230
fbshipit-source-id: 5c3ae274a182e5cc41df47100959f8cfbcb59c2d
Summary:
Move the code for the command-line tool up one directory, out of eden/fs.
This better separates the code so that eden/fs contains code for the edenfs
daemon, while eden/cli contains code for the command line tool.
Reviewed By: bolinfest
Differential Revision: D4888633
fbshipit-source-id: 5041e292c5353d05122eefe5db3257289e31239a
