Summary:
This makes the `expected_output` in `stats_test.py` cleaner to declare.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D6465282
fbshipit-source-id: 80b697c84f28a0fece77eb1050c4364b14685c0b
Summary:
Presumably for historical reasons, we still had some tests hanging around with
`from __future__ import absolute_import` in the header even though all of our
tests should be Python 3. I converted these stragglers and added a smattering of
type annotations to enforce their Python 3-ness.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D6464805
fbshipit-source-id: 6177d7663ab428a0dbbecb287f340d770679551d
Summary:
In D6446057, I added a new entry to the dict returned by
`config.get_client_info()`. I only ran the cli tests while working on D6446057,
but I should have ran all of the tests because there was an integration test
(`InfoTest`) that verified the return value of this method, so it broke due to
my change.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D6464806
fbshipit-source-id: 1b0ac0853301ba33e5e948353e4c89c0d97c0d83
Summary:
Replace the initLoggingGlogStyle() function with a more generic initLogging()
function that accepts a log config string to be parsed with parseLogConfig().
Reviewed By: bolinfest, yfeldblum
Differential Revision: D6342086
fbshipit-source-id: fb1bffd11f190b70e03e2ccbf2b30be08d655242
Summary:
Make sure `hg update --clean` clears the merge state data. In non-clean
updates this is performed in `mercurial.merge.applyupdates()`. However, we
never call `applyupdates()` on clean updates in eden.
Reviewed By: bolinfest
Differential Revision: D6456720
fbshipit-source-id: b40d02ca0fb677bcde82822a8eafd5fcf926dae6
Summary: This is not material to the test, so it's distracting.
Reviewed By: wez
Differential Revision: D6462338
fbshipit-source-id: f289516c7e67891f1fed74df721fa1eb24e5f368
Summary:
`eden doctor` is a new subcommand that attempts to diagnose
issues with Eden and autofix them, as appropriate. There is also
a `--dry-run` flag that will tell the user about issues without attempting
to fix them.
Initially, `eden doctor` checks for the following:
- If Watchman has an `inotify` watcher, `eden doctor` identifies this and
will attempt to replace it with an `eden` watcher.
- If there are some active Watchman subscriptions that appear to come
from Nuclide, warn the user if the mission-critical `filewatcher-` subscription
that watches the entire repo is missing.
- If p1 in `SNAPSHOT` and `.hg/dirstate` do not match, warn the user.
The code for `eden doctor` is organized so that it should be relatively
easy to add new conditions to check for going forward. Admittedly, the
UX could be better by formatting the output (color, boldness, etc.) to
make certain information stand out, but we can improve that in
subsequent revisions.
Note that I had to do a bit of cleanup in `eden/cli/TARGETS` as part of
this revision and I created `eden/cli/test/TARGETS` so the tests have
their own build file.
Reviewed By: chadaustin
Differential Revision: D6446057
fbshipit-source-id: ae23c5996dba4f7f70f118179e5556efc29c31c3
Summary:
Always call `umount2()` with both `MNT_FORCE` and `MNT_DETACH`. Without
`MNT_DETACH` I see unmount operations still fail or hang in some cases on
edenfs shutdown. This should hopefully help fully eliminate situations where
disconnected mount points are sometimes left behind when eden shuts down.
Reviewed By: bolinfest, chadaustin
Differential Revision: D6455971
fbshipit-source-id: 5abe456e17a33a0080ad94b4d315540bce0c2f82
Summary:
Update TreeInode::computeCheckoutActions() to short circuit on non-materialized
directories that are identical to both the destination and source state. This
speeds up `hg update --clean` operations by allowing us to skip portions of the
tree that do not require changes.
This does not have much effect for update operations without `--clean`: those
already short-circuited the operation in `TreeInode::processCheckoutEntry()`
when the source and destination hashes were the same. That code cannot
short-circuit in the forced update case, since the forced update may have to
revert local modifications.
Reviewed By: bolinfest
Differential Revision: D6455970
fbshipit-source-id: 393acb3272745751a56e06dba0c7505ff2bfad44
Summary:
Eden already disables sqldirstate as part of its post-clone hook. Add
treedirstate to this handling.
Reviewed By: bolinfest
Differential Revision: D6424041
fbshipit-source-id: 889c07ed32f58c50c15de3a9aa3135018c4761a9
Summary:
It simplifies next diff a bit.
Also removes unnecessary `makeFuture()` calls.
Reviewed By: wez
Differential Revision: D6435694
fbshipit-source-id: ac00bed90454fef645a4fdad254ba6c8d3bda0ab
Summary: Additional thread for http server is not necessary.
Reviewed By: wez
Differential Revision: D6435695
fbshipit-source-id: b4fc7096a87146f36876bd132db66ab964feba51
Summary:
Previously, we used the Mercurial code `g` when faced with an `UNTRACKED_ADDED`
file conflict, but that was allowing merges to silently succeed that should not
have. This revision changes our logic to use the code `m` for merge, which
unearthed that we were not honoring the user's `update.check` setting properly.
Because we use `update.check=noconflict` internally at Facebook, we changed the
Eden integration tests to default to verifying Hg running with this setting. To
support it properly, we had to port this code from `update.py` in Mercurial to
our own `_determine_actions_for_conflicts()` function:
```
if updatecheck == 'noconflict':
for f, (m, args, msg) in actionbyfile.iteritems():
if m not in ('g', 'k', 'e', 'r', 'pr'):
msg = _("conflicting changes")
hint = _("commit or update --clean to discard changes")
raise error.Abort(msg, hint=hint)
```
However, this introduced an interesting issue where the `checkOutRevision()`
Thrift call from Hg would update the `SNAPSHOT` file on the server, but
`.hg/dirstate` would not get updated with the new parents until the update
completed on the client. With the new call to `raise error.Abort` on the client,
we could get in a state where the `SNAPSHOT` file had the hash of the commit
assuming the update succeeded, but `.hg/dirstate` reflected the reality where it
failed.
To that end, we changed `checkOutRevision()` to take a new parameter,
`checkoutMode`, which can take on one of three values: `NORMAL`, `DRY_RUN`, and
`FORCE`. Now if the user tries to do an ordinary `hg update` with
`update.check=noconflict`, we first do a `DRY_RUN` and examine the potential
conflicts. Only if the conflicts should not block the update do we proceed with
a call to `checkOutRevision()` in `NORMAL` mode.
To make this work, we had to make a number of changes to `CheckoutAction`,
`CheckoutContext`, `EdenMount`, and `TreeInode` to keep track of the
`checkoutMode` and ensure that no changes are made to the working copy when a
`DRY_RUN` is in effect.
One minor issue (for which there is a `TODO`) is that a `DRY_RUN` will not
report any `DIRECTORY_NOT_EMPTY` conflicts that may exist. As `TreeInode` is
implemented today, it is a bit messy to report this type of conflict without
modifying the working copy along the way.
Finally, any `UNTRACKED_ADDED` conflict should cause an update to
abort to match the behavior in stock Mercurial if the user has the following
config setting:
```
[commands]
update.check = noconflict
```
Though the original name for this setting was:
```
[experimental]
updatecheck = noconflict
```
Although I am on Mercurial 4.4.1, the `update.check` setting does not seem to
take effect when I run the integration tests, but the `updatecheck` setting
does, so for now, I set both in `hg_extension_test_base.py` with a `TODO` to
remove `updatecheck` once I can get `update.check` to do its job.
Reviewed By: simpkins
Differential Revision: D6366007
fbshipit-source-id: bb3ecb1270e77d59d7d9e7baa36ada61971bbc49
Summary:
Add a method to LogHandler to return its current configuration. This will
make it possible to query the LoggerDB for its current configuration state.
Reviewed By: bolinfest
Differential Revision: D6200563
fbshipit-source-id: 2b8b9752bbeb26c8aac28d1a73b7e2312fd198c8
Summary:
I accidentally broke the flatmanifest fallback code in D6333613 by changing the
exception type thrown for errors received from hg_import_helper.py but not
updating the catch statement in HgImporter::importTreeImpl().
This updates importTreeImpl() to catch the new HgImportPyError type correctly.
I have dropped the check on the error message entirely, since the mercurial
python code can throw a variety of errors that all mean this tree data isn't
available.
Reviewed By: bolinfest
Differential Revision: D6434359
fbshipit-source-id: c62d3c1667681712293873de2b9bf6d9220da767
Summary:
Increase the eden daemon start timeout from 5 to 60 seconds.
With the 5 second timeout I sometimes see `buck test eden/...` time out in the
clone integration test which waits for eden to start. This likely takes longer
than normal when lots of other integration tests are also running in parallel.
Even when simply manually starting eden I have seen it take nearly 60 seconds
to open the RocksDB database. (I suspect RocksDB performs some extra checking
when opening a DB created by an older version of the code.) Eden requires the
RocksDB be open before it starts responding successfully to health check
requests.
Reviewed By: wez
Differential Revision: D6434357
fbshipit-source-id: 5f62ff821ecd64f7a1345e611f2299177513e411
Summary: Update the functions in cli/util.py with type annotations.
Reviewed By: bolinfest
Differential Revision: D6434356
fbshipit-source-id: 054d81427132229a390b8a133d5180be5f70bd20
Summary: Annotate more integration test functions with type information.
Reviewed By: bolinfest
Differential Revision: D6434358
fbshipit-source-id: b88351eebee58561465752378c6771b7b1f9554e
Summary:
This fixes a crash in EdenMount::destroy() if EdenMount::create() failed to
load the root inode. Previously the code called shutdownImpl() in this case
which tried to unload all inodes and crashed since the root inode was null.
This also fixes EdenMount::destroy() to properly handle the FUSE_ERROR and
FUSE_DONE cases.
Reviewed By: wez
Differential Revision: D6434355
fbshipit-source-id: 39c5f4472d6ebbcf881b4c9c8c8fd67686032ec1
Summary:
There's a bug in some combination of Eden and FUSE where open(O_TRUNC)
followed by a sequence of writes over an existing file does not flush
the kernel's VFS page cache, which manifests as an mmap larger than
the file's size not zeroing the data beyond the file's size. These
tests attempt capture that use case, but they are fiddly.
Disabling ATOMIC_O_TRUNC seems to resolve the issue.
Reviewed By: wez
Differential Revision: D6430152
fbshipit-source-id: f7626e268e778ebab60c66322e0ce42bce746ae1
Summary:
The goal is to provide a fast path for watchman to flesh
out the total set of changed files when it needs relay that information
on to consumers.
We choose not to include the full list in the Journal when checking out
between revisions because it will not always be needed and may be an
expensive `O(repo)` operation to compute. This means that watchman
needs to expand that information for itself, and that is currently
a fairly slow query to invoke through mercurial.
Since watchman is responding to journal events from eden we know that
we have tree data for the old and new hashes and thus we should be
able to efficiently compute that diff.
This implementation is slightly awful because it will instantiate an
unlinked TreeInode object for one side of the query, and will in
turn populate any children that differ as it walks down the tree.
A follow on diff will look at making a flavor of the diff code that
can diff raw Tree objects instead.
Reviewed By: bolinfest
Differential Revision: D6305844
fbshipit-source-id: 7506c9ba1f4febebcdc283c414261810a3951588
Summary:
In onboarding users, we usually tell them to run `eden clone fbsource`,
but that fails because users generally have not run `eden daemon` yet.
The simplest thing is to do it for them when they run `eden clone` when
the daemon is not running.
Reviewed By: wez
Differential Revision: D6357249
fbshipit-source-id: dc112c1efe214485e3c5c8e06522d299a100d3a0
Summary:
This should have been removed as part of D6179950.
Ideally, Buck would error when this happens, but apparently `glob()` does not
complain when patterns do not match any files, even when the pattern does not
contain any wildcards. There appears to be some code at Facebook that is
exploiting this behavior.
Reviewed By: simpkins
Differential Revision: D6421529
fbshipit-source-id: c6f982624e0e12a911bc12ab1e8239ba4358ea56
Summary:
This corrects a bug where the in-memory timestamp for new files would
be set to the last checkout time instead of when the file was created.
Reviewed By: simpkins
Differential Revision: D6366189
fbshipit-source-id: c5fa8c779726d3a75c2d57b2a161293297eb9271
Summary:
Per a discussion with wez, let's make it explicit in the comments that the kernel handles
O_EXCL for us.
Reviewed By: simpkins
Differential Revision: D6365157
fbshipit-source-id: 638f67cd89a9450ff084e0ef77c731ef738bf518
Summary:
In some workloads we're seeing folks run out of file descriptors.
We forgot that we'd taken out the code that closes the underlying fds.
This diff takes a run at adding a simple counter of the open file handle
objects that is incremented when they are constructed and decremented
when they are destroyed.
When the count falls to zero we release the file handle.
Note that we unconditionally open files when we first load the inodes
from the overlay. I tried to defer that open attempt and it broke
the timestamp overlay test. I think we can revisit that aspect in
a follow on diff; for now we should be more resilient to transiently
opened files from things like ripgrep or similar.
Reviewed By: simpkins
Differential Revision: D6097090
fbshipit-source-id: 9a48220002e760fb1ffb8d7e2a68fa7036558b78
Summary: It is currently unused. Let's bring it back if/when we need it.
Reviewed By: chadaustin
Differential Revision: D6368867
fbshipit-source-id: 096015ba597a6e04f544273ba9773576429e39ce
Summary:
This bug is part of a bigger issue in our Mercurial integration where
`UNTRACKED_ADDED` conflicts are being silently swallowed in our Hg extension
whereas stock Mercurial presents these as conflicts and forces the user to deal
with them. The Mercurial issues will be addressed in a follow-up change.
Reviewed By: simpkins
Differential Revision: D6365580
fbshipit-source-id: 831e27ce1da90ea605033b2b9988fe400ba404aa
Summary:
We now run two versions of this test: one where the file that exists in the
destination commit is untracked before the update and one in which it is added
before the update.
Reviewed By: simpkins
Differential Revision: D6334002
fbshipit-source-id: ef6bffa27bc18171b5e21dc284c7a21aa6e35da4
Summary: Addresses outstanding TODO now that D6322052 has landed.
Reviewed By: simpkins
Differential Revision: D6368884
fbshipit-source-id: 497c42466e05af0f1690bc6401b1d271de691e58
Summary:
In preparation for bringing D6097090 across the finish line, we need a
more explicit mechanism for determining which state the inode is in.
The issue is that whether an inode is materialized was checked in a
couple ways: the nonexistence of the hash field or the definedness of
the file field. This diff introduces an explicit enum indicating the
state.
Reviewed By: simpkins
Differential Revision: D6325955
fbshipit-source-id: 3682a4ebc9330193baadbb33a4dd9845f26e59a6
Summary:
Define constants at the top of EdenServer.cpp for the names of the main lock
file, the thrift socket, and the takeover socket.
Reviewed By: bolinfest
Differential Revision: D6295040
fbshipit-source-id: 8605840a50c84bc89b798123d1063bbb11ff2502
Summary:
This begins implementing the "client-side" portion of graceful takeover in
edenfs. When the --takeover flag is specified, if edenfs finds that another
edenfs process is already running it will attempt to gracefully take over its
mount points rather than exiting with an error.
This does not yet actually take over the mount points themselves--it still
sends dummy mount information during shutdown, and does not use this data
during startup. However, we do perform takeover of the eden lock file and the
thrift server socket.
Reviewed By: bolinfest
Differential Revision: D6038944
fbshipit-source-id: 42406a0559367cec79af088b4ca84c22de3f3ef3
Summary:
Update the TakeoverData to also include the thrift server socket.
Also update EdenServer to set this field when performing a takeover
shutdown.
Reviewed By: bolinfest
Differential Revision: D6038945
fbshipit-source-id: 725faa431b3b55d617ef645c8a7eae080e4fe066
Summary:
Update EdenServer to implement the TakeoverHandler API, and to exit after
sending the mount point takeover data. The actual shutdown logic itself is not
implemented yet--this just sends dummy data for now. However, this does serve
as a proof of concept that the TakeoverServer and TakeoverClient code functions
as desired.
Reviewed By: bolinfest
Differential Revision: D6018180
fbshipit-source-id: c19581928926a46b767f1ee5c1761381e5055fa9
Summary:
This adds a new class which listens on a Unix domain socket for clients that
wish to gracefully take over Eden's FUSE mount points. The goal is to
eventually enable graceful restart functionality for eden.
It would be nice if we could use the existing thrift server socket for this,
but thrift doesn't provide low-enough level APIs so that we can send
credentials and file descriptors over the socket using SCM_CREDENTIALS and
SCM_RIGHTS. Using our own separate socket is the easiest way to accomplish
this instead.
For now eden just listens on this socket and logs a message when a client
connects; this diff does not yet contain logic for performing mount point
takeover.
Reviewed By: bolinfest
Differential Revision: D5827752
fbshipit-source-id: 928e541efa2546cb612da2699ff0bd822bafaad5
Summary:
Eden attempts to initialize timestamps of newly loaded inodes with the time of
the last checkout operation performed in this mount. Unfortunately it had a
number of bugs in this logic:
EdenMount had two separate fields attempting to track the last checkout time:
`lastCheckoutTime_` and `parentInfo_.lastCheckoutTime`.
Unfortunately neither field was actually updated on checkout operations.
Additionally, `lastCheckoutTime_` did not have any locking to allow it to be
updated. `parentInfo_.lastCheckoutTime` did have locking, but it used the
mount point's checkout lock, so it could not be accessed during checkout
operations.
This diff removes `parentInfo_.lastCheckoutTime`, keeping only
`lastCheckoutTime_`. It also converts `lastCheckoutTime_` to a
`struct timespec` since this is most often needed as a `timespec`. It also
adds a new mountpoint-wide lock for synchronizing accessing to this variable.
Reviewed By: bolinfest
Differential Revision: D6356698
fbshipit-source-id: db54f9bb297b5febe4642e2b3fcc8055a6afc199
Summary:
This fixes up a minor but sadly fatal oversight from D6314115; a call to
`get_repo_config` didn't get renamed to `find_config_for_alias`
(Note: this ignores all push blocking failures!)
Reviewed By: bolinfest
Differential Revision: D6354474
fbshipit-source-id: 13c665882419feebf0b8c0596eb3e0220ee2cf13
Summary:
It seemed unnecessary to include a `\n` in the template only to strip it.
Similarly, we should be strict when parsing the output since we expect it to be
a string of hex characters.
Reviewed By: wez
Differential Revision: D6322129
fbshipit-source-id: 61c483badfd7b68ed012310360aa582d6bdf5181
Summary:
When cloning an existing Eden mount, we should be smart and inherit its
underlying config so that we inherit properties such as its bind mounts.
Reviewed By: wez
Differential Revision: D6322002
fbshipit-source-id: 3f5ba135b12ad7dcecef6676d27495cfbf0ce97b
Summary:
Previously, a user had to define a config for a repo in a file like `~/.edenrc`
in order to create a new Eden mount via `eden clone`. In practice, the
information that is hardcoded in the config can generally be inferred from an
existing repo, so this expands `eden clone` to support both modes of operation.
Note this made it possible to finally unify the `RepoConfig` and `ClientConfig`
types. This revision removes `RepoConfig`, so I dutifully renamed every
local variable named `repo_config` to `client_config`.
Reviewed By: wez
Differential Revision: D6314115
fbshipit-source-id: 9625a5fbe35b30f76b6099180580c64435a4cf72
Summary:
As explained by the comment in the file, the alias for the config that was used
to create the client should not be necessary to mount it. This builds on top of
D6310325 to rewrite some internal functions that still relied on this alias
being present.
Reviewed By: wez
Differential Revision: D6313778
fbshipit-source-id: 867300e9ec844731993376b8957935f175a242f5
Summary:
The headline changes of this revision are:
- Changes the format of the config file from INI to TOML
(the `edenrc` file under `~/local/.eden` has been replaced
with `config.toml`). This revision includes logic for automatically
performing the migration when Eden is restarted.
- Inlines data from `/etc/eden/config.d` into the TOML file.
Historically, the `edenrc` file for a client would contain the
name of the "configuration alias" defined in a config file like
`~/.edenrc` or `/etc/eden/config.d/00-defaults`. When Eden
loaded a client, it would have to first read the `edenrc` and
then reconstitute the rest of the client configuration by
looking up the alias in the set of config files that were used to
create the client in the first place.
This changes things so that all of the data that was being
cross-referenced is now inlined in the client's config file.
This makes loading a config considerably simpler at the cost
of no longer being able to change the config for multiple clients
that were cloned from the same configuration alias in one place.
It was questionable whether being able to modify a client from
a foreign config after it was created was a safe thing to do, anyway.
Eliminating the need for a historic link to the configuration alias
will make it easier to support running `eden clone` on an arbitrary
local Hg or Git repo. So long as `eden clone` can extract enough
information from the local repo to create an appropriate config file
for the new Eden client, there is no need for a configuration alias
to exist a priori.
Since we were already changing the data in the config file, this
seemed like an appropriate time to make the switch from INI to
TOML, as this was something we wanted to do, anyway.
In testing, I discovered a discrepancy between how boost's
`boost::property_tree::ptree` and Python's `ConfigParser` handled
the following section heading:
```
[repository ZtmpZsillyZeden-clone.LIkh32]
```
Apparently `hasSection("repository ZtmpZsillyZeden-clone.LIkh32")`
in boost would fail to find this section. Because
[[https://stackoverflow.com/questions/13109506/are-hyphens-allowed-in-section-definitions-in-ini-files | there is no spec for INI]],
it is not that surprising that boost and `ConfigParser` do not 100% agree
on what they accept. Moving to TOML means we have a configuration
language with the following desirable properties:
- It has a formal spec, unlike INI. This is important because there are parsers
in a wide range of programming languages that, in theory, accept a consistent
input language.
- It is reasonable for humans to write, as it supports comments, unlike JSON.
- It supports nested structures, like maps and arrays, without going crazy
on the input language it supports, unlike YAML.
Eden now depends on the following third-party TOML parsers:
* C++ https://github.com/skystrife/cpptoml
* Python https://github.com/uiri/toml
This revision also changes the organization of `~/local/.eden` slightly. For now,
there is still a `config.json` file, but the values are no longer hashes of the realpath
of the mount. Instead, we take the basename of the realpath and use that as the
name of the directory under `~/local/.eden/clients`. If there is a naming collision, we
add the first available integral suffix. Using the basename makes it easier to
navigate the `~/local/.eden/clients` directory.
Although the `edenrc` file under `~/local/.eden/clients` has been switched from INI
to TOML, the other Eden config files (`~/.edenrc` and `/etc/eden/config.d/*`) still use
INI. Migrating those to TOML will be done in a future revision.
Note this revision allowed us to eliminate `facebook::eden::InterpolatedPropertyTree`
as well as a number of uses of boost due to the elimination of
`ClientConfig::loadConfigData()` in the C++ code. Because `ClientConfig`
no longer does interpolation, a bit of `ClientConfigTest` was deleted as part of
this revision because it is no longer relevant.
Reviewed By: wez
Differential Revision: D6310325
fbshipit-source-id: 2548149c064cdf8e78a3b3ce6fe667ff70f94f84
Summary: More work towards encapsulating a FileInode's internal state machine.
Reviewed By: wez
Differential Revision: D6316013
fbshipit-source-id: 9c8303b35a0de1ba69207c7f59be88c5fb037ad8