Summary:
Only treat edenfs mount points as stale if attempting to access them fails with
ENOTCONN. If we can successfully access the mount point then it cannot be
stale and does not need to be unmounted.
This prevents `eden doctor` from incorrectly unmounting mount points owned by
another edenfs process running on this system.
Reviewed By: bolinfest
Differential Revision: D7683503
fbshipit-source-id: 412cc17442fac1e8030dd40de0e0f642c15d1253
Summary:
This command directly invoked Eden's `checkOutRevision()` thrift call. This
bypasses mercurial, so mercurial would not know about the working directory
parent change.
If we need to we could add this back as a debug command in the future.
However, this isn't something that normal users should ever invoke directly.
Reviewed By: chadaustin
Differential Revision: D7754529
fbshipit-source-id: 8d6aad70da37e56b3b8ced1ade4c8537141ab239
Summary:
Update the Eden CLI command names to more closely match the command names used
by other Unix tools like systemctl and runit.
* `eden daemon` --> `eden start`
* `eden shutdown` --> `eden stop`
* `eden health` --> `eden status`
I have added aliases for the old command names so that they still work.
Reviewed By: chadaustin
Differential Revision: D7753780
fbshipit-source-id: af2c95dd5bcad09c395496f1cd5def082bccdc5c
Summary: Also print the inode path if the assertions in EXPECT_FILE_INODE fail.
Reviewed By: simpkins
Differential Revision: D7035517
fbshipit-source-id: 6c50acb588d1c985c7e7a1586c06f04a657698f9
Summary:
Now that Eden writes tree data into the overlay even when not
materialized, checkouts can be slow as Eden actively tries to forget
allocated inode data for potentially large trees when
kPreciseInodeMemory is false. This pushes the bulk of that garbage
collection work onto a background thread, making checkouts fast again.
Reviewed By: simpkins
Differential Revision: D7683120
fbshipit-source-id: 2a22e4612a4438b3ed2527670343c49dfcd902bc
Summary: This gets rid of those pesky #ifdefs in FileInode.cpp and TreeInode.cpp.
Reviewed By: wez
Differential Revision: D7735914
fbshipit-source-id: d810461984e21f72670f43ca2d1b4f5aacbf376e
Summary:
Try treating the argument as a repository path first, rather than a well-known
repository name. If we can find a mercurial or git repository at this path,
look for a .arcconfig file in the repository, and try to identify the
repository project type from this file. Use this project name to load our
default bind-mount and hooks configuration for the repository.
If the argument is not a valid repository path, fall back to treating it as a
well-known project name in our repository configs.
This also updates `eden clone` to print a few more diagnostic messages as it
works. (For instance, printing when it is starting the edenfs daemon, and
reporting the commit ID that was checked out.)
Reviewed By: wez
Differential Revision: D7739917
fbshipit-source-id: ac927b9e93039e4d1b8afa80466c2eee3a8829e9
Summary:
Update the Eden mercurial extension to read the `.eden/root` symlink to
determine what Eden thinks the mount path is. This might be different from
what directory mercurial thinks it is in if a parent directory of the Eden
mount has been bind-mounted to an alternate location.
Maybe in the future we should update thrift clients to pass in the client ID
(currently readable via `.eden/client`) rather than the mount path. That would
make it less likely for clients to accidentally forget to read `.eden/root` and
pass in the wrong mount path.
Reviewed By: wez
Differential Revision: D7705655
fbshipit-source-id: 7bd1e8013b99a52ff06dd45f63d6669b66bdf577
Summary:
Define an `add_mount()` method to simplify the process of defining mounts in
the FakeMountTable. Also simplify the `lstat()` code to only try lookups using
strings. The keys in the dictionary are always strings, so there is no need to
ever try looking up entries using bytes.
Reviewed By: chadaustin
Differential Revision: D7704335
fbshipit-source-id: 9c81b43c0a196287254bcd749d096f9b148af8ac
Summary:
This updates the code to store ServerState using a shared_ptr rather than
having it be an inlined member variable of EdenServer.
Previously EdenMount objects contained a raw pointer to the ServerState, with
the reasoning that the EdenServer object should outlive the EdenMount objects.
It turns out that this is not quite true in practice--EdenMount::destroy() will
normally be called before the EdenServer is destroyed, but this may not
actually destroy the EdenMount object immediately.
This fixes a race condition in the FuseTest.initMount() test that could cause
this test to occasionally fail when run on a heavily loaded system.
Reviewed By: chadaustin
Differential Revision: D7720509
fbshipit-source-id: 056ff5985587c8d8c32c11d17ba637ebd7598677
Summary:
I was worried we might leak data in the overlay in this particular
sequence of operations, so here's a test that shows we don't.
Reviewed By: simpkins
Differential Revision: D7663419
fbshipit-source-id: b4b0d08f863fe7b23501e9e4943d18ba0b746e42
Summary:
When we start storing permission changes and other inode
metadata keyed on inode number, we will need to remember inode numbers
before they're accessed to handle the case where the Eden process dies
before the inode is unloaded. Otherwise, data could be left in an
inconsistent state and some writes could be missed.
This change depends on saving (and forgetting) directories being
cheap.
Reviewed By: simpkins
Differential Revision: D7598094
fbshipit-source-id: 9ab127b30a9c09ab62aa08b50c13b3eaa40be60d
Summary: I forgot to remove this comment when I fixed the test in D7565503.
Reviewed By: chadaustin
Differential Revision: D7716106
fbshipit-source-id: 58acd71ac96560f2cc454b3a7da75bb61e6e71c7
Summary:
Add a pretty basic `restart` command to the Eden CLI.
This is relatively simple at the moment. It has `--graceful` and `--full`
options, and defaults to a full restart at the moment if neither of these
arguments is specified. We probably could add most of the arguments from the
`daemon` command to `restart`, but I'm not sure if we really want to support
that complexity in this command.
In the future we can make this command smarter and have it do a better job of
reporting the status as Eden restarts.
Reviewed By: chadaustin
Differential Revision: D7682838
fbshipit-source-id: 80c8d6604c3ca120fb71d6b8a388309218b5500c
Summary:
Refactor the Eden CLI command so that all subcommands are implemented as
subclasses. This helps keep the command line argument definitions together
with the logic for the command.
This is primarily just a code refactoring change, but I did include a few minor
behavioral changes to the help output:
- The command list is now always sorted alphabetically in the help output.
- The "help" subcommand can now show help for more than just one subcommand
deep. (e.g., `eden help stats io` now works correctly)
- I made some minor improvements to a few of the help strings.
Reviewed By: chadaustin
Differential Revision: D7673021
fbshipit-source-id: dc4c6db20a0fe7452d38bdafc6273e234dba8e4e
Summary:
Rename the rule for the Eden CLI script from `cli` to `eden`. This way the
generated par file will be named `eden.par` rather than `cli.par`. This seems
like it will be slightly less confusing to new developers starting to work on
Eden--currently it doesn't seem particularly intuitive that `cli.par` gets
installed as `eden`.
Reviewed By: bolinfest
Differential Revision: D7658023
fbshipit-source-id: 7afc5e0b703d02751b509efe42f8b8be1f56bbc1
Summary:
Temporary fix to bump the takeover timeout to five minutes in the cli
code. I messed up and forgot to ship these changes with D7660642.
Reviewed By: simpkins
Differential Revision: D7676402
fbshipit-source-id: a22963fbf89663e44537e7946b3be5eb870f930c
Summary:
In practice, we have received a number of user reports where Nuclide had lost
track of its Watchman subscriptions and `eden doctor` failed to report it.
When the `NuclideHasExpectedWatchmanSubscriptions` check was first added to
`eden doctor`, in the repro case we had, it seemed that Nuclide would lose its
`filewatcher-<root-folder>` subscription, but would retain its
`hg-repository-watchman-subscription-primary` subscription. As such, we encoded
a heuristic where the presence of the `hg-repository-watchman-subscription-primary`
subscription without the accompanying `filewatcher-<root-folder>` subscription
was taken as a sign that (1) the user was connected to the directory via
Nuclide and (2) Nuclide's Watchman subscriptions had gotten corrupted.
Unfortunately, this was not a sufficient heuristic because it would not detect
the error case where Nuclide had lost //all// of its Watchman subscriptions!
Because `eden doctor` did not flag this, users kept using Nuclide even though
things were in a bad state, sometimes losing data as a result.
To fix that, we introduced a new tool, `nuclide-connections`, in D7598900. This
executable prints the list of local folders used as root folders in Nuclide
clients connected to the local Nuclide server. Now we no longer have to rely on
the presence of the `hg-repository-watchman-subscription-primary` Watchman
subscription as a proxy for a Nuclide connection.
This commit updates the `NuclideHasExpectedWatchmanSubscriptions` check in
`eden doctor` to use `nuclide-connections`. For every Nuclide connection that is
also an Eden mount, it verifies that the expected Watchman subscriptions exist.
As noted in the TODO in `NuclideHasExpectedWatchmanSubscriptions`, this
check is a bit brittle, but it is intended as a stopgap until we find the root cause
of the missing Watchman subscriptions and fix it.
Reviewed By: simpkins
Differential Revision: D7663214
fbshipit-source-id: 200b78ce25010222e59f3296f2412161086aa7ce
Summary:
D7512338 has been deployed for well over a week, so lets update edenfs to
reject the request if any clients still try to call getScmStatus() without
specifying the hash.
Reviewed By: chadaustin
Differential Revision: D7658022
fbshipit-source-id: ebd49034e076720f156f1ff2c551417ef058167a
Summary:
Split the modifyConflict() test into 3 separate test functions. This test
runs the same checks in 54 separate variations (3 path names, 3 checkout types,
with 6 inode load options).
For each variation we set up a new test mount, creating an overlay and
LocalStore on disk. Simply setting up and tearing down the test mounts can be
expensive if the temporary directory is served from spinning disk. This causes
this test to occasionally time out when run on our continuous build hosts.
Splitting the test up into 3 separate test functions will effectively give it
3x longer to run, since each test function will have a separate timeout. This
will also enable these test functions to run in parallel (although that may not
actually help with performance if they are bottlenecked on disk I/O).
Reviewed By: chadaustin
Differential Revision: D7665409
fbshipit-source-id: 4bddd68e75f38b1b6cc2d57512a5b52855f3bade
Summary:
Update the UnixSocket code to limit the number of iovecs that we send at one
time to folly::kIovMax. (This constant is set from from IOV_MAX on platforms
that provide this setting.)
The thrift serializer code emits data into 16kB chunks, so if we had many
megabytes of data to send we could end up with thousands of iovecs. The kernel
would fail the send with EMSGSIZE in this case.
Reviewed By: chadaustin
Differential Revision: D7665147
fbshipit-source-id: b7a60238d3fca973604b9037f22883cae80891ab
Summary:
These were making some noise in the Nuclide diagnostics pane.
In the course of making these changes, I ended up reformatting
some files, as well. Perhaps we should flag flake8 and autoformat
violations more aggressively in CI now?
Reviewed By: chadaustin
Differential Revision: D7658030
fbshipit-source-id: b52a29b1cc242967f868dcc8ee46ec1bb9bdfbee
Summary:
The takeover data for a fully-loaded monorepo was 20 MB. Two
monorepos was 40 MB, exceeding the previous 32 MiB limit and causing
takeover to fail. Bump to 512 MiB to minimize the chance of
allocating too much given bogus data while still supporting large
numbers of checkouts.
Reviewed By: simpkins
Differential Revision: D7661637
fbshipit-source-id: 1dc54055e7ac3ce0f6cdbf4a7e5d372c2674dfee
Summary: Temporary fix to bump the takeover timeout to five minutes
Reviewed By: simpkins
Differential Revision: D7660642
fbshipit-source-id: 5b3a0347aa1443e395e9461d33459c62ffce5b2d
Summary:
UnixSocket traverses the IOBuf chain twice. Refactor that
into a common function because the next diff caps the size of
individual iovecs.
Reviewed By: simpkins
Differential Revision: D7659062
fbshipit-source-id: 88b7d63669d8189b96434c38a6e499ed3b5ebbe6
Summary:
fdatasync() has a substantial cost and is unnecessary given
our durability guarantees (and the fact that most filesystems on Linux
try to avoid data loss in this common write + rename situation anyway)
Reviewed By: simpkins
Differential Revision: D7641131
fbshipit-source-id: d041e7090dc05a4d4400f86cad9501aa8a6988a9
Summary:
Fix a couple issues when when performing an `hg update` with updatecheck set to
`noconflict` and when conflicts are detected:
- Report the files with conflicts. The conflicts may be non-obvious to the
user if the conflicts are in ignored files that are not tracked in the
current commit but are present in the destination commit.
- Do not invoke the preupdate hook or create the .hg/updatestate file until
after we have performed the conflict check. Otherwise the repository will be
marked as in the middle of an unfinished update even though we never
attempted the update itself.
Reviewed By: quark-zju
Differential Revision: D7581782
fbshipit-source-id: e290ebd6a27f228a805bea40aabb3f3cc633cc7d
Summary:
Update the assert_status() function in the integration tests so that it also
checks for unfinished update/rebase/graft/etc operations. We unfortunately
have to manually check for the presence of these files ourselves, since
`hg status` provides no mechanism to report this data when `HGPLAIN` is set.
Reviewed By: quark-zju
Differential Revision: D7581781
fbshipit-source-id: 230234e5b8ce28cd3569cdacced686fed2a9dd32
Summary:
This rewrites the `walk()` and `status()` methods in the `eden_dirstate` class.
The changes to the `status()` function should not have any major behavior
changes: this primarily moves logic out of `EdenThriftClient.getStatus()` and
into `dirstate.status()`
The changes to the `walk()` function do fix a number of bugs. We now implement
walk by combining Eden's `getScmStatus()` results with the dirstate's
non-normal files and the current parent commit's manifest. All of the glob
matching is done purely in python now. This fixes some cases where Eden's glob
handling behavior did not support some glob patterns allowed by mercurial. The
Eden glob() call also always returned ignored files and hidden files (such as
everything inside the `.hg` directory). I believe this behavior is desired by
watchman, but it caused problems for the `walk()` code.
Reviewed By: chadaustin
Differential Revision: D7565503
fbshipit-source-id: deb91b4772501e8fbdba56e5c099a72d55fb61ee
Summary:
Update `eden_dirstate.status()` to invoke the `match.bad()` and
`match.explicitdir()` appropriately when any of the explicitly specified match
paths either don't exist or refer to directories.
Reviewed By: chadaustin
Differential Revision: D7556850
fbshipit-source-id: 2e8d88ff0e3c09cea2f3ef5b18c9a214231815ab
Summary:
Drop the log level of the "no dir data for inode" message from DBG2 to DBG3 so
we do not have it enabled by default. (Our default log settings are
eden=DBG2.)
Reviewed By: chadaustin
Differential Revision: D7620918
fbshipit-source-id: 25cf30af08054f3ba879bd57f3841b1035e0c75e
Summary:
Implement `dirstate.matches()` in Eden by calling `matches()` on the parent
commit's context, and then updating that result with the list of non-normal
files from Eden's dirstate.
Reviewed By: chadaustin
Differential Revision: D7556847
fbshipit-source-id: 34e78c0c125cd1b184a7773a149edafdca34fee1
Summary:
Add integration tests to check the behavior of the `hg journal` command in an
Eden checkout.
Reviewed By: wez
Differential Revision: D7512339
fbshipit-source-id: 9a83a1752835bc6ded9f0c3b2cb29ec24a6db631
Summary:
When running the integration tests, set HGRCPATH to point to the configs found
at `scm/hg/fb/staticfiles/etc/mercurial`
This does affect mercurial's behavior in some cases, and therefore required
changes to some of our tests:
- `hg diff` output now uses an output style similar to `git diff`
- merge conflict markers include some extra commit metadata
Reviewed By: wez
Differential Revision: D7512321
fbshipit-source-id: 6141605147797ea8b13fdb5a06f67beaeff6afe7
Summary:
This slightly refactors the way that EdenTestCase and EdenRepoTest initialize
repositories. This removes the `create_repo()` method from EdenTestCase and
replaces it with separate `create_hg_repo()` and `create_git_repo()` methods.
The `EdenRepoTest` subclasses now provide alternate `create_repo()`
implementations instead of `get_repo_class()`.
This cleans up the code a bit, since `create_repo()` no longer takes different
arguments based on what type of repository is being created. This also will
make it easier in upcoming diffs to further customize the logic that occurs in
`create_hg_repo()`.
Reviewed By: chadaustin
Differential Revision: D7512320
fbshipit-source-id: d268b0ac0ffb33f3dfd34f2dd1917d57033c81aa
Summary:
Every day I read this document and notice more of the awkward bits. :)
(Note: this ignores all push blocking failures!)
Reviewed By: pkaush
Differential Revision: D7602548
fbshipit-source-id: 71e30de869d97411e8fb86de0701ca7890942ddf
Summary:
I'm about to change the code paths through which directories
are saved to and loaded from persistent storage. To help, consolidate
those operations.
Reviewed By: simpkins
Differential Revision: D7597978
fbshipit-source-id: 9cd20027746b0a372a97b7f5daf07c1b54e6b9ce
Summary:
Add a document that begins to sketch out the various transitions that
our inode data structures make so we can reason about their
correctness
Reviewed By: simpkins
Differential Revision: D7434693
fbshipit-source-id: 5478c108c338ccdadc22e864b077195c8be7d1b7
Summary:
Once a mount point has been unmounted we no longer need to care about
outstanding FUSE reference counts--we can treat them as if they are all zero.
This updates EdenMount to tell the InodeMap when the mount point is unloaded,
and changes InodeMap::unloadInode() to make use of this information when
deciding if it needs to remember the inode information.
Previously InodeMap would save information for inodes with outstanding FUSE
reference counts. Writing all of this state to the overlay could take a
non-trivial amount of time.
Reviewed By: chadaustin
Differential Revision: D7555998
fbshipit-source-id: 0896f867ce850ab3e61c262776d536de003685ff
Summary:
Remove the EDEN_HAS_COMMON_STATS checks now that the common/stats stubs have
the required APIs needed by Eden.
Reviewed By: wez
Differential Revision: D7479593
fbshipit-source-id: cc3db50288bfea7aefd6c91391ab800628b7978f
Summary:
Add a macro to help users define the `getBaseLoggingConfig()` function.
While I would prefer to avoid macros if possible, this seems worthwhile. This
saves 4 or 5 lines of boilerplate code in each program that sets a custom base
logger setting. It also reduces the likelihood of a developer accidentally
having a typo in the function name, which would still build successfully but
not have the desired results.
Reviewed By: chadaustin
Differential Revision: D7457652
fbshipit-source-id: 1c316c7ea6949c16bd7b61c0440cc1ee69ecb83e
Summary:
Make several performance improvements to INSTRUMENT_THRIFT_CALL:
- Avoid heap-allocating the ThriftLogHelper objects.
- Avoid evaluating the log arguments if the log level is not enabled.
Reviewed By: chadaustin
Differential Revision: D7556846
fbshipit-source-id: e111e24e44499c5cf9725ded2b958a7dcb2c3e26
Summary:
Add a helper function to make sure we log commit hash arguments as hexadecimal
rather than binary data. This is similar to `hashFromThrift()`, except that it
always returns a string result and does not throw on invalid input.
Reviewed By: chadaustin
Differential Revision: D7556916
fbshipit-source-id: 87422de3f178700d378f5ddc45172efd38a13799
Summary:
Rename `getRefcount()` to `debugGetFuseRefcount()` to make it clear that this
returns the FUSE reference count rather than the pointer refcount, and that
the result should not be used for anything other than debugging or diagnostic
purposes.
There is already a `getFuseRefcount()` method to check the FUSE refcount while
the inode is being unloaded. The only time it is safe to check the FUSE
refcount for programmatic purposes, and `getFuseRefcount()` has a `DCHECK()` to
help ensure that this is the only time it is called.
(`getFuseRefcount()` also predates `getRefcount()`, which was added for the
`eden debug inode` command in D5277870.)
Reviewed By: chadaustin
Differential Revision: D7555997
fbshipit-source-id: e66f2b3d2b9eb14f8f8878d78978851929198a31
Summary:
Change EdenMount to destroy the Overlay object as soon as the `shutdown()`
operation completes. Previously the Overlay would not get destroyed until
EdenMount::destroy() was called.
During graceful restart we transfer mount information to the new process after
`shutdown()` completes, but potentially before `destroy()` has finished. This
previously resulted in a race condition where the new process could start
opening the overlay before the old process had released the lock. This change
should fix that race condition.
Reviewed By: chadaustin
Differential Revision: D7543295
fbshipit-source-id: 82ae33fe6bb0aa9f0a3b010fdd1d350c63442420
Summary: Each Overlay object should be owned only by its EdenMount.
Reviewed By: chadaustin
Differential Revision: D7543294
fbshipit-source-id: 6db40fea31ce298c61d047cba9165887e32926a1
Summary:
A couple have users have run into a bug where mercurial's `ctx.manifestnode()`
function returns an empty string. I believe treemanifest was been enabled both
times where this was seen. We did see that restarting eden (which restarts the
hg import helper) made the problem go away.
This updates the import helper to throw an exception if `ctx.manifestnode()`
ever returns the empty string. We will invalidate the cache and retry if this
occurs.
Reviewed By: quark-zju
Differential Revision: D7542922
fbshipit-source-id: 42a31835a65217473bf252b10f9cf82dd901f89b
Summary:
Change getScmStatus() so that callers must explicitly specify the commit to
diff against. This should help avoid race conditions around commit or checkout
operations where the parent commit has just changed and eden returns status
information against a commit that wasn't what the client was expecting.
This should still maintain backwards compatibility with older clients that do
not send this parameter yet: we will simply receive the hash as an empty string
in this case, and we still provide the old behavior in this case.
Reviewed By: wez
Differential Revision: D7512338
fbshipit-source-id: 1fb4645dda13b9108c66c2daaa802ea3445ac5f2
