Summary:
Some of the heavier watchman->eden queries in www today
trigger calls to `getFileInformation` with very large numbers
of paths.
For a set of paths with a lot of common prefixes there is a lot
of repeated work to load same inodes over and over again.
eg: `a/b/c/d/A` and `a/b/c/d/B` both have to load `a -> b -> c -> d` before
they can load the leaf node.
This diff pre-processes the list of paths and builds a tree-shaped plan
so that we won't need to load any inode referenced by any of the paths
more than once.
The `getSHA1` method could benefit from a similar change; I'll do that in
a follow-up diff.
Reviewed By: strager
Differential Revision: D8578261
fbshipit-source-id: e899640a2ef6dcdb6b903d2a2735e9240496d74b
Summary:
The `eden debug overlay` code has some logic in python for parsing overlay
files. This moves that logic into a new module so I can re-use it for an
`fsck` command in a subsequent diff.
Reviewed By: wez
Differential Revision: D8813715
fbshipit-source-id: 6c32561db95f5da112cfd3e06b9e4653f431b94a
Summary:
This class name looks like it was copy-and-pasted incorrectly, probably from
the `eden debug set_log_level` code. Update the name to specify "Journal"
rather than "Level"
Reviewed By: strager
Differential Revision: D8813713
fbshipit-source-id: 2a5edd7f2de8279ee70dcc72b200c2b86b60f376
Summary:
Update `eden restart` to explain that the restart will be disruptive when
performing a full restart, and prompt the user for confirmation. This prompt
can be overridden with the `--force` option. Also print more messaging after
the restart completes telling users what to do if they see "Transport endpoint
not connected" errors from programs still using the old mount points.
This also updates `eden restart` to more carefully handle cases where there is
an existing edenfs daemon running but it is not fully healthy.
Reviewed By: wez
Differential Revision: D8731004
fbshipit-source-id: 05762187b47057b2930d0a6b71b0a6fdbd4aa7e5
Summary:
Fix two debug subcommands that were calling `os.getcwd()` while setting up the
argument parser. This caused the eden CLI to throw an exception of the
current working directory was invalid (for instance, if it was inside a stale
Eden mount point).
Reviewed By: strager
Differential Revision: D8731006
fbshipit-source-id: 8b7a67815d61b3476109aad646d21af8282ac68c
Summary:
EdenConfig uses stat to determine if config files have change. However, it
should use st_mtim.tv_nsec rather than st_mtime because this provides
nano-second resolution.
Reviewed By: simpkins
Differential Revision: D8797143
fbshipit-source-id: aadf114f509116d497a4c237ecaf108eb3824d5f
Summary:
This commit focuses on the following functionality:
- Eden configuration should have reasonable default values.
- Eden should allow configuration to be specified in various manners: default, cli, user and system configuration files.
- Configuration values should have the following precedence (hi to low): cli,
user config, system config, default.
- A light-weight mechanism is required to detect and update configuration changes.
Reviewed By: simpkins
Differential Revision: D8594297
fbshipit-source-id: ed195212669b18b450a1eae359e4d23905beadb4
Summary:
This refactors `HgBackingStore::getTreeForCommit()` so that it no longer invokes
`importThreadPool_.get()` if the `LocalStore` contains the `Tree` for the
specified `commitID` (which is commonly the case). When the `LocalStore` has a
cache hit, there is no need to import any data from Mercurial, so there is no
need to involve the `importThreadPool_`.
As explained in T30958650, `importThreadPool_.get()` can be relatively expensive
because it can have the side-effect of invoking
`HgImporterThreadFactory::newThread()`, which creates a new `HgImporter`, which
in turn does some potentially expensive work (it spawns a new subprocess for the
import helper and does some treemanifest-related I/O, if appropriate).
Because `HgBackingStore::getTreeForCommit()` is called as part of `hg status`,
the existing implementation was preventing us from reliably providing <30ms
`hg status` calls because we would frequently get "unlucky" and unnecessarily
create a new `HgImporter`.
Reviewed By: wez
Differential Revision: D8764805
fbshipit-source-id: 0acf7779560ddf182e7eb50daea54c5ee57030d0
Summary:
This is part of "the great r-valuification of folly::Future":
* This is something we should do for safety in general.
* Context: `Future::get(...)` means both `Future::get()` and `Future::get(Duration)`
* Using lvalue-qualified `Future::get(...)` has caused some failures around D7840699 since lvalue-qualification hides that operation's move-out semantics - leads to some use of future operations that are really not correct, but are not obviously incorrect.
* Problems with `Future::get(...) &`: it moves-out the result but doesn't invalidate the Future - the Future remains (technically) valid even though it actually is partially moved-out. Callers can subsequently access that moved-out result via things like `future.get(...)`, `future.result()`, `future.value()`, etc. - these access an already-moved-out result which is/can be surprising.
* Reasons `Future::get(...) &&` is better: its semantics are more obvious and user-testable. It moves-out the Future, leaving it with `future.valid() == false`.
Reviewed By: LeeHowes
Differential Revision: D8756764
fbshipit-source-id: 5c75c79cebcec77658a3a4605087716e969a376d
Summary:
This is part of "the great r-valuification of folly::Future":
* This is something we should do for safety in general.
* Context: `Future::get(...)` means both `Future::get()` and `Future::get(Duration)`
* Using lvalue-qualified `Future::get(...)` has caused some failures around D7840699 since lvalue-qualification hides that operation's move-out semantics - leads to some use of future operations that are really not correct, but are not obviously incorrect.
* Problems with `Future::get(...) &`: it moves-out the result but doesn't invalidate the Future - the Future remains (technically) valid even though it actually is partially moved-out. Callers can subsequently access that moved-out result via things like `future.get(...)`, `future.result()`, `future.value()`, etc. - these access an already-moved-out result which is/can be surprising.
* Reasons `Future::get(...) &&` is better: its semantics are more obvious and user-testable. It moves-out the Future, leaving it with `future.valid() == false`.
Reviewed By: yfeldblum
Differential Revision: D8711368
fbshipit-source-id: fbfcb731097cdf9d8d98583956bc7fe614157a6b
Summary:
This moves the fsattr helper program into the helpers/ directory
created in D8373672.
Reviewed By: chadaustin
Differential Revision: D8705391
fbshipit-source-id: 43e04d8b06b5d17caefafdfc750e3e60cad1b0cc
Summary:
An upcoming change to mercurial will make trees only serialize to disk at the
end of the command. Since Eden is bypassing commands and calling prefetch
directly, it needs to also manually tell Mercurial to serialize.
Reviewed By: chadaustin
Differential Revision: D8679211
fbshipit-source-id: 865ecaf340fa6003c48a6563f713479f07a29f16
Summary:
This is part of "the great r-valuification of folly::Future":
* https://fb.facebook.com/groups/fbcode/permalink/1726007767436055/
* This is something we should do for safety in general.
* Using lvalue-qualified `Future::get()` has caused some failures around D7840699 since lvalue-qualification hides that operation's move-out semantics - leads to some use of future operations that are really not correct, but are not obviously incorrect.
* Drill-down: `Future::get() &` moves-out the result but doesn't invalidate the Future - the Future remains (technically) valid even though it actually is partially moved-out. Callers can subsequently access that moved-out result via things like `future.get()`, `future.result()`, etc., which will cause surprising results - accessing an already-moved-out result. The semantics of `Future::get() &&` are more obvious: it moves-out the Future, leaving it with `future.valid() == false`.
Codemod steps (where `get(...)` means both `Future::get()` and `Future::get(Duration)`):
* expr.get(...) ==> std::move(expr).get(...) // if expr is not already an xvalue
* expr->get(...) ==> std::move(*expr).get(...)
Note: operator precedence of that last step is safe - no need to parenthesize `expr`. Reason: `->` binds more tightly than unary `*`.
Reviewed By: yfeldblum
Differential Revision: D8625157
fbshipit-source-id: 70323ff3189631ce5597d3c9b17629c99f5d457b
Summary:
Update "eden clone" to create a README file inside the mount point directory
before the checkout is mounted. When Eden is running normally this file will
not be visible because the Eden checkout has been mounted on top of this
directory. However if the checkout is not currently mounted users will see
this file instead.
This should hopefully make it easier for users to figure out how to remount
their checkout on their own, and should prevent them from worrying that their
files have been deleted when the mount point simply is not mounted.
Reviewed By: chadaustin, wez
Differential Revision: D8631281
fbshipit-source-id: 7610450983687a839bdbc3558ff35e0422edc3be
Summary:
The eden integration tests currently run hg through the telemetry wrapper.
This wrapper script adds profiling arguments to a percentage of the commands
it runs. The extra arguments that it adds are not entirely transparent to the
user, and are unfortunately reported in the output of "hg journal".
This was causing occasional test failures in the Eden integration tests that
check the output of "hg journal".
This diff changes the Eden code that checks the journal output to try to strip
off these flags before checking the command output.
Reviewed By: wez
Differential Revision: D8631180
fbshipit-source-id: 004d3bcce291ffc2208f85a29cb34a921f16fdde
Summary:
Update the python CLI code to let the C++ edenfs code handle daemonization
rather than daemonizing in the python code.
This means that we now daemonize after the `sudo` invocation rather than
before it. This allow's sudo's wrapper process to exit once edenfs starts,
rather than remaining around for the lifetime of the edenfs daemon. This
means that the original process' stdout and stderr file descriptors are
closed properly rather than being held open by the sudo wrapper.
This also removes the logic in the CLI that waits for edenfs startup and can
potentially time out and give up.
Reviewed By: strager
Differential Revision: D8508489
fbshipit-source-id: 6a38439d5596049d6f40cb6589b6865c229f736e
Summary:
Update the C++ edenfs code to ensure that the .eden and
.eden/storage/rocks-db directories exist, rather than requiring the python CLI
code create these directories as part of `eden start`
Reviewed By: strager
Differential Revision: D8508488
fbshipit-source-id: 358521b4f5eed1d19bf37903900ca50718e2c35c
Summary:
Update edenfs to print version information in its startup messages.
More detailed version information will be printed to the log file, but it
seems nice to also include this in user-facing messages that will be printed
to the user's terminal in most situations.
Reviewed By: chadaustin
Differential Revision: D8508486
fbshipit-source-id: 9364290ed470375120acd74cfaef1ccde41fd746
Summary:
This updates edenfs to be able to daemonize itself, and moves the
daemonization logic from the python CLI code into C++.
The main benefit of this is that we can now do a better job of reporting
messages to the user during start-up. We can log around potentially slow
operations (opening the RocksDB local store), and we can print messages
directly to the user if startup fails. Previously most failure messages would
go only to the eden log and would not be printed to the user's terminal.
This also fixes some issues where stdin and stdout were not closed properly
when daemonization was performed by the CLI. sudo needed access to these file
descriptors in case it needed to prompt for a password, and it would then hold
the descriptors open until edenfs exited.
Reviewed By: wez
Differential Revision: D8373672
fbshipit-source-id: 3272bff2208596f41d26e479c82c700d6c1efe11
Summary:
This updates the `eden doctor` code to define a `Problem` class, and to update
all of the checks to generate `Problem` objects. Problems that can be
automatically fixed derive from the `FixableProblem` subclass and implement a
`perform_fix()` method.
This provides a more consistent API to report which problems were found and
which ones can be automatically fixed. This also simplifies handling of the
`--dry-run` option, as each check no longer needs to individually check if
they are running in dry-run mode or not.
I also removed the `Check` API, since it only had a single `do_check()`
method. The code now simply uses a few separate helper functions that perform
various checks and generate `Problem` objects.
Reviewed By: wez
Differential Revision: D8617702
fbshipit-source-id: f7da2b7c17799b2844ae9416e827fd59b99d93c5
Summary:
Add a helper function for defining mount points during the test, rather than
requiring each test function to manually set up all of the individual
configurations for each mount point.
Reviewed By: wez
Differential Revision: D8581496
fbshipit-source-id: 41c8f1099292f7ebdbad30278ed530c01d083543
Summary:
Add a helper function that uses `self.addCleanup()` to destroy the temporary
directory rather than having to use `try...finally` blocks.
Reviewed By: strager
Differential Revision: D8581494
fbshipit-source-id: 3e1368c876b305c1528c5ed1de948a65a3da8199
Summary:
This updates the CLI code to put a timeout on `unmount()` thrift calls to
edenfs. We've had a few reports of situations where the unmount thrift call
has hung while edenfs was waiting for all inodes to be released.
Eventually we should fix the C++ code to put a hard timeout on the operation,
(and forcibly remove the mount point from its list in EdenServer even if it is
still waiting for some inodes to be released). However as an interim solution
timing out on the CLI is better than hanging forever. In particular the old
behavior made `eden rm` unable to clean up some mount points.
Reviewed By: pkaush
Differential Revision: D8558767
fbshipit-source-id: 942e5c7bcc147229f040f377ce59fcc2565276e0
Summary:
Previously `eden rm` tried to look at `<path>/.eden/root` to find the root of
the mount point. This failed if the checkout was not currently mounted.
This updates the code to try looking for the exact input path in the config
file in this case. It also avoids sending an `unmount()` thrift call to
edenfs in this case as well.
Reviewed By: strager
Differential Revision: D8558765
fbshipit-source-id: a5d9db7eee702817713b089a457f25710438661e
Summary:
Update `eden doctor` to automatically remount any checkouts that have been
unmounted.
In general we don't expect users to need to unmount checkouts. If they have
gotten into this state it is most likely unexpected, so have `eden doctor` fix
it for them. This is also consistent with how `edenfs` automatically remounts
all configured mount points on startup.
Reviewed By: chadaustin
Differential Revision: D8558766
fbshipit-source-id: 2e22cccead69d01530c6f646a958fc435ccce6c4
Summary:
When running the integration tests in parallel in a dev mode build I
occasionally see timeouts in the shutdown code in some tests. (Usually this
seems to happen in some of the remount tests that create multiple mount points
and shutdown and restart edenfs multiple times.)
Usually when I see this error edenfs does exit successfully on its own just
after this 15 second timeout has expired.
This bumps up the timeout to 30 seconds to avoid spurious failures.
Reviewed By: pkaush
Differential Revision: D8559052
fbshipit-source-id: 567062b08b34eced8f05538607ae9f4773c4aa9c
Summary:
We provide average values for vmRSS and memory (dirty bytes) on linux systems
by parsing the proc file system. This change allows the current values
(non-average) to be queried and calculated from the thrift API. The cli has
been updated to present these values (and no longer consumes the parsed smaps
files from EdenServer).
Reviewed By: chadaustin
Differential Revision: D8549305
fbshipit-source-id: 77f6838f39784e7ebeda11d8c66dba1fa9f10591
Summary:
This adds a small helper class for printing startup status messages to the
user, and for signaling startup status over a pipe.
This will make it possible in a future diff for edenfs to daemonize, but still
communicate messages back to the parent process to report what it is doing
until it finishes the start-up steps.
Reviewed By: strager
Differential Revision: D8372250
fbshipit-source-id: 53a897944beeb1582a090a2b69afbc2b41408d52
Summary:
Previously `eden list` reported normally running active checkouts with an
`(active)` suffix, and non-mounted checkouts with no suffix.
The non-mounted situation is rare. This diff changes the command to report
non-mounted checkouts with a `(not mounted)` suffix. I removed the `(active)`
suffix from normally-running checkouts now, but let me know if people think it
is still valuable to keep this annotation in the output.
Reviewed By: chadaustin
Differential Revision: D8558770
fbshipit-source-id: ccce0f7c2031e50f2847089e49007274c31a69a7
Summary:
Update `eden unmount --destroy` to print a message recommending `eden rm`
instead. It still functions as before, but just prints this additional
message.
Also suppress the `--destroy` option from the `eden unmount --help` output.
Reviewed By: chadaustin
Differential Revision: D8558769
fbshipit-source-id: 78215a61fafeae2f7c5cfe4bc55ab0ea6682d2bc
Summary:
The rage command creates a paste by default, so tweak the wording to state
that it gathers information rather than printing it.
Reviewed By: chadaustin
Differential Revision: D8558771
fbshipit-source-id: 819a3c56a9ead173f6aadf639ac954e800c4a046
Summary:
The `debug` command was previously hidden in the `eden --help` output.
There doesn't really seem to be a strong reason to hide this subcommand. It
seems useful to expose it to users who do want to poke around at their current
edenfs instance.
Reviewed By: chadaustin
Differential Revision: D8558768
fbshipit-source-id: f23fd29b473607d83f4f0aec064e72a06dd3ae45
Summary:
Remove python code to parse /proc/self/smaps and associated tests. This is now
done in the EdenServer c++ code.
Reviewed By: simpkins
Differential Revision: D8550273
fbshipit-source-id: 6a2a5cbd25beff412376b0ada683f9dc30e15aff
Summary:
If someone's having problems with Eden and they run `eden doctor` it
sometimes says 'All is well.' even though the person is still having
issues. 'No issues detected.' is a slightly humbler (and accurate)
statement.
Reviewed By: eamonnkent
Differential Revision: D8553069
fbshipit-source-id: 5dd02dc304c6aa6f75b2b2e11757ceb333f8f506
Summary:
The OverlayGoldMasterTest.can_load_overlay_v2 unit test is failing stress test
runs because it keeps overlay data checked into the source repository and runs
the tests directly against this directory. This causes the tests to fail if
multiple test are run in parallel since they are all trying to use this same
directory simultaneously.
We need to make a copy of this directory rather than having the tests run
directly against the source tree to avoid locking issues. The overlay code
can potentially also try to upgrade the overlay format when it opens the
directory. This seems like another reason why it should not run directly
against the directory in the source repository.
This changes the test to make a copy of the overlay directory in a temporary
location, and run the tests against that directory. As part of this change I
also bundled the original input into a tar file.
Reviewed By: chadaustin
Differential Revision: D8555716
fbshipit-source-id: bf24bd96a0a31c097d9cf8e0fbe8b0bfaf009943
Summary:
Eden doctor checks to that configured bind mounts are present for each
Eden repository. It reports errors for missing directories and the device type is incorrect. The device type must differ from the device type of the repository's Eden path.
Eden doctor can fix (by creating) missing bind point directories and then calling the eden thrift server to mount.
Reviewed By: chadaustin
Differential Revision: D8244287
fbshipit-source-id: c78e5ecce63002761a266c5925f2d6618e648e4a
Summary:
We need to call sqlite3_close() if sqlite3_open() fails to avoid leaking the
DB object that was allocated.
Reviewed By: chadaustin, strager
Differential Revision: D8529745
fbshipit-source-id: 1087dac8343b8b3120c89bd3c9a250970e69df8e
Summary:
I saw RemountTest.test_restart_twice() fail once when running locally, but I
haven't been able to reproduce the failure yet. This updates the checks in
this code to hopefully provide better information about what went wrong in
case we see this failure again in the future. This now checks the `eden list`
output first so if something goes wrong we will hopefully have more
information about which checkouts were or weren't remounted.
Reviewed By: strager
Differential Revision: D8528544
fbshipit-source-id: 4f50145aa97956f45f50813070fb73836bc7acde
Summary:
We calculate vmRSS bytes on linux systems by parsing the /proc/self/status
file. The value in that file is in kB so we need to convert it to bytes.
Reviewed By: chadaustin
Differential Revision: D8549009
fbshipit-source-id: 88b18543cb561372dc5eee84293e79fddca8efcb
Summary:
Expand the data we collect in fb 303 collector. Currently we extract data from
procprint it is only reliable for a day (or so).
Here we add:
- memory vm rss bytes (from /proc/self/status)
- memory private bytes (from /proc/self/smaps)
Reviewed By: chadaustin
Differential Revision: D8380917
fbshipit-source-id: dca6fac7af44321c7a6615edb0fde0cb7c8827d0
Summary:
Update the hg_import_helper.py code to ignore CMD_PREFETCH_FILES requests with
more than 4M files.
Old edenfs daemons that send JSON data for this field can sometimes end up
starting newer versions of the hg_import_helper.py script. If they send JSON
data here we want to make sure the `hg_import_helper.py` script does not
actually try to parse the request, which can end up consuming a lot of CPU and
memory if it tries to deserialize data that isn't actually what it expects.
The older JSON data format should always start with '[', which means it will
appear to have at least 1,526,726,656 files.
Reviewed By: chadaustin
Differential Revision: D8508651
fbshipit-source-id: c1e7726398517f97ccc1deafd30620306a9ad80d
Summary:
We've seen what appears to be phantom calls to shutdown() so we'd like
to add some degree of auditing. This diff adds a new method with some
context; this will allow us to distinguish between `eden stop`, `eden restart`
and eden server internal calls to the `shutdown` method. It may still
be possible that something else is calling our shutdown method, but it
seems unlikely as we're only accessible to our own code via a unix domain
socket.
Reviewed By: chadaustin
Differential Revision: D8341595
fbshipit-source-id: 50d58ea0b56e5f42cd37c404048d710bde0d13a3
Summary:
Update edenfs to fork the privhelper process as the first thing we do, before
calling folly::init(). This allows us to drop privileges before processing
command line arguments and doing any other startup work.
Reviewed By: wez
Differential Revision: D8212778
fbshipit-source-id: d67e3700305fdb01cb6188645b37875ceb53d21f
Summary:
Add a call to tell the privhelper process to redirect its log messages to a
different file handle.
This call will make it possible for the privhelper processed to be forked
before the main edenfs process has processed the --logPath argument. The main
edenfs process can now drop privileges before it processes arguments and opens
the log file, and it can then pass the file handle to the privhelper process.
Reviewed By: wez
Differential Revision: D8212776
fbshipit-source-id: 3ec6bfc82ee090216d66c5bfd1b8c2b8819c1f45
Summary:
Add a PrivHelper::detachEventBase() method, and rename PrivHelper::start() to
attachEventBase().
This makes it possible to detach a running PrivHelper from its EventBase and
re-attach it to an EventBase later to restart it. This will be useful in
upcoming diffs to allow performing calls to the PrivHelper before the main
thrift server EventBase has started.
Reviewed By: wez
Differential Revision: D8212777
fbshipit-source-id: d5a9bf672afa8b16e53201ac747d77337e1cc307
Summary:
I've noticed that we don't shut down aux processes when
we remove a mount point.
This can cause `eden rm` to hang if the buck daemon is still running in the background.
Reviewed By: strager
Differential Revision: D8499623
fbshipit-source-id: 9a7f515820a3ffaec86c8c81110283bc0eec3750
Summary:
D8065370 changed the HgImporter code to make a blocking `get()` call on a
future from the hg importer thread pool. This caused deadlocks, since all of
the hg importer threads could become stuck waiting on these `get()` calls to
complete. These would be waiting on RocksDbLocalStore threads which were in
turn all busy waiting to schedule operations on the HgImporter threads.
This fixes the code to use `Future::then()` rather than `Future::get()` to
avoid blocking the HgImporter threads on these operations.
Reviewed By: wez
Differential Revision: D8438777
fbshipit-source-id: a0d647b10ef5a182be2d19f636c2dbc24eab1b23
Summary:
Move the HgProxyHash code out of HgImporter.cpp and into its own top-level
file. This does not have any functional changes other than moving the code
around.
This will make it easier to perform HgProxyHash operation from inside
HgBackingStore.cpp. Currently all of the HgProxyHash operations are done from
inside HgImporter threads. This operation does not need to be done in the
importer threads however, and for operations like putBatch() that complete
asynchronously it generally should not be done from the HgImporter threads.
Reviewed By: wez
Differential Revision: D8438776
fbshipit-source-id: 344652f47e5ccdc6ef20b143dc52d0eeac2886e6
Summary:
There was a conflict between the DirContents refactoring and some of
the Overlay diffs.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D8427680
fbshipit-source-id: a01a2a8456f7a35249d8b65f2b5fcd55825438ee
