Summary:
When running the integration tests in parallel in a dev mode build I
occasionally see timeouts in the shutdown code in some tests. (Usually this
seems to happen in some of the remount tests that create multiple mount points
and shutdown and restart edenfs multiple times.)
Usually when I see this error edenfs does exit successfully on its own just
after this 15 second timeout has expired.
This bumps up the timeout to 30 seconds to avoid spurious failures.
Reviewed By: pkaush
Differential Revision: D8559052
fbshipit-source-id: 567062b08b34eced8f05538607ae9f4773c4aa9c
Summary:
We provide average values for vmRSS and memory (dirty bytes) on linux systems
by parsing the proc file system. This change allows the current values
(non-average) to be queried and calculated from the thrift API. The cli has
been updated to present these values (and no longer consumes the parsed smaps
files from EdenServer).
Reviewed By: chadaustin
Differential Revision: D8549305
fbshipit-source-id: 77f6838f39784e7ebeda11d8c66dba1fa9f10591
Summary:
This adds a small helper class for printing startup status messages to the
user, and for signaling startup status over a pipe.
This will make it possible in a future diff for edenfs to daemonize, but still
communicate messages back to the parent process to report what it is doing
until it finishes the start-up steps.
Reviewed By: strager
Differential Revision: D8372250
fbshipit-source-id: 53a897944beeb1582a090a2b69afbc2b41408d52
Summary:
Previously `eden list` reported normally running active checkouts with an
`(active)` suffix, and non-mounted checkouts with no suffix.
The non-mounted situation is rare. This diff changes the command to report
non-mounted checkouts with a `(not mounted)` suffix. I removed the `(active)`
suffix from normally-running checkouts now, but let me know if people think it
is still valuable to keep this annotation in the output.
Reviewed By: chadaustin
Differential Revision: D8558770
fbshipit-source-id: ccce0f7c2031e50f2847089e49007274c31a69a7
Summary:
Update `eden unmount --destroy` to print a message recommending `eden rm`
instead. It still functions as before, but just prints this additional
message.
Also suppress the `--destroy` option from the `eden unmount --help` output.
Reviewed By: chadaustin
Differential Revision: D8558769
fbshipit-source-id: 78215a61fafeae2f7c5cfe4bc55ab0ea6682d2bc
Summary:
The rage command creates a paste by default, so tweak the wording to state
that it gathers information rather than printing it.
Reviewed By: chadaustin
Differential Revision: D8558771
fbshipit-source-id: 819a3c56a9ead173f6aadf639ac954e800c4a046
Summary:
The `debug` command was previously hidden in the `eden --help` output.
There doesn't really seem to be a strong reason to hide this subcommand. It
seems useful to expose it to users who do want to poke around at their current
edenfs instance.
Reviewed By: chadaustin
Differential Revision: D8558768
fbshipit-source-id: f23fd29b473607d83f4f0aec064e72a06dd3ae45
Summary:
Remove python code to parse /proc/self/smaps and associated tests. This is now
done in the EdenServer c++ code.
Reviewed By: simpkins
Differential Revision: D8550273
fbshipit-source-id: 6a2a5cbd25beff412376b0ada683f9dc30e15aff
Summary:
If someone's having problems with Eden and they run `eden doctor` it
sometimes says 'All is well.' even though the person is still having
issues. 'No issues detected.' is a slightly humbler (and accurate)
statement.
Reviewed By: eamonnkent
Differential Revision: D8553069
fbshipit-source-id: 5dd02dc304c6aa6f75b2b2e11757ceb333f8f506
Summary:
The OverlayGoldMasterTest.can_load_overlay_v2 unit test is failing stress test
runs because it keeps overlay data checked into the source repository and runs
the tests directly against this directory. This causes the tests to fail if
multiple test are run in parallel since they are all trying to use this same
directory simultaneously.
We need to make a copy of this directory rather than having the tests run
directly against the source tree to avoid locking issues. The overlay code
can potentially also try to upgrade the overlay format when it opens the
directory. This seems like another reason why it should not run directly
against the directory in the source repository.
This changes the test to make a copy of the overlay directory in a temporary
location, and run the tests against that directory. As part of this change I
also bundled the original input into a tar file.
Reviewed By: chadaustin
Differential Revision: D8555716
fbshipit-source-id: bf24bd96a0a31c097d9cf8e0fbe8b0bfaf009943
Summary:
Eden doctor checks to that configured bind mounts are present for each
Eden repository. It reports errors for missing directories and the device type is incorrect. The device type must differ from the device type of the repository's Eden path.
Eden doctor can fix (by creating) missing bind point directories and then calling the eden thrift server to mount.
Reviewed By: chadaustin
Differential Revision: D8244287
fbshipit-source-id: c78e5ecce63002761a266c5925f2d6618e648e4a
Summary:
We need to call sqlite3_close() if sqlite3_open() fails to avoid leaking the
DB object that was allocated.
Reviewed By: chadaustin, strager
Differential Revision: D8529745
fbshipit-source-id: 1087dac8343b8b3120c89bd3c9a250970e69df8e
Summary:
I saw RemountTest.test_restart_twice() fail once when running locally, but I
haven't been able to reproduce the failure yet. This updates the checks in
this code to hopefully provide better information about what went wrong in
case we see this failure again in the future. This now checks the `eden list`
output first so if something goes wrong we will hopefully have more
information about which checkouts were or weren't remounted.
Reviewed By: strager
Differential Revision: D8528544
fbshipit-source-id: 4f50145aa97956f45f50813070fb73836bc7acde
Summary:
We calculate vmRSS bytes on linux systems by parsing the /proc/self/status
file. The value in that file is in kB so we need to convert it to bytes.
Reviewed By: chadaustin
Differential Revision: D8549009
fbshipit-source-id: 88b18543cb561372dc5eee84293e79fddca8efcb
Summary:
Expand the data we collect in fb 303 collector. Currently we extract data from
procprint it is only reliable for a day (or so).
Here we add:
- memory vm rss bytes (from /proc/self/status)
- memory private bytes (from /proc/self/smaps)
Reviewed By: chadaustin
Differential Revision: D8380917
fbshipit-source-id: dca6fac7af44321c7a6615edb0fde0cb7c8827d0
Summary:
Update the hg_import_helper.py code to ignore CMD_PREFETCH_FILES requests with
more than 4M files.
Old edenfs daemons that send JSON data for this field can sometimes end up
starting newer versions of the hg_import_helper.py script. If they send JSON
data here we want to make sure the `hg_import_helper.py` script does not
actually try to parse the request, which can end up consuming a lot of CPU and
memory if it tries to deserialize data that isn't actually what it expects.
The older JSON data format should always start with '[', which means it will
appear to have at least 1,526,726,656 files.
Reviewed By: chadaustin
Differential Revision: D8508651
fbshipit-source-id: c1e7726398517f97ccc1deafd30620306a9ad80d
Summary:
We've seen what appears to be phantom calls to shutdown() so we'd like
to add some degree of auditing. This diff adds a new method with some
context; this will allow us to distinguish between `eden stop`, `eden restart`
and eden server internal calls to the `shutdown` method. It may still
be possible that something else is calling our shutdown method, but it
seems unlikely as we're only accessible to our own code via a unix domain
socket.
Reviewed By: chadaustin
Differential Revision: D8341595
fbshipit-source-id: 50d58ea0b56e5f42cd37c404048d710bde0d13a3
Summary:
Update edenfs to fork the privhelper process as the first thing we do, before
calling folly::init(). This allows us to drop privileges before processing
command line arguments and doing any other startup work.
Reviewed By: wez
Differential Revision: D8212778
fbshipit-source-id: d67e3700305fdb01cb6188645b37875ceb53d21f
Summary:
Add a call to tell the privhelper process to redirect its log messages to a
different file handle.
This call will make it possible for the privhelper processed to be forked
before the main edenfs process has processed the --logPath argument. The main
edenfs process can now drop privileges before it processes arguments and opens
the log file, and it can then pass the file handle to the privhelper process.
Reviewed By: wez
Differential Revision: D8212776
fbshipit-source-id: 3ec6bfc82ee090216d66c5bfd1b8c2b8819c1f45
Summary:
Add a PrivHelper::detachEventBase() method, and rename PrivHelper::start() to
attachEventBase().
This makes it possible to detach a running PrivHelper from its EventBase and
re-attach it to an EventBase later to restart it. This will be useful in
upcoming diffs to allow performing calls to the PrivHelper before the main
thrift server EventBase has started.
Reviewed By: wez
Differential Revision: D8212777
fbshipit-source-id: d5a9bf672afa8b16e53201ac747d77337e1cc307
Summary:
I've noticed that we don't shut down aux processes when
we remove a mount point.
This can cause `eden rm` to hang if the buck daemon is still running in the background.
Reviewed By: strager
Differential Revision: D8499623
fbshipit-source-id: 9a7f515820a3ffaec86c8c81110283bc0eec3750
Summary:
D8065370 changed the HgImporter code to make a blocking `get()` call on a
future from the hg importer thread pool. This caused deadlocks, since all of
the hg importer threads could become stuck waiting on these `get()` calls to
complete. These would be waiting on RocksDbLocalStore threads which were in
turn all busy waiting to schedule operations on the HgImporter threads.
This fixes the code to use `Future::then()` rather than `Future::get()` to
avoid blocking the HgImporter threads on these operations.
Reviewed By: wez
Differential Revision: D8438777
fbshipit-source-id: a0d647b10ef5a182be2d19f636c2dbc24eab1b23
Summary:
Move the HgProxyHash code out of HgImporter.cpp and into its own top-level
file. This does not have any functional changes other than moving the code
around.
This will make it easier to perform HgProxyHash operation from inside
HgBackingStore.cpp. Currently all of the HgProxyHash operations are done from
inside HgImporter threads. This operation does not need to be done in the
importer threads however, and for operations like putBatch() that complete
asynchronously it generally should not be done from the HgImporter threads.
Reviewed By: wez
Differential Revision: D8438776
fbshipit-source-id: 344652f47e5ccdc6ef20b143dc52d0eeac2886e6
Summary:
There was a conflict between the DirContents refactoring and some of
the Overlay diffs.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D8427680
fbshipit-source-id: a01a2a8456f7a35249d8b65f2b5fcd55825438ee
Summary:
As code review follow-up from D8321501 and D8330070, have getFilePath
return InodePath rather than the index to the null terminator.
Also move getFileTmpPath and ensureTmpDirectoryCreated into their own
functions
Reviewed By: strager
Differential Revision: D8363811
fbshipit-source-id: 42718ed4af07fc7a6600564cc7d934f85fe333ed
Summary:
Creating new Overlay entries in tmp/ and then moving to their home is
even faster than creating them in the root of the overlay.
Reviewed By: strager
Differential Revision: D8330070
fbshipit-source-id: 620dde8fb6ccf9bfdc10872f5911d02fea28fdb0
Summary:
Writing to the overlay is excessively expensive on XFS and, to a
lesser extent, Btrfs. Placing temporary files in the root of the
overlay is dramatically faster than placing them next to their
destination.
Reviewed By: simpkins, strager
Differential Revision: D8321501
fbshipit-source-id: f6c9d5dff9d3eb1d393ff6c760518bc65c361aa2
Summary:
To get a read on whether there's anything egregious about the
overlay on various filesystem types and whether we can make any quick
improvements, I wrote up this quick benchmark.
Reviewed By: strager
Differential Revision: D8310783
fbshipit-source-id: 15fa1ea4bcbb7e4896e52b4acf0f8802133dbf3c
Summary:
The `HgImporter::prefetchFiles()` method was previously sending the file
information to the `hg_import_helper.py` script as JSON. This caused
problems since the python JSON code decodes the path names as Unicode strings.
These then get passed to the mercurial code that expects binary data. It
tries encoding the unicode strings to ASCII, which can throw
UnicodeEncodeErrors.
This updates the code to use a simple binary encoding scheme, since we don't
really want to convert these pathnames from binary to unicode and back again.
The file names are not necessarily even guaranteed to be valid unicode data.
Reviewed By: chadaustin
Differential Revision: D8393757
fbshipit-source-id: 3ec4dcf2bea57e5400af94e9139d0636c446c1f0
Summary:
Update the HgRepo class to set the `HGRCPATH` environment variable to the
empty string to ensure that we always use consistent settings, and are not
affected by the current system's hgrc files.
Reviewed By: chadaustin
Differential Revision: D8410265
fbshipit-source-id: 477a3721860c067f39fbef4bb7d3d14cae3a14b6
Summary:
Per a conversation with simpkins when code reviewing D7882648, this
diff removes the inheritance relationship between TreeInodeState and
DirContents. It doesn't change the binary layout of anything, but
defines DirContents as a typedef of PathMap<DirEntry>.
Reviewed By: strager
Differential Revision: D8232052
fbshipit-source-id: a2166f3ca2ab90fabbded0e48307b8a92a2b0250
Summary:
Several places in edenfs need to represent empty future objects, and were
written before the Future::makeEmpty() method was added. These locations
used Optional<Future> as a workaround.
This updates the code to simply use empty Futures instead of Optional<Future>
now.
Reviewed By: wez
Differential Revision: D8393712
fbshipit-source-id: eeb9e347d0973a4ab602500ee24fba77277d01ea
Summary:
[Folly] In Futures Core, destroy context when destroying callback since they basically go together.
Also removes expectations in an Eden FS test case which affirm the old behavior, along with a TODO to remove the expectations once the affirmed behavior is fixed.
Reviewed By: marshallcline
Differential Revision: D8347040
fbshipit-source-id: ed9fec932ad1e0aa1e40675cf70081d19bbe4325
Summary:
The two FileInode::isSameAs() methods called FileInode::getSha1(), which
returns a folly::Future object, and immediately called `value()` on the
resulting future without waiting for it to complete first.
This bug dates back to D5430128. However, I suspect it was D7888344 that
actually exposed this issue recently. D7888344 updates the
`RocksDbLocalStore` code to perform work in an I/O thread pool. Before this
change the SHA1 futures were likely always immediately ready, so this bug was
not causing problems.
Reviewed By: wez
Differential Revision: D8393671
fbshipit-source-id: ec2116751ddda31a119bfe85eab5612b622f83cf
Summary:
Profiling revealed that we spend a lot of time spookyhashing things
during a big `eden prefetch '**' --silent --no-prefetch` operation, so this
does the obvious and dumb thing to avoid it.
Reviewed By: simpkins
Differential Revision: D8373604
fbshipit-source-id: 16772c0680949792045560f168294239f4cd513b
Summary:
D7946842 moved some of the daemon management code from util.py to daemon.py
but accidentally broke handling of TimeoutError exceptions. The code in
daemon.py was trying to catch python's built-in `TimeoutError` type, while the
code in util.py throws its own `util.TimeoutError` type.
This removes the custom `TimeoutError` type in util.py and just uses the
built-in python type to be consistent everywhere.
Reviewed By: wez
Differential Revision: D8373739
fbshipit-source-id: 8523c444fef74b0758c38b4fe2644d4f68d38442
Summary:
Update the `SnapshotDirstateConsistencyCheck` to catch exceptions thrown when
trying to read the `.hg/dirstate` file.
Reviewed By: wez
Differential Revision: D8373862
fbshipit-source-id: 875330c5fc4c8cbb7ba0393e02d7fbe7722d69fd
Summary:
mkdtemp deliberately removes these bits and doesn't respect
the umask. We perhaps should respect the umask ourselves here.
Reviewed By: chadaustin
Differential Revision: D8335539
fbshipit-source-id: 98b8f3e2db17c65904b12627e52966d5f6b729ef
Summary:
This updates the privhelper code to use the UnixSocket class for performing
I/O. This reduces the number of separate implementations of code we have for
sending file descriptors across Unix domain sockets, and also makes the
privhelper APIs non-blocking.
This will make it easier to clean up some of the initialization ordering in
the future. It will also make it easier to send file descriptors to the
privhelper server, instead of just receiving them. This may be helpful for
passing a file descriptor to use for logging to the privhelper process, which
will make it easier to fork the privhelper before logging redirection has
occurred.
Reviewed By: bolinfest
Differential Revision: D8053422
fbshipit-source-id: 1f8fdf22afc797eead0213be1352ea530762140d
Summary:
Up until now all of the privhelper APIs have been blocking calls. This
changes the privhelper functions to return Futures, and updates all users of
these APIs to be able to handle the results using Futures.
One benefit of this change is that all existing mount points are remounted in
parallel now during startup, rather than being mounted serially. The old code
performed a blocking `get()` call on the future returned by
`EdenServer::mount()`.
The privhelper calls themselves are still blocking for now--they block until
complete and always return completed Future objects. I will update the
privhelper code in a subsequent diff to actually make it asynchronous.
Reviewed By: bolinfest
Differential Revision: D8053421
fbshipit-source-id: 342d38697f67518f6ca96a37c12dd9812ddb151d
Summary:
Update the HgRepo class to pass through the original $PATH environment
variable. Previously it was incorrectly setting $PATH to the repository path.
Reviewed By: chadaustin
Differential Revision: D8359187
fbshipit-source-id: ce9ce374d70c325aa45b5fd643c3d55d30d34cdc
Summary:
This file contained one helper function that didn't really save us much code.
This simply removes it for now.
Reviewed By: wez
Differential Revision: D8329592
fbshipit-source-id: 5172ac0746fc051288c76522c6c3d5ac4097f588
Summary:
This removes the `is_facebook_internal()` function from eden/DEFS, and updates
code that was using it.
The open source build uses CMake rather than buck now, so the DEFS files are
currently only used for Facebook-internal builds. `is_facebook_internal()` is
therefore always true when the DEFS file is being used.
Reviewed By: chadaustin, wez
Differential Revision: D8329591
fbshipit-source-id: d154e9453f78dea2e46d0262a0fcbe58844445e3
Summary:
Some folks might attempt to load this extension despite it
not being useful with eden. The extension knows not to hook in to
the repo when eden is in used, but the various commands to manipulate
the sparse config do not.
This diff adds a little helper function to show a more reasonable
error message than an ugly stack trace when attempting to use them.
Reviewed By: simpkins
Differential Revision: D8328229
fbshipit-source-id: d7cc4c7047b11f8da96e26ab70bd522b52546151
Summary:
Per
35ba669307,
if the return value of DCHECK_NOT_NULL is expected to be unused,
DCHECK should be used instead.
Reviewed By: strager
Differential Revision: D8336319
fbshipit-source-id: 9ea758502baead8941b274dc0ed38ce59b1cc136
Summary:
Fix the integration tests to set CHG_BIN correctly when run manually rather
than via `buck test`.
Reviewed By: wez, quark-zju
Differential Revision: D8329593
fbshipit-source-id: 79fa3c9e9538e4b30a98ef777515527562e1ac1b
Summary:
In some cases we decide to reopen the repo speculatively to see if a
missing cache invalidation was preventing us from importing some data. We're
seeing some leaky behavior and so we're now trying to use the `close` method to
see if that helps clean some things up.
Reviewed By: strager
Differential Revision: D8323584
fbshipit-source-id: 7b022edd58f4ac555b508c1e181677ddc7bc60a4
