Summary:
The `HgImporter::prefetchFiles()` method was previously sending the file
information to the `hg_import_helper.py` script as JSON. This caused
problems since the python JSON code decodes the path names as Unicode strings.
These then get passed to the mercurial code that expects binary data. It
tries encoding the unicode strings to ASCII, which can throw
UnicodeEncodeErrors.
This updates the code to use a simple binary encoding scheme, since we don't
really want to convert these pathnames from binary to unicode and back again.
The file names are not necessarily even guaranteed to be valid unicode data.
Reviewed By: chadaustin
Differential Revision: D8393757
fbshipit-source-id: 3ec4dcf2bea57e5400af94e9139d0636c446c1f0
Summary:
Update the HgRepo class to set the `HGRCPATH` environment variable to the
empty string to ensure that we always use consistent settings, and are not
affected by the current system's hgrc files.
Reviewed By: chadaustin
Differential Revision: D8410265
fbshipit-source-id: 477a3721860c067f39fbef4bb7d3d14cae3a14b6
Summary:
Per a conversation with simpkins when code reviewing D7882648, this
diff removes the inheritance relationship between TreeInodeState and
DirContents. It doesn't change the binary layout of anything, but
defines DirContents as a typedef of PathMap<DirEntry>.
Reviewed By: strager
Differential Revision: D8232052
fbshipit-source-id: a2166f3ca2ab90fabbded0e48307b8a92a2b0250
Summary:
Several places in edenfs need to represent empty future objects, and were
written before the Future::makeEmpty() method was added. These locations
used Optional<Future> as a workaround.
This updates the code to simply use empty Futures instead of Optional<Future>
now.
Reviewed By: wez
Differential Revision: D8393712
fbshipit-source-id: eeb9e347d0973a4ab602500ee24fba77277d01ea
Summary:
[Folly] In Futures Core, destroy context when destroying callback since they basically go together.
Also removes expectations in an Eden FS test case which affirm the old behavior, along with a TODO to remove the expectations once the affirmed behavior is fixed.
Reviewed By: marshallcline
Differential Revision: D8347040
fbshipit-source-id: ed9fec932ad1e0aa1e40675cf70081d19bbe4325
Summary:
The two FileInode::isSameAs() methods called FileInode::getSha1(), which
returns a folly::Future object, and immediately called `value()` on the
resulting future without waiting for it to complete first.
This bug dates back to D5430128. However, I suspect it was D7888344 that
actually exposed this issue recently. D7888344 updates the
`RocksDbLocalStore` code to perform work in an I/O thread pool. Before this
change the SHA1 futures were likely always immediately ready, so this bug was
not causing problems.
Reviewed By: wez
Differential Revision: D8393671
fbshipit-source-id: ec2116751ddda31a119bfe85eab5612b622f83cf
Summary:
Profiling revealed that we spend a lot of time spookyhashing things
during a big `eden prefetch '**' --silent --no-prefetch` operation, so this
does the obvious and dumb thing to avoid it.
Reviewed By: simpkins
Differential Revision: D8373604
fbshipit-source-id: 16772c0680949792045560f168294239f4cd513b
Summary:
D7946842 moved some of the daemon management code from util.py to daemon.py
but accidentally broke handling of TimeoutError exceptions. The code in
daemon.py was trying to catch python's built-in `TimeoutError` type, while the
code in util.py throws its own `util.TimeoutError` type.
This removes the custom `TimeoutError` type in util.py and just uses the
built-in python type to be consistent everywhere.
Reviewed By: wez
Differential Revision: D8373739
fbshipit-source-id: 8523c444fef74b0758c38b4fe2644d4f68d38442
Summary:
Update the `SnapshotDirstateConsistencyCheck` to catch exceptions thrown when
trying to read the `.hg/dirstate` file.
Reviewed By: wez
Differential Revision: D8373862
fbshipit-source-id: 875330c5fc4c8cbb7ba0393e02d7fbe7722d69fd
Summary:
mkdtemp deliberately removes these bits and doesn't respect
the umask. We perhaps should respect the umask ourselves here.
Reviewed By: chadaustin
Differential Revision: D8335539
fbshipit-source-id: 98b8f3e2db17c65904b12627e52966d5f6b729ef
Summary:
This updates the privhelper code to use the UnixSocket class for performing
I/O. This reduces the number of separate implementations of code we have for
sending file descriptors across Unix domain sockets, and also makes the
privhelper APIs non-blocking.
This will make it easier to clean up some of the initialization ordering in
the future. It will also make it easier to send file descriptors to the
privhelper server, instead of just receiving them. This may be helpful for
passing a file descriptor to use for logging to the privhelper process, which
will make it easier to fork the privhelper before logging redirection has
occurred.
Reviewed By: bolinfest
Differential Revision: D8053422
fbshipit-source-id: 1f8fdf22afc797eead0213be1352ea530762140d
Summary:
Up until now all of the privhelper APIs have been blocking calls. This
changes the privhelper functions to return Futures, and updates all users of
these APIs to be able to handle the results using Futures.
One benefit of this change is that all existing mount points are remounted in
parallel now during startup, rather than being mounted serially. The old code
performed a blocking `get()` call on the future returned by
`EdenServer::mount()`.
The privhelper calls themselves are still blocking for now--they block until
complete and always return completed Future objects. I will update the
privhelper code in a subsequent diff to actually make it asynchronous.
Reviewed By: bolinfest
Differential Revision: D8053421
fbshipit-source-id: 342d38697f67518f6ca96a37c12dd9812ddb151d
Summary:
Update the HgRepo class to pass through the original $PATH environment
variable. Previously it was incorrectly setting $PATH to the repository path.
Reviewed By: chadaustin
Differential Revision: D8359187
fbshipit-source-id: ce9ce374d70c325aa45b5fd643c3d55d30d34cdc
Summary:
This file contained one helper function that didn't really save us much code.
This simply removes it for now.
Reviewed By: wez
Differential Revision: D8329592
fbshipit-source-id: 5172ac0746fc051288c76522c6c3d5ac4097f588
Summary:
This removes the `is_facebook_internal()` function from eden/DEFS, and updates
code that was using it.
The open source build uses CMake rather than buck now, so the DEFS files are
currently only used for Facebook-internal builds. `is_facebook_internal()` is
therefore always true when the DEFS file is being used.
Reviewed By: chadaustin, wez
Differential Revision: D8329591
fbshipit-source-id: d154e9453f78dea2e46d0262a0fcbe58844445e3
Summary:
Some folks might attempt to load this extension despite it
not being useful with eden. The extension knows not to hook in to
the repo when eden is in used, but the various commands to manipulate
the sparse config do not.
This diff adds a little helper function to show a more reasonable
error message than an ugly stack trace when attempting to use them.
Reviewed By: simpkins
Differential Revision: D8328229
fbshipit-source-id: d7cc4c7047b11f8da96e26ab70bd522b52546151
Summary:
Per
35ba669307,
if the return value of DCHECK_NOT_NULL is expected to be unused,
DCHECK should be used instead.
Reviewed By: strager
Differential Revision: D8336319
fbshipit-source-id: 9ea758502baead8941b274dc0ed38ce59b1cc136
Summary:
Fix the integration tests to set CHG_BIN correctly when run manually rather
than via `buck test`.
Reviewed By: wez, quark-zju
Differential Revision: D8329593
fbshipit-source-id: 79fa3c9e9538e4b30a98ef777515527562e1ac1b
Summary:
In some cases we decide to reopen the repo speculatively to see if a
missing cache invalidation was preventing us from importing some data. We're
seeing some leaky behavior and so we're now trying to use the `close` method to
see if that helps clean some things up.
Reviewed By: strager
Differential Revision: D8323584
fbshipit-source-id: 7b022edd58f4ac555b508c1e181677ddc7bc60a4
Summary:
We need to explicitly move the continuation to the right thread.
I don't know why our tests didn't catch this case, and I'm just throwing
up the diff to unblock the other one I'm working on.
Reviewed By: chadaustin
Differential Revision: D8323051
fbshipit-source-id: e7c39a086ced5dc6487c8f6014132e856c91af46
Summary:
When testing D8108649 I accidentally deleted all of my trees
but didn't delete my commit2tree mapping. This diff allows Eden to
recover from that situation.
Reviewed By: wez
Differential Revision: D8108728
fbshipit-source-id: 94a9393294ca259303026c297683dac4b3ecfac4
Summary:
I've been troubleshooting eden integration test failures on my
devserver and traced it to some slightly off behavior in the telemetry
wrapper.
The wrapper was setting `CHGHG` to `hg.real` rather than the computed
path to the `hg.real` executable. In the eden integration tests this
path is the buck generated `hg.par`. The problem this caused was running
the installed hg.real rather than the one from the test environment
and this caused resolution of the eden extension to fail.
Once I fixed that up I found that chg had detected a problem with the
paths to the hg executable that were being used; we were picking up `chg`
from the system path and had a similar issue to above.
I introduced an environmental variable `CHG_BIN` to hold the desired path
and set it to the buck built `chg` binary.
In the process of this I found that `chg` was triggering a UBSAN issue
by passing a nullptr as the second argument to `memcpy`. I've included
the trivial fix for that in this diff also.
Reviewed By: quark-zju
Differential Revision: D8274636
fbshipit-source-id: 7ee0740cbfb447ab41b9e08308767d42790ba296
Summary:
I didn't consider this when I added the prefetch stuff, and it is
currently causing the watchman eden integration tests to fail. It was bisected
to the watchman prefetch diff.
Reviewed By: chadaustin
Differential Revision: D8275074
fbshipit-source-id: 0a2c6d8680b07157de26dafe437551c7314b86df
Summary:
Previously the ManifestEntry node field was public and consumers could
read it directly. This required that the node field be updated manually, which
added complications to other code paths. In a future diff we'll be simplifying
the SubTreeIterator code to not require setting node, and to do so we make node
calculated in this accessor.
Reviewed By: quark-zju
Differential Revision: D8156807
fbshipit-source-id: e1908b2149bc0ec8fb1279e37b8a66988ff125c6
Summary: This parameter was only supported for fbbuild.
Reviewed By: yfeldblum
Differential Revision: D8246482
fbshipit-source-id: 95db878a34dce5694639364f2838bb4cccd723d3
Summary:
Preparation for adding an HgBackingStore test.
Per @[1077184107:simpkins], use_hg_tree_manifest defaults on and folly::logging is automatically initialized now, so we can just remove main() entirely.
Reviewed By: wez
Differential Revision: D8226799
fbshipit-source-id: 5b5038bc311f62c1ef9ed943f57ce8f7d68bdc89
Summary:
We don't use the Allocator template parameter and it makes forward
declaring PathMap harder, so remove it.
Reviewed By: simpkins
Differential Revision: D8231982
fbshipit-source-id: 5c0faed8f61dc7699c8bb6660292f255feb10e29
Summary:
While I'm in here, borrow the top two bits from mode_t for hasHash_
and hasInodePointer_, making DirEntry fit in four words.
Eventually I want to replace mode_t with dtype_t, but that can't be
done until migration to the InodeMetadataTable is mostly complete. If
I made this change too early, we might lose some of the mode bits
specified when creating a file. If said mode bits resulted in a change
to u+x, the file could look changed relative to source control.
I updated some of the DirEntry documentation while I was at it.
Reviewed By: simpkins
Differential Revision: D7941582
fbshipit-source-id: f62e58f3737c1189ea17cd434b6fef14af359e0a
Summary:
This fixes a bug simpkins pointed out in D6891479 - we weren't
updating mtime and ctime on renames.
Reviewed By: simpkins
Differential Revision: D7937303
fbshipit-source-id: 08fd8f4fe5d99d33e9f7629965d6146330c8f35b
Summary:
Per code review feedback from D6891479, this diff enforces that
metadata writes and reads are done while the corresponding inode's
state lock is held.
Reviewed By: simpkins
Differential Revision: D7884463
fbshipit-source-id: d0e7a95415c280441276452ece7233d4cbf5e942
Summary:
1. Enabled a number of additional C++ compiler warnings in Eden.
2. Fixed warnings-turned-errors that resulted from this change.
Reviewed By: simpkins
Differential Revision: D8132543
fbshipit-source-id: 2290ffaaab55024d582e29201a1bcaa1152e6b3e
Summary:
Like D7867399, split TreeInode's synchronized state into a top-level
class. This is a step towards using the type system to perform
lock-safe metadata updates.
Reviewed By: simpkins
Differential Revision: D7882648
fbshipit-source-id: 27262df8ed9137c8478c68ebf4c4f13878655754
Summary:
Eden will often have a significant number of trees loaded - this saves
8 bytes per entry per loaded TreeInode. It also makes it clear that
once an inode pointer is assigned, the inode number is redundant.
Reviewed By: simpkins
Differential Revision: D7869662
fbshipit-source-id: 21a8266ff5189d3ba9cb614a325cc9d8c3ca305e
Summary:
Shrinking Entry to five words was a bit tricky with some subtle
situations, so I split it into two diffs to verify the tests passed
after each stage.
This diff replaces folly::Optional<Hash> with a boolean and Hash.
Reviewed By: simpkins
Differential Revision: D7869502
fbshipit-source-id: 2df109472d9565e96e8621407f62a63b4f1dbcad
Summary:
FileInode and TreeInode's State classes are complicated enough that
they deserve to be lifted out. In addition, this is necessary for
using the type system to enforce that contents locks are held in
InodeBase's metadata accessors.
Reviewed By: simpkins
Differential Revision: D7867399
fbshipit-source-id: 6ce082149ba02099487e8caed33a7bd8510dfebb
Summary:
Have getBlobMetadata always return a Future. It's a little unfortunate
that this will always allocate, but it sounds like we might decide to
put all RocksDB access on a background thread to increase CPU
parallelism.
Reviewed By: bolinfest
Differential Revision: D8101464
fbshipit-source-id: 6e9ec95050c366c7c57519e3f68b311470b2addd
Summary: We are changing `folly::collectAll` to return `SemiFuture` rather than `Future` and this is needed as an interim step. After all calls to `collectAll` are changed to `collectAllSemiFuture`, we'll be renaming it back to `collectAll`.
Reviewed By: yfeldblum
Differential Revision: D8210974
fbshipit-source-id: e4a7464f4a1c3ede157b8377a4df97d943001f60
Summary:
Remove getTreeFuture and have getTree always return a Future. It's a
little unfortunate that this will always allocate, but it sounds like
we might decide to put all RocksDB access on a background thread to
increase CPU parallelism.
Reviewed By: bolinfest
Differential Revision: D8101430
fbshipit-source-id: e12b7ab07b3468114a58753768655c107265b8af
Summary:
Remove getBlobFuture and have getBlob always return a Future. It's a
little unfortunate that this will always allocate, but it sounds like
we might decide to put all RocksDB access on a background thread to
increase CPU parallelism.
Reviewed By: bolinfest
Differential Revision: D8101402
fbshipit-source-id: d6cbbd7fe4fe55bad661c9158297db2f03f7d352
Summary:
I kept running into issues trying to get graceful restart and
flush_cache to work together in the hg integration suite, so add a
test to ensure flush_cache succeeds after a graceful restart in the
main integration suite.
Also, to make the test's output easier to follow, add logging when
invalidating inodes.
Reviewed By: simpkins
Differential Revision: D8215961
fbshipit-source-id: 33db4292af3969ae23940c3027ba513ed20c53fb
Summary:
Remember the maximum inode number in a file when the Overlay is shut
down cleanly, avoiding a full Overlay scan.
Reviewed By: simpkins
Differential Revision: D7912647
fbshipit-source-id: 8adbafed15259af668a221baa829e1b1f44090d7
Summary:
Add a command line option to start edenfs only if there is at least one
Eden checkout already configured.
This makes it easier to automatically start edenfs on system boot only if the
user needs edenfs.
Reviewed By: wez
Differential Revision: D8174819
fbshipit-source-id: bf585b8c5aa719d3b6424567f2e21c1c63850d66
Summary:
This adds a debug command to blow away all RocksDB information that
can be reproduced from Mercurial. We will use it to help an Eden user
recover from a corrupted blob.
Reviewed By: bolinfest
Differential Revision: D8108649
fbshipit-source-id: 056dec19d51b9e430b3c2a249747b26830cfc875
Summary:
Add a clearCaches function to LocalStore that deletes all data from
LocalStore that could be retrieved from Mercurial.
Reviewed By: wez
Differential Revision: D8101365
fbshipit-source-id: d46d0db94e6f85aaf542d9f6b9b96fbdcc548b57
Summary:
The Overlay is the natural home for nextInodeNumber_ now that every
directory allocates inode numbers for its children right away. This
also simplifies serializing nextInodeNumber_ to disk in the following
diff.
Reviewed By: simpkins
Differential Revision: D8192442
fbshipit-source-id: 9b776a73c8d7653002b55985d592b1746e52f878
