Summary:
Now that the import bug has been fixed for some time, it's likely few
people have cached empty files. And if they do, `eden gc; eden
restart` is a fine workaround.
On the other hand, reimporting empty files gums up the importer, and
I've seen several people recently complaining about performance that
could be partially attributed to the fact that their Eden needed to
verify empty files. simpkins has talked about adding a bit to the
cached blob to determine whether it needs reimporting or not, but it's
probably going to take a while for anyone to implement that, and this
reverification logic is hurting people today.
Reviewed By: strager
Differential Revision: D10456519
fbshipit-source-id: 657bc377ee16ce93494075bde4388aed59dceecf
Summary: This was previously hard coded and making it configurable is operationally desirable.
Reviewed By: chadaustin
Differential Revision: D10298279
fbshipit-source-id: 124c3fa319c44aacb3d17ef6e380e9dec49db8be
Summary: Taking (#ifdef'ing) out some of the mononoke code from the Windows builds. Mononoke support will come once we compile the proxygen on Windows.
Reviewed By: strager
Differential Revision: D10079379
fbshipit-source-id: df431f9c57832d43811af41d4512674f1108cacf
Summary:
There's a lot of logic here that need not be offloaded to the
importer threads
Reviewed By: chadaustin
Differential Revision: D9539811
fbshipit-source-id: 1379eef390df6400291a61263d003b493a474b81
Summary:
Add a new unit test that exercises HgImporter and HgBackingStore using an
alternate hg_import_helper.py script that can inject errors into the response
data stream. This tests the behavior of HgImporter when there are
serialization errors with the data, as opposed to legitimately-serialized
error responses containing a python exception message.
Reviewed By: wez
Differential Revision: D9510568
fbshipit-source-id: ae86f69d5559701e9e8a9a18569986d64488eaf4
Summary:
When we load an empty blob from the LocalStore double check with the
BackingStore to confirm that it should actually be empty.
We have seen multiple instances of files being incorrectly imported as empty.
So far this error has always been fixed by a re-import. We still haven't
tracked down the root cause, but this change should help workaround the issue
by ensuring that we double check the file contents before returning the data.
Reviewed By: chadaustin
Differential Revision: D9476522
fbshipit-source-id: 6d57cf15c42736ecbcb106a731259b77db06d8f1
Summary: This commit let `HgImporter` pick up configuration values from `EdenConfig`.
Reviewed By: chadaustin
Differential Revision: D9346293
fbshipit-source-id: cb63f7d13a86058e9bf076eddb52212560a64cb1
Summary:
To improve the determinism of our C++ tests, I am planning to switch
TestMount to a ManualExecutor. This adds a ManualExecutor constructor
to UnboundedQueueExecutor.
In Rust, I'd use a trait, but a simple class with two constructors works fine.
Reviewed By: strager
Differential Revision: D8846553
fbshipit-source-id: c52752105255503d26f1e65494c32b3f62882e44
Summary:
This refactors `HgBackingStore::getTreeForCommit()` so that it no longer invokes
`importThreadPool_.get()` if the `LocalStore` contains the `Tree` for the
specified `commitID` (which is commonly the case). When the `LocalStore` has a
cache hit, there is no need to import any data from Mercurial, so there is no
need to involve the `importThreadPool_`.
As explained in T30958650, `importThreadPool_.get()` can be relatively expensive
because it can have the side-effect of invoking
`HgImporterThreadFactory::newThread()`, which creates a new `HgImporter`, which
in turn does some potentially expensive work (it spawns a new subprocess for the
import helper and does some treemanifest-related I/O, if appropriate).
Because `HgBackingStore::getTreeForCommit()` is called as part of `hg status`,
the existing implementation was preventing us from reliably providing <30ms
`hg status` calls because we would frequently get "unlucky" and unnecessarily
create a new `HgImporter`.
Reviewed By: wez
Differential Revision: D8764805
fbshipit-source-id: 0acf7779560ddf182e7eb50daea54c5ee57030d0
Summary:
D8065370 changed the HgImporter code to make a blocking `get()` call on a
future from the hg importer thread pool. This caused deadlocks, since all of
the hg importer threads could become stuck waiting on these `get()` calls to
complete. These would be waiting on RocksDbLocalStore threads which were in
turn all busy waiting to schedule operations on the HgImporter threads.
This fixes the code to use `Future::then()` rather than `Future::get()` to
avoid blocking the HgImporter threads on these operations.
Reviewed By: wez
Differential Revision: D8438777
fbshipit-source-id: a0d647b10ef5a182be2d19f636c2dbc24eab1b23
Summary:
When testing D8108649 I accidentally deleted all of my trees
but didn't delete my commit2tree mapping. This diff allows Eden to
recover from that situation.
Reviewed By: wez
Differential Revision: D8108728
fbshipit-source-id: 94a9393294ca259303026c297683dac4b3ecfac4
Summary:
Remove getTreeFuture and have getTree always return a Future. It's a
little unfortunate that this will always allocate, but it sounds like
we might decide to put all RocksDB access on a background thread to
increase CPU parallelism.
Reviewed By: bolinfest
Differential Revision: D8101430
fbshipit-source-id: e12b7ab07b3468114a58753768655c107265b8af
Summary:
This removes the main point of contention for eden prefetch
in two ways:
1. We batch up the complete list of blobs so that they can be processed
in bulk rather than stalling the tree walk
2. We can ask remotefilelog to check and fetch that list to the local
hgcache, again as a batch, rather than by forcing the data to be
loaded through into the local store
The goal of this prefetch is to bulk load data from the mercurial server
so that a subsequent file access doesn't have to make a one-off ssh session
for each one, rather than making sure that all the data is loaded into
the local store.
Reviewed By: chadaustin
Differential Revision: D7965818
fbshipit-source-id: 753400460d633b5467c5110e3f5608ce06106e00
Summary:
Add EdenCPUThreadPool and UnboundedQueueThreadPool types to make it clearer
that it's always okay for prefetch, deferred diff entry, and hg import to
shuttle work back to the main thread pool.
This diff changes no behavior - it just makes some invariants explicit.
Reviewed By: wez, simpkins
Differential Revision: D6504117
fbshipit-source-id: 3400ad55c00b3719ecba31807fd992442f622cd9
Summary:
Have HgBackingStore hold multiple HgImporters in its own thread pool. Incoming
requests are processed by threads in the thread pool.
Reviewed By: wez
Differential Revision: D6265043
fbshipit-source-id: b2d4f345b772f296c5335a7fbcadfce1d93245fd
Summary:
Per discussion with bolinfest, this brings Eden in line with clang-format.
This diff was generated with `find . \( -iname '*.cpp' -o -iname '*.h' \) -exec bash -c "yes | arc lint {}" \;`
Reviewed By: bolinfest
Differential Revision: D6232695
fbshipit-source-id: d54942bf1c69b5b0dcd4df629f1f2d5538c9e28c
Summary:
D5067763 introduced a potential deadlock. The issue is that all of the FUSE threads were blocked
on the HgImporter thread pool, which completes its futures back on serverEventThread_. The
FUSE threads were blocked on Future::get() in ensureDataLoaded().
Eventually, the right fix is some combination of eliminating ensureDataLoaded() and
replacing it with an explicitly-asynchronous API.
Reviewed By: bolinfest
Differential Revision: D6212858
fbshipit-source-id: 42b17d3e20a200f26b87588784edb5ee51e96a4a
Summary:
Have HgBackingStore hold multiple HgImporters in its own thread pool. Incoming
requests are processed by threads in the thread pool.
Reviewed By: wez
Differential Revision: D5067763
fbshipit-source-id: d666b1026455d13442367673010b5934ff4cdb48
Summary:
Update HgBackingStore and GitBackingStore to accept the repository path as an
AbsolutePathPiece rather than as a plain StringPiece.
Reviewed By: bolinfest
Differential Revision: D5309172
fbshipit-source-id: aca4818c3add11d07ece796298f6175ca4fb1448
Summary:
This is generated by applying clang-tidy `-checks=modernize-use-override` to all the c++ code in project eden.
It enforces the use of the keywords `virtual`, `verride` and `final` in a way compliant to the style guide.
Reviewed By: igorsugak
Differential Revision: D5108807
fbshipit-source-id: 596f2d73f1137de350114416edb1c37da5423ed5
Summary:
Update copyright statements to "2016-present". This makes our updated lint
rules happy and complies with the recommended license header statement.
Reviewed By: wez, bolinfest
Differential Revision: D4433594
fbshipit-source-id: e9ecb1c1fc66e4ec49c1f046c6a98d425b13bc27
Summary:
Update the ObjectStore and BackingStore classes to have APIs that return
folly::Future objects, rather than blocking until the requested data is loaded.
For now most users still call the blocking versions of getBlob() and getTree().
Furthermore, all of the Future-based implementations actually still block
until the data is ready. I will update the code to use these new APIs in
future diffs, and then deprecate the non-future based versions.
Reviewed By: bolinfest
Differential Revision: D4318055
fbshipit-source-id: a250c23b418e69b597a4c6a95dbe80c56da5c53b
Summary:
Move the eden/fs/importer/hg code to eden/fs/store/hg. This import code is
only used by the HgBackingStore, so logically it belongs together.
The eden/fs/importer/git directory was already moved into eden/fs/store/git by
D3448752, so this makes the mercurial code similar.
Reviewed By: bolinfest
Differential Revision: D3458128
fbshipit-source-id: 54b1f707197e41ff5f5f4a163a34d3d1ab432e21
Summary:
This adds an HgBackingStore implementation which can load tree data from a
mercurial repository. Blob loading is not implemented yet, but will come in a
separate diff.
This also adds a minimal GitBackingStore class. The GitBackingStore has nearly
no functionality, but is needed to keep the existing git functionality working.
Reviewed By: bolinfest
Differential Revision: D3409743
fbshipit-source-id: dbebf53e9de08bd1469e489baa48b84cbf889511
