Summary: This diff addresses the unreleased mapped pack files in the old datapack code.
Reviewed By: chadaustin
Differential Revision: D19399737
fbshipit-source-id: 86a6254a2939fd69e1ce2b25b8bfcb36b0deb16b
Summary:
As reported by JT, EdenFS may hold the file descriptor of mapped pack files too long even when it is deleted by external processes, thus taking more disk spaces.
This diff fixes this problem by making EdenFS periodically rescan the pack files.
Reviewed By: chadaustin
Differential Revision: D19395439
fbshipit-source-id: 4bfd6a7ac13dceb3099d2704d62b3825433aff4b
Summary:
We want to have a shared pool of `HgImporter` across different threads, but that would require `HgImporter` is safe to be *passed* between threads. (This is not making `HgImporter` entirely thread safe.)
However, `HgImporter` currently holds a pointer to the thread local fb303 counter. This diff pushes down `EdenStats` so we only access the thread local stats variable when we need to add counters. This might be a little slower but it does make it safer.
Reviewed By: chadaustin
Differential Revision: D19053250
fbshipit-source-id: 44a897acc90c6042ae22a0417eece39e099ee13f
Summary: The only reason `HgImporter` still holds a pointer to LocalStore is for looking up proxy hashes. We can directly pass in the Mercurial hash and path instead.
Reviewed By: chadaustin
Differential Revision: D19049039
fbshipit-source-id: 45d1e1f363ed73dca447b22e5891495cf9ad202b
Summary:
Remove some half-baked, unnecessary logic for caching sizes separately
from SHA-1. Eden's backing stores do not support chunking large files
yet, so there's no value in caching content SHA-1 and size
separately. This fixes a scenario where fetching blob size and then
SHA-1 would result in two backing store imports.
Reviewed By: fanzeyi
Differential Revision: D19169096
fbshipit-source-id: dc32f3313e5f4230c06a5bbaa67da7bf0febaba8
Summary:
Troditionally, ObjectStore relied on HgBackingStore writing directly
to LocalStore in order to cache trees. This had the unfortunate side
effect that other backing store implementations did not benefit from
tree caching.
Move tree caching into ObjectStore so all backing stores benefit from
tree caching.
Reviewed By: simpkins, fanzeyi
Differential Revision: D19168211
fbshipit-source-id: b1019591ebb4760cc8b933b9adb82174b8e5fa1f
Summary:
If a GC ran at the same time as HgBackingStore imported a commit, the
import would fail. Rather than round-tripping the tree through the
LocalStore, just return it.
Reviewed By: genevievehelsel
Differential Revision: D19167798
fbshipit-source-id: eddb280def349483289d2e300a58eff3ad7416e1
Summary:
This diff turns the return type of `BackingStore::getBlob` from `folly::Future` into `folly::SemiFuture` to prevent executor leaks.
This also enable us to remove the need of holding `serverThreadPool` from backing stores.
----
**Changes**
* `ObjectStore` now needs to hold a `folly::Executor::KeepAlive` that is used to turn `SemiFuture`s it gets from backing stores into `Future`.
* Signature changes of the implementations of `BackingStore` class.
* For tests, I chose to use `QueuedImmediateExecutor` in place of `UnboundedQueueExecutor` as it will basically execute tasks inline. I'm concerned introducing thread pool executor in tests may turn tests flaky.
Reviewed By: wez
Differential Revision: D18669664
fbshipit-source-id: 0cae89f365dcf8b345b49d64469a530cf25d4ac5
Summary:
The last remaining user of ObjectStore::getBlobMetadata was a debug
Thrift call. Remove it and update the Thrift call to use getBlobSize
and getBlobSha1.
Reviewed By: pkaush
Differential Revision: D18663376
fbshipit-source-id: 86baefc9004a07aac4ddf5849870431be04c75f2
Summary:
A spike in automatic GCs usually implies something has gone wrong. Log
an event for each one, recording the cache size prior to the GC and
the cache size after.
Reviewed By: simpkins
Differential Revision: D18902580
fbshipit-source-id: 158b2635733a415a9fcc7c412b2c0f44ed04aa01
Summary: Enabling getScmStatusBetweenRevisions to work on Windows and its tests.
Reviewed By: simpkins
Differential Revision: D18431271
fbshipit-source-id: eee82538e2fc3d7e371c96fc271cd9662ea6d737
Summary:
Add an option `experimental:use-edenapi` to `EdenConfig`.
See the next diff for usage.
Reviewed By: chadaustin
Differential Revision: D18605549
fbshipit-source-id: 2786c21bb38a76229078662cc5c1ddf906d1be4a
Summary: Adds a function which takes both the manifestID and the commitID to get a Tree. This will be used in `checkOutRevision()` and this allows us to skip looking up the manifestID since the caller can just pass it in themselves.
Reviewed By: wez
Differential Revision: D18719405
fbshipit-source-id: 919f0a7c84bff4a2f0bc20110c45bd272f9e9107
Summary:
It is very painful for Eden's automatic GC to delete trees from the
local store. An unfortunate number of programs scan the entire repo,
causing every tree to be loaded. If Eden's automatic GC kicks in, it
causes all trees to be downloaded again, blocking both FUSE and Thrift
requests.
Temporarily mark trees as persistent.
Reviewed By: fanzeyi
Differential Revision: D18842066
fbshipit-source-id: 4a22defcb1151912337d4167ac49852519c7789d
Summary: Saves loadFileFromPath in the DiffContext. This allows the source control differ ( at `eden/fs/store/Diff.cpp`) to call this function while it is trying to load gitIgnores and resolve symlinks.
Reviewed By: simpkins
Differential Revision: D18647088
fbshipit-source-id: bddff0158130f0c96ac263bcce81dc51fc017800
Summary:
This message is pretty noisy when running EdenFS with an increased debug
logging level. Bump the debug level down so it doesn't spam the logs when
running at a slightly elevated debug level. Also fix a minor typo.
Reviewed By: fanzeyi
Differential Revision: D18820281
fbshipit-source-id: 668367337f4a8cf2a38003c8e0034969d335c2c0
Summary: There are a couple of functions in `store/Diff.h` which are not used elsewhere except for testing. These functions are tested by `diffCommitsForStatus()` anyways so these standalone functions are not needed.
Reviewed By: chadaustin
Differential Revision: D18690006
fbshipit-source-id: f2b24575c17403d7241896f35f4e0e16bb03b7ce
Summary:
Add default case handling for some switch statements that do not handle the
case of an enum variable being set to something outside of one of the defined
values.
Reviewed By: fanzeyi
Differential Revision: D18652102
fbshipit-source-id: abcb3f9dc1f6b0dc761170c7115ddd9ad5145bc4
Summary:
Now that the fb-mercurial sources are available in the Eden repository, update
our CMake build files to always build them. This moves the build logic from
the centralized `FBMercurialFeatures.cmake` file into `CMakeLists.txt` files
in the appropriate subdirectories.
Reviewed By: chadaustin
Differential Revision: D18588011
fbshipit-source-id: ded9decde5c2ec766aae0bb0f4f5b021d1044a98
Summary:
Update the CMake build to always build the Rust datapack libraries, even on
Windows. This allows us to completely eliminate the `EDEN_HAVE_RUST_DATAPACK`
checks.
Note that I did leave the `EDEN_HAVE_RUST_DATAPACK` macro in place, as we
still do not build the Rust datapack code on Mac when building with Buck.
Reviewed By: fanzeyi
Differential Revision: D18588008
fbshipit-source-id: 1a4c9ceec5372d0e6a7313d2eb87edabd1e60a96
Summary:
Update Eden's top-level CMakeLists.txt file to build the Rust datapack
libraries. Previously these were built by invoking CMake separately inside th
`eden/scm` subdirectory. Now that the code has been combined into a single
location we can use a single CMake invocation to drive the build of both these
components.
The old code did not build the Rust datapack code on Windows, and this diff
does not change that behavior. I'm not aware of any reason to skip building
this on Windows, so I plan to enable building this code on Windows in a
subsequent diff.
Reviewed By: pkaush
Differential Revision: D18588006
fbshipit-source-id: 20f4f0ea9fef8595a9dd35a21115952b2808c824
Summary: Even if blobs have different hashes, they could have the same contents. For example, if between the two revisions being compared, if a file was changed and then later reverted. In that case, the contents would be the same but the blobs would have different hashes. Currently, `getScmStatusBetweenRevisions()` would report false positives in this case. This is also needed so we do not report false positives in `getScmStatus()` when hit this code path
Reviewed By: simpkins
Differential Revision: D18647086
fbshipit-source-id: 66e12648a24fd7e5612eee5e599a5b81c7c5f2d1
Summary:
Annotate some import statements so autodeps won't complain about them, since
it doesn't know where they come from in the build.
Note that `hg_import_helper.py` isn't used any more by the current code, and
should ideally be killed off soon. However, just annotating these import
statements for now in the short term is the easiest way to make autodeps
happy.
Reviewed By: fanzeyi
Differential Revision: D18652104
fbshipit-source-id: 310bc5c63447e3bf4dfd081d7b3930eed4d0498b
Summary: Adding an experimental option to EdenFS so we can disable blob caching if we need.
Reviewed By: chadaustin
Differential Revision: D18441665
fbshipit-source-id: 56751c0199d6658bfbf7ec3865f746a7279324ab
Summary:
This diff adds the C++ conversation methods that converts the Rust struct introduced in the previous Diff into EdenFS's `Tree`.
We have to keep the RocksDB write batch in for storing proxy hashes.
Reviewed By: wez
Differential Revision: D18365504
fbshipit-source-id: 2a2cd2cfebd9f0ae4e279db608becc5890c5ccdf
Summary:
Merge the fb-mercurial code into the Eden repository, under the
`eden/scm` subdirectory.
Reviewed By: quark-zju
Differential Revision: D18445774
fbshipit-source-id: fc3307f9937e0c7e1c8f7d03c5102c4fe5dedb10
Summary:
The `HgDatapackStore.cpp` file currently depends on code not available on Mac,
so do not attempt to build it there.
Reviewed By: fanzeyi
Differential Revision: D18467948
fbshipit-source-id: cce37bb480426e6d972045321f15c1f2aa4457ee
Summary:
I spent hours debugging a deadlock in my code that was caused by
accidental usage of the deprecated Synchronized::operator->. Avoid
that in the future by building Eden deprecation-clean.
Reviewed By: genevievehelsel
Differential Revision: D18354757
fbshipit-source-id: fe74250635f779bb5010f907d57951132b9edea4
Summary:
This diff makes EdenFS to be able to import manifest from Rust code.
We are only able to import raw manifest data from Rust for now. This code still
send the imported data to the ctreemanifest for parsing. Follow up diff will
fix this.
Reviewed By: chadaustin
Differential Revision: D18097701
fbshipit-source-id: b821a1d595f43a19d9ce9cd6d5ff5174d49e34e6
Summary:
Background: https://fb.workplace.com/groups/sourcecontrolteam/permalink/2440123159442349/?comment_id=2443624539092211
This may or may not be the root cause but this is definitely gonna be causing issues.
Between D17468473 and D17866320, turning on this option will make EdenFS to use the old Rust code that does not strip copyrev data.
Reviewed By: wez
Differential Revision: D18250237
fbshipit-source-id: b3816bd08b8dc3b2a90931829c368a58731c7843
Summary:
The Windows build spews a great many warnings. Address some of them by
enabling the unused-exception-parameter warning on Clang/Linux too.
Reviewed By: yfeldblum
Differential Revision: D18178930
fbshipit-source-id: efecb605b84d4f06c8c8411a23d17904bbdff746
Summary: Formatting had diverged in a few places. Fix that up.
Reviewed By: fanzeyi
Differential Revision: D18123219
fbshipit-source-id: 832cdd70789642f665a029196998928a9173be81
Summary: We should handle errors from Rust code gracefully so it does not abort the mounting process.
Reviewed By: chadaustin
Differential Revision: D18096335
fbshipit-source-id: 17cec6f619ea2cc8c1719b0cf97287c815ff6f0a
Summary: This diff switches EdenFS from using the `revisionstore` crate into `backingstore`
Reviewed By: chadaustin
Differential Revision: D17866320
fbshipit-source-id: f6de1fe568ae893e4c2555039df6edfb0a450885
Summary:
krallin noted that we were not collecting LFS stats from the import helper.
This diff should get us to run the import helper with those stats enabled, although
we don't yet know how useful those will be in practice.
We originally defaulted to `hg.real` to bypass some startup overhead and make our tests
run faster. We've evolved more control over this in the test harness over time, so I
think we're probably fine to make the runtime non-test default just be the normal hg
entrypoint.
Reviewed By: chadaustin
Differential Revision: D18039164
fbshipit-source-id: 7593cdca9afe30fa2fc928b84ba552dbecfed388
Summary: Removing an import that is not really used.
Reviewed By: pkaush
Differential Revision: D17997050
fbshipit-source-id: 7ba6a869937d51d640c72c22417a26fcf42c6ab3
Summary: Tracing was not an accurate name for what this directory had become. So rename it to telemetry.
Reviewed By: wez
Differential Revision: D17923303
fbshipit-source-id: fca07e8447d9b9b3ea5d860809a2d377e3c4f9f2
Summary:
Instead of having accessors for every config setting in EdenConfig,
just expose the ConfigSettings directly.
Reviewed By: fanzeyi
Differential Revision: D17847805
fbshipit-source-id: 8c6c1010c010113cf859677449797ea916f2a2a5
Summary: Removes `TreeDiffer` class and passes `DiffContext` through standalone `TreeDiffer` functions as first argument as per comment on D17400466 for setup for processing gitignores in the `TreeDiffer` codepath. (also this allows for easy implementation of short circut of `future_getScmStatusBetweenRevisions` similar to D17531102)
Reviewed By: chadaustin
Differential Revision: D17717977
fbshipit-source-id: d480d212474bd80aeac9cd9bb901f97562b62b13
