Summary:
This is a slight behavior change, but I need queue events published to
ActivityBuffer so I can track the live set of fetches.
The impact will be that `eden trace hg --retroactive` will show 2/3 of
the number of events that it did. We can bump the capacity if that's
problematic.
Reviewed By: xavierd
Differential Revision: D44594520
fbshipit-source-id: 5b22169403402d729ff929b4b4bec6bb76d54737
Summary:
ActivityBuffer is enabled by default and we rely on `eden trace hg
--retroactive` working. For HgQueuedBackingStore, unconditionally
enable ActivityBuffer, and give a capacity config setting independent
from the others.
Reviewed By: xavierd
Differential Revision: D44593656
fbshipit-source-id: 284bad0d19315d250a4ab69f06439393dbeaf5d1
Summary: We only capitalize type names.
Reviewed By: xavierd
Differential Revision: D44593407
fbshipit-source-id: 9f0d841e6ff5c40d6a6c6ca3637a40fefcf5d084
Summary:
While reading through our uses of ActivityBuffer, I noticed an
opportunity to amortize the cost of publishing import events.
Add some comments to note that.
Reviewed By: xavierd
Differential Revision: D44643270
fbshipit-source-id: 1b68f904b86d91fab91af01a4e8b169dc381ad33
Summary:
Now that EdenFS can fetch blob metadata from the server, let's make sure to
plug it to the `eden debug blobmeta` command.
Reviewed By: kmancini
Differential Revision: D44186943
fbshipit-source-id: a64f1384cf312e3c677505c330cfc82469fb83f3
Summary:
Now that the Mercurial backingstore knows how to fetch aux data, let's thread
this through the ObjectStore.
Reviewed By: kmancini
Differential Revision: D44110102
fbshipit-source-id: c57da05066d80fee199e45b4a4223168a196e3de
Summary:
Currently a single eden fetch causes 13 attempts to fetch the data.
By default hg hs 3 configured retries for edenapi fetch operations.
In the direct rust fetching code we have disabled these retries because we
use the importer as a retry, so these are not worth it.
We should disable them in the importer too.
There are still many retries. The importer internally will attempt 2 retries
and eden retries the importer once when it fails. So there are at least 5
attempts to fetch data still.
Those other retries are perhaps ensuring correctness, so we need to be more
careful with them. But these should be safe to remove.
This is a 3x reduction in retry attempts.
Reviewed By: fanzeyi
Differential Revision: D44482325
fbshipit-source-id: 96a501c8e9421e5d8aef92b697a6ed5e19c69194
Summary:
DurationScope has a pair of atomic reference count operations at the
beginning and end of every recorded duration. To avoid that overhead,
reference EdenStats with RefPtr and, in production EdenFS, store a
global, unowned EdenStats object.
This gives us the benefits of explicit dependencies and no global
state in tests, while avoiding atomic RMWs in the release build.
Reviewed By: xavierd
Differential Revision: D44323723
fbshipit-source-id: 1b3384d2e6a0a2959fd79774a8ba46afc4c176ca
Summary:
ObjectStore.h includes too many headers. Replace several with forward
declarations.
Reviewed By: xavierd
Differential Revision: D44323736
fbshipit-source-id: 944cd72fd758df4363493f87a573d91245a45b6a
Summary:
As of the libgit2 upgrade from 0.27.2 to 0.28.5 in D43195307, asan has been complaining about memory leaks coming from libgit2.
It initially caused 468 test failures.
D44096350 fixed 403 of these tests, but 65 are still failing.
Running one of these, we can see two other memory leaks (see test plan) being higlighted.
Mute them.
Differential Revision: D44254572
fbshipit-source-id: 8ae49c49542dda954f66de18cb756709d0c624d3
Summary:
As of the `libgit2` upgrade from `0.27.2` to `0.28.5` in D43195307, `asan` has been complaining about memory leaks coming from `libgit2`.
This caused 468 tests failures as reported in this task: T147770506.
An initial investigation hasn't quickly led to a resolution of the source of the memory leaks.
Get the tests back to green by suppressing the memory leaks detection from ASAN.
AddressSanitizer: 480 byte(s) leaked in 4 allocation(s).
/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/testcase.py:92: === clean up done at 93.718s (+10.623s)
ERROR
======================================================================
ERROR: test_eden_list (eden.integration.rc_test.RCTestNFSGit)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/fbcode/platform010/lib/python3.8/unittest/async_case.py", line 72, in _callCleanup
self._callMaybeAsync(function, *args, **kwargs)
File "/usr/local/fbcode/platform010/lib/python3.8/unittest/async_case.py", line 84, in _callMaybeAsync
ret = func(*args, **kwargs)
File "/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/testcase.py", line 158, in <lambda>
self.addCleanup(lambda: self.eden.cleanup())
File "/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/edenclient.py", line 134, in cleanup
self.kill()
File "/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/edenclient.py", line 143, in kill
self.shutdown()
File "/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/edenclient.py", line 434, in shutdown
raise Exception(
Exception: eden exited unsuccessfully with status 1
----------------------------------------------------------------------
Ran 1 test in 93.725s
FAILED (errors=1)
Exception: eden exited unsuccessfully with status 1
File "/usr/local/fbcode/platform010/lib/python3.8/unittest/case.py", line 60, in testPartExecutor
yield
File "/usr/local/fbcode/platform010/lib/python3.8/unittest/case.py", line 717, in doCleanups
self._callCleanup(function, *args, **kwargs)
File "/usr/local/fbcode/platform010/lib/python3.8/unittest/async_case.py", line 72, in _callCleanup
self._callMaybeAsync(function, *args, **kwargs)
File "/usr/local/fbcode/platform010/lib/python3.8/unittest/async_case.py", line 84, in _callMaybeAsync
ret = func(*args, **kwargs)
File "/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/testcase.py", line 158, in <lambda>
self.addCleanup(lambda: self.eden.cleanup())
File "/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/edenclient.py", line 134, in cleanup
self.kill()
File "/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/edenclient.py", line 143, in kill
self.shutdown()
File "/data/sandcastle/boxes/fbsource/buck-out/v2/gen/fbcode/94ab8015be307995/eden/integration/__rc__/rc#link-tree/eden/integration/lib/edenclient.py", line 434, in shutdown
raise Exception(
Buck UI: https://www.internalfb.com/buck2/947a3357-9a80-4a15-b067-fbad6a615fd0
Test UI: https://www.internalfb.com/intern/testinfra/testrun/3096224848534397
RE: reSessionID-765a77c0-c485-4182-be99-f7431d285752 Up: 312 KiB Down: 498 KiB
Jobs completed: 31. Time elapsed: 109.0s. Cache hits: 100%. Commands: 2 (cached: 2, remote: 0, local: 0)
Tests finished: Pass 4. Fail 4. Fatal 0. Skip 0. 0 builds failed
4 TESTS FAILED
✗ fbcode//eden/integration:rc - test_unmount_rmdir (eden.integration.rc_test.RCTestNFSGit)
✗ fbcode//eden/integration:rc - test_eden_list (eden.integration.rc_test.RCTestGit)
✗ fbcode//eden/integration:rc - test_unmount_rmdir (eden.integration.rc_test.RCTestGit)
✗ fbcode//eden/integration:rc - test_eden_list (eden.integration.rc_test.RCTestNFSGit)B
```
After:
```
Buck UI: https://www.internalfb.com/buck2/7091ae16-eb5f-425d-8922-cf278987acb3
Test UI: https://www.internalfb.com/intern/testinfra/testrun/3940649832205800
RE: reSessionID-a06bbda7-b3c4-46c6-998f-7e2280bb7ad8 Up: 558 B Down: 0 B
Jobs completed: 11. Time elapsed: 113.5s.
Tests finished: Pass 8. Fail 0. Fatal 0. Skip 0. 0 builds failed
```
Reviewed By: kmancini
Differential Revision: D44096350
fbshipit-source-id: 1c8a5308759f2dc90e00b33b8f4e19f2ca53f54f
Summary:
Nothing is being enqueued just yet, but this teaches EdenFS how to fetch
batches of aux data.
In the case where the aux data cannot be fetched, a nullptr auxdata is returned
to the higher level which will then trigger a blob fetch to compute aux data.
Reviewed By: chadaustin
Differential Revision: D44110104
fbshipit-source-id: 53df1496addf3a9dae521ffcdba5b060b8fce16a
Summary:
As a first step towards enabling aux data fetching, we need to allow aux data
to be batch fetched.
Reviewed By: chadaustin
Differential Revision: D44110105
fbshipit-source-id: f9c3f55ffdc5554784b5f38457ef675689664263
Summary:
Currently, EdenFS doesn't perform any aux data fetching, instead it fallback to
fetching blobs. In some cases, this can be extremely expensive when the aux
data store is emptied/flushed/rotated leading to non deterministic performance.
On Windows in particular, directory listing always need blob sizes, and thus
getting aux data reliably is critical.
As a first step, let's add some watches through the stack and expose these to
Thrift. This will allow `eden top` and `eden trace hg` to display these
fetches correctly.
Reviewed By: chadaustin
Differential Revision: D44105497
fbshipit-source-id: a3dc5cce1bc3115a2a4effcece6fa0cf0b16f6c8
Summary: By using reference everywhere, we can save on several moves.
Reviewed By: chadaustin
Differential Revision: D44101377
fbshipit-source-id: 9aa9d89570fb0432477e632d108109d292fa63bb
Summary:
Understanding how many trees and blobs were accessed but not fetched during
checkout is a useful metric that informs us as to how much was served from
caches. This is useful as checkout is more or less proportional in time to the
number of accessed inodes and the number of accessed trees+blobs will thus tell
us how many TreeInode and FileInode were loaded and traversed.
Reviewed By: chadaustin
Differential Revision: D44117153
fbshipit-source-id: 56991c51f2e4f501486d64ab5598f149f2708b77
Summary: This is no longer used, remove it.
Reviewed By: chadaustin
Differential Revision: D44073360
fbshipit-source-id: b8d1c6228718d8e0b58dc39ab2b70ef0f0cab192
Summary:
A Hash20 could be anything, while a HgProxyHash is well defined. Use the latter
to clear some confusion and to be consistent with the Tree and Blob functions.
Reviewed By: genevievehelsel
Differential Revision: D44046388
fbshipit-source-id: 38dd08cd1c321145facc7f660aacccc3198a219b
Summary:
The Sapling backing store now provides both size and sha1 in the TreeEntry,
thus we can pre-compute the metadata and store it in the LocalStore to avoid a
roundtrip to the Saplig backing store when that metadata will be needed very
shortly.
Reviewed By: kmancini
Differential Revision: D43132760
fbshipit-source-id: 38dcba76d41e5801c8b716197af179543386df08
Summary:
All of these functions are returning an ImmediateFuture already, so there is no
need to construct another ImmediateFuture on top.
Reviewed By: mshroyer, fanzeyi
Differential Revision: D42788619
fbshipit-source-id: b40cd1027d183920e858cb21c91ee7b14d2ea929
Summary:
Some times EdenFS has corrupt data, its helpful to be able to see where that
corrupt data is stored. I updated `eden debug blob` a while back to show
blob data from multiple places: D41165544 (224cde1d5c).
In this stack I am going to do the same for blob metadata, so we can
understand where corrupt data is comming from.
This diff implements that new endpoint and adds some tests for it.
Reviewed By: mshroyer
Differential Revision: D42283193
fbshipit-source-id: 8f5f684176d2a905c74c8d3add19abd57c4599ad
Summary: This diff adds `TaskTraceBlock` to globbing code so we can understand which thread are these code being executed on.
Reviewed By: chadaustin
Differential Revision: D41527107
fbshipit-source-id: a001f473795e413d2bf4a8125389867a1629515b
Summary:
Google Benchmark automatically measures iteration time, so there's no
need to manually measure the cost of an operation. Instead, we can
just lift all of the setup work out of the benchmark iteration loop.
This diff also removes use of divisions and floats in the benchmark
loop.
Reviewed By: xavierd
Differential Revision: D42180588
fbshipit-source-id: d51621ed1057acd26e92342c28dfa9f10b0c11be
Summary:
Most of the time, FaultInjector is a no-op, so it's a perfect
candidate for ImmediateFuture.
Reviewed By: xavierd
Differential Revision: D42086041
fbshipit-source-id: e93f0c707a8d8f42be967b2b394d1724a0b83649
Summary:
For parity with Future and SemiFuture, remove ImmediateFuture's
default constructor and replace it with `makeEmpty()` and a
`std::in_place` constructor overload.
Reviewed By: xavierd
Differential Revision: D41293865
fbshipit-source-id: 43abae81fed3d99a5b4124d8c598dcce921ef65b
Summary:
I'm about to introduce an interface named ConfigSource, which
awkwardly conflicts with this enum. Rename ConfigSource to
ConfigSourceType. I considered ConfigSourceSlot, but slot seems more
of an implementation detail.
Reviewed By: kmancini
Differential Revision: D41830066
fbshipit-source-id: 85a54f4243b14c693087c7555e27b5473e90562e
Summary: EdenConfig had some remnants of old cruft. Remove them.
Reviewed By: xavierd
Differential Revision: D41744767
fbshipit-source-id: d5fd50fa599c15ec8b748c323c45ea1e7eb8ddcd
Summary:
Obtaining the SHA1 of a blob while cheap can start becoming very expensive when
tons of them are requested. Let's do a cheap ObjectID comparison first to avoid
them.
Reviewed By: chadaustin
Differential Revision: D41660610
fbshipit-source-id: 192e9d51cf1eeba3ea3148b90adda685fe7d251f
Summary:
Exception safety has costs. Namely, exception-safe code needs to
support rolling back, and reasoning about correctness is
hard. Therefore, now that our Path and PathPiece types are
noexcept-move, have ImmediateFuture require the type that it holds be
noexcept-move. Types that aren't noexcept-move can be passed by
unique_ptr.
Reviewed By: kmancini
Differential Revision: D41293189
fbshipit-source-id: 770bd4ac7818256d83f8ff50f27c6977147b8209
Summary:
EdenFS's tests still fall back on a debugedenimporthelper process to
import trees and blobs. That's because the test repos still use the
deprecated revlog format. Migrate the test repos to use indexedlog
storage so that SaplingNativeBackingStore hits.
Note: This repo configuration is awkward and temporary, and the
Sapling team is planning to migrate to "Eager" repos to support the
`hg init` case in the future.
Reviewed By: quark-zju
Differential Revision: D41447075
fbshipit-source-id: 23c2f13dfc9f964d9386d7f4c920222db8f25144
Summary:
get_tree and get_blob were misleading counters. They did not include
reads from disk. In preparation for adding encompassing get_tree and
get_blob counters, rename to fetch_tree and fetch_blob.
Reviewed By: xavierd
Differential Revision: D41566434
fbshipit-source-id: eae98035f4d0c366f58c6c22eda42ae5c537c8e4
Summary:
EdenFS's C++ unit tests unnecessarily differed in how they initialized
test repositories. Move
scmstore/treemanifest/remotefilelog/etc. configuration into hgInit so
it's the same everywhere.
Reviewed By: genevievehelsel
Differential Revision: D41364290
fbshipit-source-id: 24f882766bf00c6546663e0712b6029153a4a699
Summary:
In preparation for removing EdenFS's fall back onto hg importer tree
and blob fetches, add config knobs that allow us to slowly roll out
the behavior change across platforms.
Reviewed By: xavierd
Differential Revision: D41346959
fbshipit-source-id: a177b0fefd5d471951435a5254318306091630d6
Summary: Small refactorings to reduce noise in further diffs.
Reviewed By: genevievehelsel
Differential Revision: D41346800
fbshipit-source-id: 6824487f91f811f040cd51c9b41c0227f49fdd49
Summary:
I'm standardizing all of our *Store classes on the following method
order: tree, blob, blob metadata.
This diff has no behavior changes, and only moves code.
Reviewed By: genevievehelsel
Differential Revision: D41346763
fbshipit-source-id: bb5f5fb63c9cfccdffff90c404d677c4cf751d40
Summary:
The previous SaplingNativeBackingStore API dropped errors on the
floor. But it would be useful to propagate them outwards, so return a
Try and ensure the callback is invoked for every requested node.
Reviewed By: xavierd
Differential Revision: D41328023
fbshipit-source-id: e5e5bbe26aff16e99efe8a8c3ff6d020d3691615
Summary:
Benchmarks for individual subsystems generally live next to the
tests. `eden/benchmarks` is primarily integration-level benchmarks, so
move the HgImportRequestQueue benchmarks next to the tests.
Reviewed By: genevievehelsel
Differential Revision: D41568442
fbshipit-source-id: a5278a9dc0b7310bd409e9ab64d05d05c7917701
Summary:
We've had a couple of failures after landing D40818724 (f199e93924) due to some places still
building AbsolutePath from a literal. On Windows, this is almost always a bug
due to paths usually not being UNC.
To avoid this issue, we can simply forbid an AbsolutePath from being built in
the first place from a path, instead forcing the use of canonicalPath. This was
suggeted by mshroyer in the orignal diff.
Reviewed By: chadaustin
Differential Revision: D41229194
fbshipit-source-id: 3451bdcba276c87f98326b025e05f6a4eddbd1b7
Summary:
This is follow-on from D41033644. Remove the name parameter from our
FFI bindings, since it is no longer necessary. In the future, if paths
are required again, we can store them in a variable-width node
component.
Reviewed By: genevievehelsel
Differential Revision: D41034415
fbshipit-source-id: a279e01d447d9f509e283f09453b831e0ec5cedd
Summary:
All of the lambdas involved are called within the outer
scope. Therefore, the captures should be implicit and by
reference. This avoids a vector copy per batch.
Reviewed By: xavierd
Differential Revision: D41034079
fbshipit-source-id: 057e397c1c98f619599472af46ddcedbc15a2288
Summary:
On the hg side, let's consistently use the namespace sapling, and
instead of using the word "Rust" in the API, let's call them bindings.
Reviewed By: xavierd
Differential Revision: D40959727
fbshipit-source-id: 687c8c78fd69e2478dfbf5b0573c3bccec340ce3
Summary:
Instead of introducing our binding types in the global namespace with
the prefix "Rust", use a sapling namespace. The fact that the code is
in Rust is an implementation detail: the important thing is that these
types are on the other side of the project boundary.
Reviewed By: genevievehelsel
Differential Revision: D40925131
fbshipit-source-id: 85c25a0e989e9267625459a355d3739ff8686638
Summary:
std::string_view has noexcept accessors and folly::Range doesn't, so
this allows us to make Path and PathPiece noexcept.
Reviewed By: kmancini
Differential Revision: D41145426
fbshipit-source-id: 046f6f6a532d8d0da8508ccf7896c914e19a25ec
Summary:
We no longer need a kill switch on edenapi fetches. Stop accessing the
config and always configure backing stores with edenapi on.
Reviewed By: genevievehelsel
Differential Revision: D40909096
fbshipit-source-id: fbdb01fc74add870b95d0bcc46be6b4bf9323898
