Summary:
Previously, fetch heavy event's cmdline was delimited by '\x00' when logged to Scuba. (for example: `grep--color=auto-rtest.`)
Now we replace \x00 with a space, so command name and args will be separated by space. ( `grep --color=auto -r test .` )
Reviewed By: kmancini
Differential Revision: D22772868
fbshipit-source-id: 4ab42e78c7bc786767eee3413b9586739a12e8ac
Summary:
This adds logging for files fetched in prefetch like was aleady added for
blob and tree fetches.
This is needed to log the fetches caused by the glob files thrift call. The
purpose of this to help debug the cause of unexpected data fetches (See
D22448048 for more motivation).
Reviewed By: genevievehelsel
Differential Revision: D22561619
fbshipit-source-id: 5ae78b99fb0c7d863d8223b93492b0d0210ddf9e
Summary:
This adds logging for data fetches that come from the thrift checkout call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: chadaustin
Differential Revision: D22489504
fbshipit-source-id: 3b732a1e5627c2130f561ec0138a1df270e1925d
Summary:
We have seen that some of the unexpected data fetches do not originate from
FUSE. This adds parity to the logging for data fetches that come from the thrift
interface. Adding this logging improves the overall observability of eden, and
will help us debug the cause of unexpected data fetching.
This introduces plumbing to allow logging data fetches that originate from
thrift requests.
Reviewed By: chadaustin
Differential Revision: D22448048
fbshipit-source-id: a39dde72467c4922c07c569c14fb499341d40258
Summary: Previously, `BackingStore` and all its sub-classes' `getBlob` and `getTree` methods accepted both `ObjectFetchContext` and `ImportPriority` as arguments. Now, `ImportPriority` is removed because we can get the priority from `ObjectFetchContext `
Reviewed By: kmancini
Differential Revision: D22650629
fbshipit-source-id: e1b0c57a059f11504b28b2c17d698bb58f51e1ee
Summary:
check fetch count before `getPriority()` is used. If fetch count has exceeded `fetchThreshold_`, lower the priority by 1.
Note: this diff only guarantees that the `getPriority()` function is returning the lowered priority. How the returned value is used for scheduling is handled by `HgQueuedBackingStore`
Reviewed By: kmancini
Differential Revision: D22550640
fbshipit-source-id: c032f8f72ca658618ac118dfb3ad3dcae61e9735
Summary: Previously `getPriority()` always returned a fixed priority. Now that we want `ObjectFetchContext` to lower its priority, it is better to make it have its own copy of `ImportPriority`.
Reviewed By: kmancini
Differential Revision: D22550587
fbshipit-source-id: 029c797def477ae4533f66cfba146a3972cfb65d
Summary: This diff ensures that the value of ImportPriority is always positive and changes offset from 0 to half of the maximum offset to allow lowering offset in the future.
Reviewed By: kmancini
Differential Revision: D22550462
fbshipit-source-id: 69f45369359c7b2c39a0c6831e9b33982e26a16a
Summary: add a thrift call to clear `pidFetchCount_` in `ObjectStore` and call it in `eden debug gc_process_fetch`. Users might want this command to start a new recording of process fetch counts.
Reviewed By: kmancini
Differential Revision: D22583430
fbshipit-source-id: eba7d63b08da5134fd09b7512895aba06f6a7ca5
Summary:
lower the debug level for scs proxy hash logging so that this does not cause so
much noise in the logs of production eden
Reviewed By: chadaustin
Differential Revision: D22668574
fbshipit-source-id: 1a7c2a4706514c0ef0bb848424681eef9316d296
Summary:
We are unifying C++ APIs for accessing optional and unqualified fields:
https://fb.workplace.com/groups/1730279463893632/permalink/2541675446087359/.
This diff migrates code from accessing data members generated from unqualified
Thrift fields directly to the `field_ref` API, i.e. replacing
```
thrift_obj.field
```
with
```
*thrift_obj.field_ref()
```
The `_ref` suffixes will be removed in the future once data members are private
and names can be reclaimed.
The output of this codemod has been reviewed in D20039637.
The new API is documented in
https://our.intern.facebook.com/intern/wiki/Thrift/FieldAccess/.
drop-conflicts
Reviewed By: yfeldblum
Differential Revision: D22631599
fbshipit-source-id: 9bfcaeb636f34a32fd871c7cd6a2db4a7ace30bf
Summary:
Replace calls to Future<T>::getTry() with Future<T>::result().
This change is behaviour-neutral. It enables us to make the behavior of getTry
match expectations and be blocking, like get, and r-value qualfiied, like
SemiFuture<T>::getTry().
Reviewed By: chadaustin
Differential Revision: D22510034
fbshipit-source-id: bd45cc6a404293089371654829a63c81b1c706aa
Summary:
Buck uses the content SHA-1 to identify each of the source files for a target.
During the parsing phase it needs these SHAs, though the content of the
files is not yet needed, and may never be needed if the file has already
been built and is in the buck cache.
Currently, if we do not already have metadata cached for a file when
requested we fetch the contents of the file, and compute the hash.
We want to avoid this.
Eventually this data will be available from the Mononoke EdenAPI server,
but for now we want a temporary solution to unblock the Buck team, and
ship benefits early.
Reviewed By: chadaustin
Differential Revision: D21820913
fbshipit-source-id: 56a7e32519f0fb04881518306d94aaed33527fd9
Summary:
Prefetching metadata for the entries in a tree when we fetch it saves us
an extra round trip to the server to fetch a blob when only the metadata
for that blob is fetched. (This can happen often while parsing targets in
builds)
We will to prefetch the metadata for each of the entries in a tree when
we fetch the tree and store the metadata for each entry under that
entries id (to make looking up the entry metadata by its id quick)
However, we also don't want to unnecessarily fetch data from
the server if we already done so.
To accomplish this we will also store the metadata for each entry under the tree
id in the local store. This will: 1) allow us to check if we have already fetched
the metadata from the server when we are fetching a tree (we only have the
tree id easily available here to storing the metadata under the tree id makes
it much easier/less expensive to do this check). 2) allow us to refil the
metadata for each entry stored under that entries blob id if it has been cleared
from the local store (this may happen is the local store is gets large and gets
partially cleaned to reclaim space).
This implements the method to store tree metadata for all entries under the tree
id and under the blob id for each entry.
Reviewed By: chadaustin
Differential Revision: D22239173
fbshipit-source-id: d4e0ffd642ce0b4034188cfc4eeaf2ea05f54e77
Summary:
This will allow adding custom MetadataImporters in different eden builds.
DefaultMetadataImporter provides a no-op version of the interface to be
used by default.
Reviewed By: chadaustin
Differential Revision: D21960834
fbshipit-source-id: aec8a3627ab1223f74466b92a0ebe3290b67b7ed
Summary:
Previously the BackingStore kept a raw pointer to the LocalStore. To do this we relied on EdenServer ensuring the lifetime of the LocalStore exceeds that of the BackingStore.
This makes the LocalStore pointer a shared pointer to explictly make sure that the LocalStores lifetime matches the BackingStores lifetime.
Reviewed By: chadaustin
Differential Revision: D22394597
fbshipit-source-id: c81cb26c6fc8f834bc46d8576ced06ba6a96ac2c
Summary:
This introduces a class to manipulate the metadata for all the entries in a
tree. This adds serialization and deserialization to this class so that it can
be written to the local store.
Why do we need this? We need some way to easily check when we have already
fetched metadata for a tree and do not need to refetch this from the server to
avoid expensive network requests. Later diffs add functionally to store the metadata
for tree entries in the local store under the tree hash using this class.
Reviewed By: chadaustin
Differential Revision: D21959015
fbshipit-source-id: 0c0e8750737f3076c1f9604d0319cab7f2658656
Summary:
In following diffs we will use scs to prefetch meta-data for files, so that this data
will be available with out fetching the file content (which will improve build times
on eden).
This builds up the proxy hash index that serves as a conversion between eden
specific identifiers and commit and path which we will use to index into scs.
Reviewed By: chadaustin
Differential Revision: D21820909
fbshipit-source-id: 17891f6772f49c7c183061d7a4df2fe0a3be9d25
Summary:
In following diffs we will use scs to prefetch meta-data for files, so that this data
will be available with out fetching the file content (which will improve build times
on eden).
SCS indexes trees by an scs specific hash (blake2 content hash) or by the commit
hash and path. Since this is different from the eden hashes and mercurial
hashes, we need another index to go between the current ids we have in eden
and identifiers for scs.
This introduces a proxy hash that serves as this conversion. Because we have
commit hashes around in eden right now, this is an easier route to indexing
into scs currently.
Reviewed By: chadaustin
Differential Revision: D21237648
fbshipit-source-id: 79115ac034a5f062ae879713cd2c1a17f348c725
Summary:
proxy_import_helper.py exists for compatibility with older EdenFS
builds. None of those builds are running anymore, so remove it.
Reviewed By: genevievehelsel
Differential Revision: D22451196
fbshipit-source-id: 4d258b3fafe13bb67bd11259f5d1193a7e5575e6
Summary:
From the Rocks DB documentation:
> When opening a DB in a read-write mode, you need to specify all Column
Families that currently exist in a DB. If that's not the case, DB::Open call
will return Status::InvalidArgument()
This can cause problems for us in a couple of situations:
- When we need to rollback from an eden version where we added a column to
our configuration for RocksDB
- When we delete a column from our configuration for RocksDB
To make sure we do not encounter this error we need to make sure that we still
open all the columns existing in the database, even if they are not in our
configured list of family columns.
Reviewed By: wez
Differential Revision: D22425310
fbshipit-source-id: 9822b22cfedf4633f65bbed96f95a546dd3614f4
Summary:
This is much better than having `ObjectFetchContext` itself owns a copy of `ImportPriority`. We can actually customize how different fetch context manages these priority.
We set all FUSE requests to a higher priority and prefetch requests to a lower priority
Reviewed By: xavierd
Differential Revision: D22342802
fbshipit-source-id: b9c1d0f2ddbc7a5e5d619bc2c2222e5df0e702af
Summary:
This commit adds `ImportPriority` to `ObjectFetchContext`. By doing so we can tweak priority for a request at different stage.
This commit also provides a default implementation for the virtual methods in `ObjectFetchContext` so we can create one to carry `ImportPriority` in some specific cases.
Reviewed By: chadaustin
Differential Revision: D21872718
fbshipit-source-id: 6e8cfd84959b368e6fe69fda2baf0debf7a88295
Summary: This diff made fetch threshold configurable, so we can change it later as repository size grows.
Reviewed By: fanzeyi
Differential Revision: D22337850
fbshipit-source-id: 4b46420cb4e7164a3f1080279d67fa5f90549cd8
Summary: This diff updated `ObjectStore` to send a `FetchHeavy` event to Scuba when the number of fetching requests of a process has reached 2000.
Reviewed By: fanzeyi
Differential Revision: D22292104
fbshipit-source-id: b7ac48412868216ea960c8681a5fb71c587952bc
Summary:
Both optional and pid_t weren't found and the right includes needed to be
provided. On Windows, the ProcessNameCache isn't compiled (yet), and since it
looks like the process name is optional in the BackingStoreLogger, let's not
provide it for now.
Reviewed By: fanzeyi
Differential Revision: D22215581
fbshipit-source-id: 31a7e7be62cd3d14108dc437d3dfabfb9e62f8d5
Summary:
Eden can sometimes unexpectedly fetch files from the server, and we want
to know why this is happening. This adds logging for the source of
data fetching in edens backing store to help obviate why these fetches
are happening.
This temporarily adds the logging in the HgQueuedBacking store to get a naive
version of logging rolled out sooner. Follow up changes will move this logging
closer to the data fetching itself if possible (in HgDatapackStore and HgImporter).
Reviewed By: chadaustin
Differential Revision: D22012572
fbshipit-source-id: b1b012ce4ee133fbacecd586b7365c3c5a5386df
Summary:
We have seen that eden will unexpectedly fetch data, we want to know why.
This adds the plumbing to interact with edens current logging to be able to
log when eden fetches data from the server and what caused eden to do this
fetch. Later changes will use the classes created here to log the cause of data
fetches.
Reviewed By: chadaustin
Differential Revision: D22051013
fbshipit-source-id: 27d377d7057e66f3e7a304cd7004f8aa44f8ba62
Summary:
Recently the server team added an un-used directory to test that eden would not
fetch these as a test for the upcoming repo merge. They saw that these files
were fetched a non trivial number of times. We want to know why eden is causing
these fetches.
This adds the pid and interface through which the client is interacting with eden to
ObjectFetchContext for this purpose. This information will be logged in later
changes.
note: currently this is only for fetches through Fuse (thrift interface to follow).
Reviewed By: chadaustin
Differential Revision: D22050919
fbshipit-source-id: 49b93257a0e6d910f48b1e8ec6471527e056d22a
Summary:
This passes ObjectFetchContext into the backing store to prepare for adding
logging for the cause of server fetches.
In following changes I will add logging in the HgQueuedBackingStore.
Ultimately we will want to move this logging to be closer to the data fetching
(in HgDatapackStore and HgImporter), but I plan to temporarily add logging to
the HgQueuedBackingStore to simplify so that we can more quickly roll out.
Reviewed By: chadaustin
Differential Revision: D22022992
fbshipit-source-id: ccb428458cbf7a1e33aaf9be9d0d766c45acedb3
Summary:
In following changes I will be threading ObjectFetchContext into the backing
store importing process, since this will start to be used more outside of the
ObjectStore, I am moving this class into its own files.
Reviewed By: chadaustin
Differential Revision: D22022488
fbshipit-source-id: 1a291fea6e0fd56855936962363dfc9f6de8533d
Summary: This diff adds a PID-fetchCounts map to `ObjectStore` and makes `ObjectStore` update that map after every `didFetch`
Reviewed By: kmancini
Differential Revision: D22100413
fbshipit-source-id: 740342c7b4a453fe482344c2db9542381c3772e4
Summary:
On Windows (haven't verified on other platforms), ThreadLocalPtr don't appear
to be releasing resources when a thread die. This means that when the importer
thread dies, the actual importer (hg.real) would still run and use resources,
with no way of talking to it.
To fix this, let's manually reset it when the main thread function returns,
this forces the importer to be destroyed and therefore the various handles to
hg.real to be released, effectively terminating it.
I'm not sure if this is the proper fix, but delving into folly feels a bit
daunting. Keeping a TODO for later to go back to it and fix it properly in
folly.
Reviewed By: chadaustin
Differential Revision: D22012540
fbshipit-source-id: 99f994bb5128b38ccf8474031763b8a21055759a
Summary:
Previously we check if a request is a fuse request when we fetch anything from backing store, so we can collect number of fetches happened for each process in eden top.
This is creating a dependency from store to fuse, which is a little awkward. Instead, we could make `RequestData` a `ObjectFetchContext` and record the fetches when that happens.
Similarly in the future we should also have something equivalent in our Thrift layer.
Reviewed By: kmancini
Differential Revision: D21775919
fbshipit-source-id: 95056830ddbe7c999051c43e0d8eca9a67350904
Summary:
NOTE: This stack works together, they are separated for easier reviewing. Check D21723465 if you want to read this stack in one place.
This diff finally connects EdenAPI with EdenFS. Previously we only had basic EdenAPI implemented but the performance was not acceptable. This is largely due to overheads in talking with the remote server. Now EdenFS is able to import multiple files at once, we efficiently use EdenAPI to import these data now.
Reviewed By: chadaustin
Differential Revision: D21576569
fbshipit-source-id: a45e832ec63d057730138551393ff7547fa2c22f
Summary:
The CreateProcess API allows inheritable handles to be available in the spawned
process by passing a flag to it. What the documentation for this API leave to
the imagination is what happens when processes are spawned concurrently in
multiple threads and handles are opened and made inheritable while doing this.
The answer is obvious but the consequences aren't: they are inherited.
When anonymous Pipes are used, one end needs to be inheritable for the spawned
process to being able use it, but if one is created concurrently with spawning
a process, that other process may have an open handle to that unrelated pipe.
And since to detect that a pipe is closed, all handles to the other end needs
to be closed, this can lead to never being able to detect that it is closed...
This scenario is exactly what is happening in eden when spawning the Mercurial
process, and when one of these processes would die, eden would just hang trying
to write to the pipe, not knowing that the process was already gone. To unblock
eden, all the hg.real processes had to be killed, as this would close all the
pipe handles, and then eden would detect that the pipe was closed and re-spawn
Mercurial (only for the exact same thing to happen).
In order to fix this, we need to tell CreateProcess to only inherit the pipes
handles, and nothing else.
Reviewed By: wez
Differential Revision: D21895537
fbshipit-source-id: 3c84a1d0316b50b5750f554fa20f72f59a718882
Summary: This function won't go ask remote server and is able to produce a result very quick. This avoids the waiting time in the queue.
Reviewed By: chadaustin
Differential Revision: D21586511
fbshipit-source-id: db3793cc4e4e493d1beb6837c0bd91358e7e845f
Summary: Reading from hgcache directly has been rolled out for quite a while now. Let's remove the configuration and make it default behavior.
Reviewed By: chadaustin
Differential Revision: D21585482
fbshipit-source-id: 7f30e262642adf58388a0580aa5a63595fa89155
Summary:
gcc and clang disagree here, so explicitly construct a Future from the
unique_ptr.
Reviewed By: genevievehelsel
Differential Revision: D21687654
fbshipit-source-id: 338174f5d4d051f509faace6e48d1c0fb4d898e1
Summary: Fix some RVO warnings and a compilation error when compiling with gcc.
Reviewed By: xavierd
Differential Revision: D21481738
fbshipit-source-id: 0621f5886df40c24ef1a6a68ccd957e38f2f4122
Summary: Now that Rust libraries can be linked into fbcode binaries with mode/mac, enable Rust datapack on all platforms.
Reviewed By: xavierd
Differential Revision: D21241075
fbshipit-source-id: ae2abee25a5ad7b9db9ac5e6b8687c5f79376926
Summary:
While this isn't the right fix, this is what shipped in our packages, for the
sake of being able to reproduce the package, let's land this as it is. A
future change will remove this ifdef.
Below is pkaush original description:
In Eden Windows we treat all the files as regular files and don't have a
concept of symlinks and executable files. Fixing the TreeEntryType::getType()
to return REGULAR_FILE for executable file and symlink.
Reviewed By: wez
Differential Revision: D20481051
fbshipit-source-id: 0b0c4d7aea28134383ef45aeafc02930b420286b
Summary:
These 3 tests compile without issues on Windows. The RocksDB one is weird,
while it compiles with no hickups, I simply cannot run the resulting test
binary, and I'm not sure how to debug this. The local store one fails in folly.
Reviewed By: chadaustin
Differential Revision: D21393724
fbshipit-source-id: db90bf20a9d116bc8aa493703997c5e8da76eb1f
Summary:
vs2017 is not able to compile the static assertion in KeySpace.cpp.
Previously we thought that this would be resolved in a later release of vs2017
but now that is here it is clear that it hasn't been fixed.
This commit pushes the version requirement to vs2019 (see
https://dev.to/yumetodo/list-of-mscver-and-mscfullver-8nd for a mapping between
product versions and compiler versions), but we cannot build with vs2019
because folly and rangev3 don't compile with vs2019, so this assertion (heh!)
has literally not been tested.
This commit also fixes up an oversight in the gating logic: the intent is that
we perform the assertion on all systems except known broken MSVC. We were
accidentally restricting it to later versions of MSVC.
Reviewed By: simpkins
Differential Revision: D21432890
fbshipit-source-id: e11ffccc53bf8dffdf2db45ad4f3cf199b1cc70d
Summary: Instead of having `HgBackingStore` fetch blobs separately, we now try to read from hgcache and fetch from `HgImporter` as batches.
Reviewed By: chadaustin
Differential Revision: D20903245
fbshipit-source-id: d8e404d6765f1bcbacbf2a39f83eab0a351a3fe0
Summary: This diff makes `HgImportRequestQueue` to be able to return multiple requests in the queue at once.
Reviewed By: chadaustin
Differential Revision: D20197070
fbshipit-source-id: 8cff1780d6e56321a756d30ac0e9b9d5d319c049
Summary: The life of a request is only able to be finished with one of the method. So we can instead having the tracker destroyed when the future is resolved.
Reviewed By: chadaustin
Differential Revision: D20995819
fbshipit-source-id: 5dac2f762513b5d0bcacaab7d0669fc8fdb61e80
Summary:
Readability improvements.
`HgQueuedBackingStore` will need to call these functions individually.
Reviewed By: chadaustin
Differential Revision: D20683321
fbshipit-source-id: 9a9bd766c34559048bd0971f17304090abbb2774
Summary:
During a graceful restart, while the fuse mount is not handling
requests, avoid a possible deadlock between the edenfs, hg
debugedenimporthelper, and watchman processes.
See the comment in HgImporter.cpp.
Reviewed By: fanzeyi
Differential Revision: D21342275
fbshipit-source-id: df8fb5df5d5cd1490e88b42054b34cbb2acdb692
Summary: So the linter will stop nagging at my changes to this file.
Reviewed By: wez
Differential Revision: D21396011
fbshipit-source-id: dbc3479637cca83aa1a11ff94a9033dfa42fc2a6
Summary:
Talked with xavierd last week and we can use LocalStore's `get_missing` to determine if a blob is present locally. In this way we can prevent the backingstore crate from accidentally asking EdenAPI for a blob, so better control at EdenFS level.
With this change, we can use this function at the time where a blob import request is created with confidence that this should be short cheap call.
This diff should not change any behavior or performance.
Reviewed By: xavierd
Differential Revision: D21391959
fbshipit-source-id: fd31687da1e048262cb4eae2974cab6d8915a76d
Summary:
In the initial stages of the windows port we had
problems building rocksdb on windows, so we disabled it.
These days we're able to build it and detect it--we even
require it in the cmake code, but hadn't gotten around
to telling the rest of the code that we can use it.
This commit re-enables it in the build but leaves sqlite
as the default engine until we're able to perform some
benchmarking.
Rocksdb itself has some build issues on Windows; it doesn't
use cmake to locate dependencies, so even though we built
snappy it doesn't know how to find it without modifying the
source:
https://github.com/facebook/rocksdb/blob/master/thirdparty.inc#L4
For that reason, we disable the use of Snappy in the Windows build.
However, in the version of rocksdb that we were using, it would
default to trying to use Snappy even though it wasn't compiled in
and throw an exception.
I've upgraded to a newer version of rocksdb that will simply not
use compression if no compression was enabled at build time.
Given that we mostly store relatively small objects, I'm assuming
that the lack of compression is fine for now.
Reviewed By: xavierd
Differential Revision: D21319896
fbshipit-source-id: 2a2d06d4bd5382706e9220f9b4a2de99dc18311d
Summary:
Move the `UserInfo` code from `fuse/privhelper` to `utils`, and also unify the
POSIX and Windows implementations of this class.
This code was originally put in the `fuse/privhelper` directory since it was
written at the same time as the privhelper. However, it's really a
lower-level library that doesn't depend on FUSE or any of the other code in
the `fuse/` subdirectory.
Reviewed By: wez
Differential Revision: D21296594
fbshipit-source-id: f58682f6ce86bba0328472c491bb4c0dc3370319
Summary:
This sets up the counters that will allow us to expose the number
of pending FUSE requests in Eden top.
As D20846826 mentions adding metrics for FUSE request gives
visibility into fuse requests and overall health of eden.
This provides more insight beyond the metrics for live FUSE requests
since it shows the kernels view of FUSE requests. Looking at the difference
between the number of pending and live request, can identify issues
that arise at the interface between eden and FUSE and monitor how
quickly fuse workers are processing requests.
**note**: this is only for linux since macos has no equivalent to
`/sys/fs/fuse/connections`
Reviewed By: chadaustin
Differential Revision: D21074489
fbshipit-source-id: c0951f0dfd4fa764be28d8686d08cd0dd807db37
Summary: On Windows we are seeing frequent hang issues while using telemetry hg binary. Switching to using the hg.real for all the platforms.
Reviewed By: simpkins
Differential Revision: D20480875
fbshipit-source-id: 82892da7f8d46f2413442c37251243400180f9db
Summary: Close the Pipe handles that are inherited in the child and are not needed in the parent process. This will also make sure the pipe is closed when the child process ends.
Reviewed By: simpkins
Differential Revision: D20923588
fbshipit-source-id: aa58e99148d2f5c3bf666a3237266eac725be563
Summary:
This exposes metrics for the live FUSE requests (the duration
of the longest outstanding request and the number of outstanding
requests).
Because FUSE is the interface through which the user mostly interacts
with the file system they provide good metrics to judge if the perfomance
of eden is normal, or there may be an issue.
Exposing these counters this way will send them to ods, so it will not only
allow for debuging current issues, but can be used to look back at previous
problems. This data could also be used for alerting or more proactive
remediation.
Metrics are exposed per checkout to allow seeing which checkout was
having issues. This data will aggregated in `eden top` to be used as
an overall health indicator, but should more information be needed it
will be logged in ods.
Reviewed By: chadaustin
Differential Revision: D20922194
fbshipit-source-id: 16208883417acb77b62bf712cfdd9068c5420303
Summary: It's a little weird to have get batch returns a pair of `RelativePath` and `Hash`.
Reviewed By: chadaustin
Differential Revision: D20844545
fbshipit-source-id: c0c823e0aca6ea2326cb637226dc0de43fd4f3cd
Summary: This diff makes `HgProxyHash` moveable & copyable so we can return it from `getBatch` method instead returning a pair of `RelativePath` & `Hash`. See next diff for usage.
Reviewed By: chadaustin
Differential Revision: D20841434
fbshipit-source-id: c082c217f75d53bcbfb4853e2055d1458a040188
Summary: gcc warned that these functions did not return with a value on all code paths.
Reviewed By: genevievehelsel
Differential Revision: D21004224
fbshipit-source-id: 3ccd38b19bb4f4c0dbe0b75b4d11f351b6690591
Summary:
This code isn't really working and is giving out corrupted file content. Let's remove it for reading blobs.
We still cannot remove the C++ mercurial code entirely as we rely on it to import manifests. I tested on switching to the Rust manifest implementation but it wasn't working due to new pack file scanning.
Reviewed By: chadaustin
Differential Revision: D20938001
fbshipit-source-id: 9cc7b1ed9cd8228afd6fc5374cda001d1309a379
Summary:
Since imports are now queued in `HgQueuedBackingStore`
the metrics for queued imports were broken
the queueing use to happen in `HgBackingStore`, but
now the queue happens before the imports get here,
in `HgQueuedBackingStore`, thus these metrics were
only measuring live imports
=> this moves the metrics for queued imports into
`HgQueuedBackingStore` so that they capture both
live and queued imports
**note**: since prefetching in `HgQueuedBackingStore`
is in progress, these metrics are only set-up for when
prefetching this is finished
Reviewed By: chadaustin
Differential Revision: D20714494
fbshipit-source-id: 392337aaa60b44b75184889e6c92cb5d2c2577cc
Summary:
As mentioned in D20629833, adding metrics for live
imports in `eden top` gives more transparency to the
imports process and makes identifying import related
issues easier. This is set up to expose metrics for live
imports like those for pending imports in `eden top`.
Similar to D20611728 exposing this via these counters
will log this data. Having this data persisted will allow
tracking the performance of imports, and does the set
up for more pro-active fixing of issues. Further we can
look back to see issues that are no longer occurring, but
still of interest.
This also refactors the registration code so that it requires
no copy pasting to add a new counter. Avoiding copy paste
errors when adding more counters and making it easier to
maintain.
Reviewed By: chadaustin
Differential Revision: D20630813
fbshipit-source-id: 8a7a2a0135c7b7a5cde960b84dcb434c6c99eaeb
Summary: This is actually missing from `HgQueuedBackingStore`. This diff fixes this and assign low priority to these prefetch requests.
Reviewed By: chadaustin
Differential Revision: D20655681
fbshipit-source-id: f3c92b358e16e980390ac7adcae27d41ae5a7277
Summary: Ergonomics improvements. This allows us to have type safety instead of dynamically check for types at runtime.
Reviewed By: chadaustin
Differential Revision: D20740318
fbshipit-source-id: d0a3255105bef22dd236695be9fc9c4220547b83
Summary:
Previously `eden top` shows metrics for imports that are queued
and fetching data (though there is a bug here fixed above), we
want to provide more granularity to help with debugging:
This explicitly separates "current" and "pending" imports:
- "live": The import is live, ie it is currently importing data
- having this metric allows debugging the case that there
is a problem fetching data
- "pending": The import is queued, live, or getting data from the
cache
- having this metric allows debugging the case that there is a
problem initiating the request, for example a request is being
starved on the queue
**note**: I moved the watches closer to the import function call
instead of renaming the currently broken pending import watches
to make it more clear in the code what these are suppose to be
timing
Reviewed By: chadaustin
Differential Revision: D20629833
fbshipit-source-id: 84ef75057149a648a51418a5cc93be87e3b3d6b5
Summary:
Migration from Future-returning executor-erasing collectX forms to
SemiFuture-returning forms, that are less risky in particular with coroutines.
Earlier changes added collectXSemiFuture and collectXUnsafe as a migration
path. We then migrated collectX callsites to collectXSemiFuture or
collectXUnsafe and switched the implementation of collectX to the SemiFuture
form.
This diff renames uses of collectXSemiFuture to collectX, as the two are
identical.
Reviewed By: simpkins
Differential Revision: D20833464
fbshipit-source-id: c14aa40f8cce121632ff922300ef271721c1d695
Summary:
The primary purpose of exposing counters for the maximum length of duration is to add this to `eden top`.
As discussed in D20611704, the duration of time that imports are queued for is a strong indicator of the health of the import process. Adding it to `eden top` will help give users insight into what eden is doing when it hangs for a long time or when there is an issue, indicate whether that issue is due to the import process.
Additionally we choose to use these counters to expose this data because we want to log this data. This data can be used to measure performance of the queueing and import process. In the future this data could be used to set up alerting for regressions and allow more proactive fixes.
Reviewed By: chadaustin
Differential Revision: D20611728
fbshipit-source-id: 9307c1ad749ac5fe356ba9eaf868de41b1a8a3a7
Summary:
Here we are tracking the duration of each of the current imports in `HgBackingStore`. This changes is setup to display the duration of the longest outstanding import in `eden top`.
Displaying the duration of the longest import in eden top will help debug where the problem is when eden hangs. These specifically help determine if eden is hanging because of a queueing issue or not being able to fetch an object from the server.
This is a stronger signal to the user that there is an issue in importing over just a count of imports (the previous addition) because many pending imports is not always abnormal. For example a program may prefetch a big batch of files to read, and this could cause a large number of pending imports. A single import should not hang due to a big prefetch and other normal uses.
Thus we want to display this in addition to the number of imports and use this as our warning indicator. (i.e. a high duration of import will mark the metrics as yellow or red in `eden top` to give a warning to the user)
*code changes*:
- adding lists to store watches to time the duration of each of the imports in `HgBackingStore`
- removing the counters previously used to count the number of imports in `HgBackingStore` in favor of using the size of these lists as this is constant time, so there is no benefit to tracking this value ourselves
Reviewed By: chadaustin
Differential Revision: D20611704
fbshipit-source-id: 8bbe0aed8d10688f7bfb7ecfa2fa5ac181945a76
Summary:
fixes up some functions to avoid narrowing warnings
and others to adopt `folly::to_narrow` to accept them.
This reduces the number of warnings in the windows build.
Reviewed By: chadaustin
Differential Revision: D20562026
fbshipit-source-id: f18ba50f914e142415d1efefebc6297e8c68d38e
Summary:
This is a rough pass that resolves a linker issue on MSVC by
switching to inline static member functions.
Reviewed By: chadaustin
Differential Revision: D20529163
fbshipit-source-id: 578ed440758c685091d3e039e261638e027db17a
Summary:
s/CURLE_SSL_CONNECT_ERROR/CURLE_RECV_ERROR/.
Note CURLE_SSL_CONNECT_ERROR is not explicitly checked in
code in the eden hierarchy (or anywhere else significant in fbcode).
Reviewed By: meyering
Differential Revision: D20490844
fbshipit-source-id: be4a66f49eb4a9eabaf73785f9a203f0aa6905a4
Summary: This diff bumps the open call from FUSE to High priority (which is higher than any other blob open request atm). This has shown improvement on the user experience of EdenFS when it's importing many other things from other channels (thrift, etc.)
Reviewed By: chadaustin
Differential Revision: D20287389
fbshipit-source-id: 319bc44ef8be5c904d7cf0db7cc2f8be28b4760a
Summary: This diff implements `HgQueuedBackingStore` that uses `HgImportRequestQueue` to provide SCM data importing with priorities. This can allow us to customize how we import things and batch importing.
Reviewed By: chadaustin
Differential Revision: D19184826
fbshipit-source-id: da579b5bbff0b1449e9689e2c0159d4a3a475a83
Summary: This diff adds `HgImportRequestQueue` that is responsible for managing incoming requests by their priorities. This queue is later used in the `HgQueuedBackingStore` to prioritize works to the workers of the backing store.
Reviewed By: chadaustin
Differential Revision: D20197069
fbshipit-source-id: 246bbc086054a8021226e9ba6ab26d3bf0cfb7a3
Summary:
This class is used to represent an import request that will be used later in the queue implementation.
When the EdenFS needs to import a blob, it creates an instance of this request and send it to the worker. Then it waits for the promise associated with the request.
In the future, we should be able to change the owned `Promise` into a non-owned `SemiFuture` to a `Promise` somewhere else for merging repetitive import requests.
Reviewed By: chadaustin
Differential Revision: D19184824
fbshipit-source-id: 823aabbed1156acf6306b7aefc76580a540d310d
Summary: This diff adds `Priority` added in the previous diff to the `BackingStore` interface with the default value set to `Priority::Normal`.
Reviewed By: chadaustin
Differential Revision: D20197071
fbshipit-source-id: a92f1b49bb82e3478042e5e3b79b047d834755ea
Summary:
This diff introduces a `Priority` type for EdenFS. This type is used to pass along the priority of a request.
The priority class itself contains two parts, `kind` and `offset`. `kind` uses the first 4-bytes and the reset 12-bytes are used to store offset. The idea is that we can roughly assign a priority kind to most of the requests and offset is used to dynamically tweak the priority of some particular requests. For example, when we saw a process is generate millions of requests we can use this to express "normal priority but less important than other process's normal priority".
Reviewed By: chadaustin
Differential Revision: D20287652
fbshipit-source-id: 9a849fb6cc6ba5e443fea978d5b4dc3ab8ca906a
Summary:
Migration from Future-returning executor-erasing collectX forms to
SemiFuture-returning forms, that are less risky in particular with coroutines.
Earlier diffs added SemiFuture and Unsafe versions. This codemod migrates
collect versions to the Unsafe versions to allow the basic collect versions to
be made safe.
Reviewed By: simpkins
Differential Revision: D20331206
fbshipit-source-id: efc8dff487d45f7d53ee55e8c4696bd3eed0e6da
Summary: expose the counters for number of pending imports (blobs, trees, prefetches) to allow use in tooling
Reviewed By: chadaustin
Differential Revision: D20269853
fbshipit-source-id: d2b7e2110520290751699c4a891d41ebd5b374cf
Summary:
adds a counter to track the imports queued to enable more statistics exposure.
- Add a counters to track the number of blob, tree, prefetch imports that are in the pending
- have the counters increment (increment in constructor of wrapper struct) when the import is about to be queued
- have counters decrement once the load has completed (decrement in destructor of wrapper struct)
Reviewed By: chadaustin
Differential Revision: D20256410
fbshipit-source-id: 5536b46307b30fc19dc5747414727a86961c78e1
Summary: - added logging only around the import tree call to capture non-queue related wait time
Reviewed By: chadaustin, fanzeyi
Differential Revision: D20207472
fbshipit-source-id: d88bb34ce224a26ff2be100d7789ddeff608006d
Summary:
- added logging only around the import blob call to capture non-queue related wait time
- added to `test_reading_file_gets_file_from_hg` in `integration.stats_test.HgBackingStoreStatsTest` to test import blob logging in addition to the get blob loging
(not yet done for importing trees, will do in next diff)
Reviewed By: chadaustin
Differential Revision: D20201215
fbshipit-source-id: c89281fe7d3d6e89d111ac8cce9014adff44ac40
Summary:
A bunch of files include folly/executors/GlobalExecutors.h transitively through thrift/lib/cpp2/async/Stream.h, which is going away. Explicitly include the header (and add dependency to target) in preparation for deleting Stream.h
drop-conflicts
Reviewed By: vitaut
Differential Revision: D20141838
fbshipit-source-id: 21c58cf82136287fc2d84ba5badec6b872106015
Summary:
The problem is that the datapack files are not flushed to disk when it is prefetched. By having a pair of brackets around the `HgBackingStore`, it will ensure the `HgImporter` is closed by the time when we verify the prefetch with `hg cat` since it will terminate the `debugedenimporthelper` process in its destructor, which flushes the datapack files.
The real cause of the test failure is still unclear but I believe this is the correct way of doing this test.
Reviewed By: xavierd
Differential Revision: D20090249
fbshipit-source-id: 8e3966936a402c92311919433282027846d065e8
Summary: so that it can be shared more easily with AsyncUDPSocket
Reviewed By: yangchi
Differential Revision: D19851480
fbshipit-source-id: ec8cdb852519724db6f89cf70c4a4169de5028b6
Summary:
This commit manually synchronizes the internal move of
fbcode/scm/mononoke under fbcode/eden/mononoke which couldn't be
performed by ShipIt automatically.
Reviewed By: StanislavGlebik
Differential Revision: D19722832
fbshipit-source-id: 52fbc8bc42a8940b39872dfb8b00ce9c0f6b0800
Summary: In addition to duration and success, log object fetch counts and checkout type to Scuba.
Reviewed By: fanzeyi
Differential Revision: D19334276
fbshipit-source-id: dabf52427f2ebda2b58df93194df39d52f4fcb4f
Summary: Log the number of object lookups and cache hit rates for a checkout operation.
Reviewed By: simpkins
Differential Revision: D19191201
fbshipit-source-id: 5e9ad501e704810f072dabcda3fce86d027c452e
Summary:
During checkout and stats, count every object fetch and which level of
cache it was served from.
Reviewed By: simpkins
Differential Revision: D19186333
fbshipit-source-id: fc0a74db297b9c723682e245996a7befd762f933
Summary:
Add a fetch context interface to ObjectStore that allows tracing cache
hits, backing store fetches, and fetch durations in the context of a
diff or checkout operation.
Reviewed By: simpkins
Differential Revision: D19135625
fbshipit-source-id: d0d8f134b1c89f7ba4971a404a46a69a1704ba5c
Summary:
There were some unsafe uses of folly::collect, especially in
the diff code. Switch to collectSafe.
Reviewed By: simpkins
Differential Revision: D19529784
fbshipit-source-id: e0ac920a2d2b44767a453f2c41a9166e72246ab6
Summary: the request will be passed with the context, so simply check the context at all three "looping" points to see if the request was cancelled. I split this into its own diff due to its unique test plan.
Reviewed By: chadaustin
Differential Revision: D18647091
fbshipit-source-id: ccb7ac765f8f3b0a60e936779cc27b658b9fd8b9
