Summary:
Eden can sometimes unexpectedly fetch files from the server, and we want
to know why this is happening. This adds logging for the source of
data fetching in edens backing store to help obviate why these fetches
are happening.
This temporarily adds the logging in the HgQueuedBacking store to get a naive
version of logging rolled out sooner. Follow up changes will move this logging
closer to the data fetching itself if possible (in HgDatapackStore and HgImporter).
Reviewed By: chadaustin
Differential Revision: D22012572
fbshipit-source-id: b1b012ce4ee133fbacecd586b7365c3c5a5386df
Summary:
This passes ObjectFetchContext into the backing store to prepare for adding
logging for the cause of server fetches.
In following changes I will add logging in the HgQueuedBackingStore.
Ultimately we will want to move this logging to be closer to the data fetching
(in HgDatapackStore and HgImporter), but I plan to temporarily add logging to
the HgQueuedBackingStore to simplify so that we can more quickly roll out.
Reviewed By: chadaustin
Differential Revision: D22022992
fbshipit-source-id: ccb428458cbf7a1e33aaf9be9d0d766c45acedb3
Summary:
On Windows (haven't verified on other platforms), ThreadLocalPtr don't appear
to be releasing resources when a thread die. This means that when the importer
thread dies, the actual importer (hg.real) would still run and use resources,
with no way of talking to it.
To fix this, let's manually reset it when the main thread function returns,
this forces the importer to be destroyed and therefore the various handles to
hg.real to be released, effectively terminating it.
I'm not sure if this is the proper fix, but delving into folly feels a bit
daunting. Keeping a TODO for later to go back to it and fix it properly in
folly.
Reviewed By: chadaustin
Differential Revision: D22012540
fbshipit-source-id: 99f994bb5128b38ccf8474031763b8a21055759a
Summary:
NOTE: This stack works together, they are separated for easier reviewing. Check D21723465 if you want to read this stack in one place.
This diff finally connects EdenAPI with EdenFS. Previously we only had basic EdenAPI implemented but the performance was not acceptable. This is largely due to overheads in talking with the remote server. Now EdenFS is able to import multiple files at once, we efficiently use EdenAPI to import these data now.
Reviewed By: chadaustin
Differential Revision: D21576569
fbshipit-source-id: a45e832ec63d057730138551393ff7547fa2c22f
Summary:
The CreateProcess API allows inheritable handles to be available in the spawned
process by passing a flag to it. What the documentation for this API leave to
the imagination is what happens when processes are spawned concurrently in
multiple threads and handles are opened and made inheritable while doing this.
The answer is obvious but the consequences aren't: they are inherited.
When anonymous Pipes are used, one end needs to be inheritable for the spawned
process to being able use it, but if one is created concurrently with spawning
a process, that other process may have an open handle to that unrelated pipe.
And since to detect that a pipe is closed, all handles to the other end needs
to be closed, this can lead to never being able to detect that it is closed...
This scenario is exactly what is happening in eden when spawning the Mercurial
process, and when one of these processes would die, eden would just hang trying
to write to the pipe, not knowing that the process was already gone. To unblock
eden, all the hg.real processes had to be killed, as this would close all the
pipe handles, and then eden would detect that the pipe was closed and re-spawn
Mercurial (only for the exact same thing to happen).
In order to fix this, we need to tell CreateProcess to only inherit the pipes
handles, and nothing else.
Reviewed By: wez
Differential Revision: D21895537
fbshipit-source-id: 3c84a1d0316b50b5750f554fa20f72f59a718882
Summary: This function won't go ask remote server and is able to produce a result very quick. This avoids the waiting time in the queue.
Reviewed By: chadaustin
Differential Revision: D21586511
fbshipit-source-id: db3793cc4e4e493d1beb6837c0bd91358e7e845f
Summary: Reading from hgcache directly has been rolled out for quite a while now. Let's remove the configuration and make it default behavior.
Reviewed By: chadaustin
Differential Revision: D21585482
fbshipit-source-id: 7f30e262642adf58388a0580aa5a63595fa89155
Summary:
gcc and clang disagree here, so explicitly construct a Future from the
unique_ptr.
Reviewed By: genevievehelsel
Differential Revision: D21687654
fbshipit-source-id: 338174f5d4d051f509faace6e48d1c0fb4d898e1
Summary: Fix some RVO warnings and a compilation error when compiling with gcc.
Reviewed By: xavierd
Differential Revision: D21481738
fbshipit-source-id: 0621f5886df40c24ef1a6a68ccd957e38f2f4122
Summary: Now that Rust libraries can be linked into fbcode binaries with mode/mac, enable Rust datapack on all platforms.
Reviewed By: xavierd
Differential Revision: D21241075
fbshipit-source-id: ae2abee25a5ad7b9db9ac5e6b8687c5f79376926
Summary: Instead of having `HgBackingStore` fetch blobs separately, we now try to read from hgcache and fetch from `HgImporter` as batches.
Reviewed By: chadaustin
Differential Revision: D20903245
fbshipit-source-id: d8e404d6765f1bcbacbf2a39f83eab0a351a3fe0
Summary: This diff makes `HgImportRequestQueue` to be able to return multiple requests in the queue at once.
Reviewed By: chadaustin
Differential Revision: D20197070
fbshipit-source-id: 8cff1780d6e56321a756d30ac0e9b9d5d319c049
Summary: The life of a request is only able to be finished with one of the method. So we can instead having the tracker destroyed when the future is resolved.
Reviewed By: chadaustin
Differential Revision: D20995819
fbshipit-source-id: 5dac2f762513b5d0bcacaab7d0669fc8fdb61e80
Summary:
Readability improvements.
`HgQueuedBackingStore` will need to call these functions individually.
Reviewed By: chadaustin
Differential Revision: D20683321
fbshipit-source-id: 9a9bd766c34559048bd0971f17304090abbb2774
Summary:
During a graceful restart, while the fuse mount is not handling
requests, avoid a possible deadlock between the edenfs, hg
debugedenimporthelper, and watchman processes.
See the comment in HgImporter.cpp.
Reviewed By: fanzeyi
Differential Revision: D21342275
fbshipit-source-id: df8fb5df5d5cd1490e88b42054b34cbb2acdb692
Summary: So the linter will stop nagging at my changes to this file.
Reviewed By: wez
Differential Revision: D21396011
fbshipit-source-id: dbc3479637cca83aa1a11ff94a9033dfa42fc2a6
Summary:
Talked with xavierd last week and we can use LocalStore's `get_missing` to determine if a blob is present locally. In this way we can prevent the backingstore crate from accidentally asking EdenAPI for a blob, so better control at EdenFS level.
With this change, we can use this function at the time where a blob import request is created with confidence that this should be short cheap call.
This diff should not change any behavior or performance.
Reviewed By: xavierd
Differential Revision: D21391959
fbshipit-source-id: fd31687da1e048262cb4eae2974cab6d8915a76d
Summary:
This sets up the counters that will allow us to expose the number
of pending FUSE requests in Eden top.
As D20846826 mentions adding metrics for FUSE request gives
visibility into fuse requests and overall health of eden.
This provides more insight beyond the metrics for live FUSE requests
since it shows the kernels view of FUSE requests. Looking at the difference
between the number of pending and live request, can identify issues
that arise at the interface between eden and FUSE and monitor how
quickly fuse workers are processing requests.
**note**: this is only for linux since macos has no equivalent to
`/sys/fs/fuse/connections`
Reviewed By: chadaustin
Differential Revision: D21074489
fbshipit-source-id: c0951f0dfd4fa764be28d8686d08cd0dd807db37
Summary: On Windows we are seeing frequent hang issues while using telemetry hg binary. Switching to using the hg.real for all the platforms.
Reviewed By: simpkins
Differential Revision: D20480875
fbshipit-source-id: 82892da7f8d46f2413442c37251243400180f9db
Summary: Close the Pipe handles that are inherited in the child and are not needed in the parent process. This will also make sure the pipe is closed when the child process ends.
Reviewed By: simpkins
Differential Revision: D20923588
fbshipit-source-id: aa58e99148d2f5c3bf666a3237266eac725be563
Summary:
This exposes metrics for the live FUSE requests (the duration
of the longest outstanding request and the number of outstanding
requests).
Because FUSE is the interface through which the user mostly interacts
with the file system they provide good metrics to judge if the perfomance
of eden is normal, or there may be an issue.
Exposing these counters this way will send them to ods, so it will not only
allow for debuging current issues, but can be used to look back at previous
problems. This data could also be used for alerting or more proactive
remediation.
Metrics are exposed per checkout to allow seeing which checkout was
having issues. This data will aggregated in `eden top` to be used as
an overall health indicator, but should more information be needed it
will be logged in ods.
Reviewed By: chadaustin
Differential Revision: D20922194
fbshipit-source-id: 16208883417acb77b62bf712cfdd9068c5420303
Summary: It's a little weird to have get batch returns a pair of `RelativePath` and `Hash`.
Reviewed By: chadaustin
Differential Revision: D20844545
fbshipit-source-id: c0c823e0aca6ea2326cb637226dc0de43fd4f3cd
Summary: This diff makes `HgProxyHash` moveable & copyable so we can return it from `getBatch` method instead returning a pair of `RelativePath` & `Hash`. See next diff for usage.
Reviewed By: chadaustin
Differential Revision: D20841434
fbshipit-source-id: c082c217f75d53bcbfb4853e2055d1458a040188
Summary: gcc warned that these functions did not return with a value on all code paths.
Reviewed By: genevievehelsel
Differential Revision: D21004224
fbshipit-source-id: 3ccd38b19bb4f4c0dbe0b75b4d11f351b6690591
Summary:
This code isn't really working and is giving out corrupted file content. Let's remove it for reading blobs.
We still cannot remove the C++ mercurial code entirely as we rely on it to import manifests. I tested on switching to the Rust manifest implementation but it wasn't working due to new pack file scanning.
Reviewed By: chadaustin
Differential Revision: D20938001
fbshipit-source-id: 9cc7b1ed9cd8228afd6fc5374cda001d1309a379
Summary:
Since imports are now queued in `HgQueuedBackingStore`
the metrics for queued imports were broken
the queueing use to happen in `HgBackingStore`, but
now the queue happens before the imports get here,
in `HgQueuedBackingStore`, thus these metrics were
only measuring live imports
=> this moves the metrics for queued imports into
`HgQueuedBackingStore` so that they capture both
live and queued imports
**note**: since prefetching in `HgQueuedBackingStore`
is in progress, these metrics are only set-up for when
prefetching this is finished
Reviewed By: chadaustin
Differential Revision: D20714494
fbshipit-source-id: 392337aaa60b44b75184889e6c92cb5d2c2577cc
Summary:
As mentioned in D20629833, adding metrics for live
imports in `eden top` gives more transparency to the
imports process and makes identifying import related
issues easier. This is set up to expose metrics for live
imports like those for pending imports in `eden top`.
Similar to D20611728 exposing this via these counters
will log this data. Having this data persisted will allow
tracking the performance of imports, and does the set
up for more pro-active fixing of issues. Further we can
look back to see issues that are no longer occurring, but
still of interest.
This also refactors the registration code so that it requires
no copy pasting to add a new counter. Avoiding copy paste
errors when adding more counters and making it easier to
maintain.
Reviewed By: chadaustin
Differential Revision: D20630813
fbshipit-source-id: 8a7a2a0135c7b7a5cde960b84dcb434c6c99eaeb
Summary: This is actually missing from `HgQueuedBackingStore`. This diff fixes this and assign low priority to these prefetch requests.
Reviewed By: chadaustin
Differential Revision: D20655681
fbshipit-source-id: f3c92b358e16e980390ac7adcae27d41ae5a7277
Summary: Ergonomics improvements. This allows us to have type safety instead of dynamically check for types at runtime.
Reviewed By: chadaustin
Differential Revision: D20740318
fbshipit-source-id: d0a3255105bef22dd236695be9fc9c4220547b83
Summary:
Previously `eden top` shows metrics for imports that are queued
and fetching data (though there is a bug here fixed above), we
want to provide more granularity to help with debugging:
This explicitly separates "current" and "pending" imports:
- "live": The import is live, ie it is currently importing data
- having this metric allows debugging the case that there
is a problem fetching data
- "pending": The import is queued, live, or getting data from the
cache
- having this metric allows debugging the case that there is a
problem initiating the request, for example a request is being
starved on the queue
**note**: I moved the watches closer to the import function call
instead of renaming the currently broken pending import watches
to make it more clear in the code what these are suppose to be
timing
Reviewed By: chadaustin
Differential Revision: D20629833
fbshipit-source-id: 84ef75057149a648a51418a5cc93be87e3b3d6b5
Summary:
The primary purpose of exposing counters for the maximum length of duration is to add this to `eden top`.
As discussed in D20611704, the duration of time that imports are queued for is a strong indicator of the health of the import process. Adding it to `eden top` will help give users insight into what eden is doing when it hangs for a long time or when there is an issue, indicate whether that issue is due to the import process.
Additionally we choose to use these counters to expose this data because we want to log this data. This data can be used to measure performance of the queueing and import process. In the future this data could be used to set up alerting for regressions and allow more proactive fixes.
Reviewed By: chadaustin
Differential Revision: D20611728
fbshipit-source-id: 9307c1ad749ac5fe356ba9eaf868de41b1a8a3a7
Summary:
Here we are tracking the duration of each of the current imports in `HgBackingStore`. This changes is setup to display the duration of the longest outstanding import in `eden top`.
Displaying the duration of the longest import in eden top will help debug where the problem is when eden hangs. These specifically help determine if eden is hanging because of a queueing issue or not being able to fetch an object from the server.
This is a stronger signal to the user that there is an issue in importing over just a count of imports (the previous addition) because many pending imports is not always abnormal. For example a program may prefetch a big batch of files to read, and this could cause a large number of pending imports. A single import should not hang due to a big prefetch and other normal uses.
Thus we want to display this in addition to the number of imports and use this as our warning indicator. (i.e. a high duration of import will mark the metrics as yellow or red in `eden top` to give a warning to the user)
*code changes*:
- adding lists to store watches to time the duration of each of the imports in `HgBackingStore`
- removing the counters previously used to count the number of imports in `HgBackingStore` in favor of using the size of these lists as this is constant time, so there is no benefit to tracking this value ourselves
Reviewed By: chadaustin
Differential Revision: D20611704
fbshipit-source-id: 8bbe0aed8d10688f7bfb7ecfa2fa5ac181945a76
Summary:
fixes up some functions to avoid narrowing warnings
and others to adopt `folly::to_narrow` to accept them.
This reduces the number of warnings in the windows build.
Reviewed By: chadaustin
Differential Revision: D20562026
fbshipit-source-id: f18ba50f914e142415d1efefebc6297e8c68d38e
Summary:
This is a rough pass that resolves a linker issue on MSVC by
switching to inline static member functions.
Reviewed By: chadaustin
Differential Revision: D20529163
fbshipit-source-id: 578ed440758c685091d3e039e261638e027db17a
Summary: This diff implements `HgQueuedBackingStore` that uses `HgImportRequestQueue` to provide SCM data importing with priorities. This can allow us to customize how we import things and batch importing.
Reviewed By: chadaustin
Differential Revision: D19184826
fbshipit-source-id: da579b5bbff0b1449e9689e2c0159d4a3a475a83
Summary: This diff adds `HgImportRequestQueue` that is responsible for managing incoming requests by their priorities. This queue is later used in the `HgQueuedBackingStore` to prioritize works to the workers of the backing store.
Reviewed By: chadaustin
Differential Revision: D20197069
fbshipit-source-id: 246bbc086054a8021226e9ba6ab26d3bf0cfb7a3
Summary:
This class is used to represent an import request that will be used later in the queue implementation.
When the EdenFS needs to import a blob, it creates an instance of this request and send it to the worker. Then it waits for the promise associated with the request.
In the future, we should be able to change the owned `Promise` into a non-owned `SemiFuture` to a `Promise` somewhere else for merging repetitive import requests.
Reviewed By: chadaustin
Differential Revision: D19184824
fbshipit-source-id: 823aabbed1156acf6306b7aefc76580a540d310d
Summary: This diff adds `Priority` added in the previous diff to the `BackingStore` interface with the default value set to `Priority::Normal`.
Reviewed By: chadaustin
Differential Revision: D20197071
fbshipit-source-id: a92f1b49bb82e3478042e5e3b79b047d834755ea
Summary: expose the counters for number of pending imports (blobs, trees, prefetches) to allow use in tooling
Reviewed By: chadaustin
Differential Revision: D20269853
fbshipit-source-id: d2b7e2110520290751699c4a891d41ebd5b374cf
Summary:
adds a counter to track the imports queued to enable more statistics exposure.
- Add a counters to track the number of blob, tree, prefetch imports that are in the pending
- have the counters increment (increment in constructor of wrapper struct) when the import is about to be queued
- have counters decrement once the load has completed (decrement in destructor of wrapper struct)
Reviewed By: chadaustin
Differential Revision: D20256410
fbshipit-source-id: 5536b46307b30fc19dc5747414727a86961c78e1
Summary: - added logging only around the import tree call to capture non-queue related wait time
Reviewed By: chadaustin, fanzeyi
Differential Revision: D20207472
fbshipit-source-id: d88bb34ce224a26ff2be100d7789ddeff608006d
Summary:
- added logging only around the import blob call to capture non-queue related wait time
- added to `test_reading_file_gets_file_from_hg` in `integration.stats_test.HgBackingStoreStatsTest` to test import blob logging in addition to the get blob loging
(not yet done for importing trees, will do in next diff)
Reviewed By: chadaustin
Differential Revision: D20201215
fbshipit-source-id: c89281fe7d3d6e89d111ac8cce9014adff44ac40
Summary:
The problem is that the datapack files are not flushed to disk when it is prefetched. By having a pair of brackets around the `HgBackingStore`, it will ensure the `HgImporter` is closed by the time when we verify the prefetch with `hg cat` since it will terminate the `debugedenimporthelper` process in its destructor, which flushes the datapack files.
The real cause of the test failure is still unclear but I believe this is the correct way of doing this test.
Reviewed By: xavierd
Differential Revision: D20090249
fbshipit-source-id: 8e3966936a402c92311919433282027846d065e8
Summary:
Add a fetch context interface to ObjectStore that allows tracing cache
hits, backing store fetches, and fetch durations in the context of a
diff or checkout operation.
Reviewed By: simpkins
Differential Revision: D19135625
fbshipit-source-id: d0d8f134b1c89f7ba4971a404a46a69a1704ba5c
Summary:
It's not often obvious to people where edenfs.log is located on the
filesystem, so redirect people to `eden debug log` instead.
Reviewed By: simpkins
Differential Revision: D19560709
fbshipit-source-id: 6b031c80bab2c21dfddba4d7ec37d4e70218ea08
