Summary: This function won't go ask remote server and is able to produce a result very quick. This avoids the waiting time in the queue.
Reviewed By: chadaustin
Differential Revision: D21586511
fbshipit-source-id: db3793cc4e4e493d1beb6837c0bd91358e7e845f
Summary: Reading from hgcache directly has been rolled out for quite a while now. Let's remove the configuration and make it default behavior.
Reviewed By: chadaustin
Differential Revision: D21585482
fbshipit-source-id: 7f30e262642adf58388a0580aa5a63595fa89155
Summary:
gcc and clang disagree here, so explicitly construct a Future from the
unique_ptr.
Reviewed By: genevievehelsel
Differential Revision: D21687654
fbshipit-source-id: 338174f5d4d051f509faace6e48d1c0fb4d898e1
Summary: Fix some RVO warnings and a compilation error when compiling with gcc.
Reviewed By: xavierd
Differential Revision: D21481738
fbshipit-source-id: 0621f5886df40c24ef1a6a68ccd957e38f2f4122
Summary: Now that Rust libraries can be linked into fbcode binaries with mode/mac, enable Rust datapack on all platforms.
Reviewed By: xavierd
Differential Revision: D21241075
fbshipit-source-id: ae2abee25a5ad7b9db9ac5e6b8687c5f79376926
Summary:
While this isn't the right fix, this is what shipped in our packages, for the
sake of being able to reproduce the package, let's land this as it is. A
future change will remove this ifdef.
Below is pkaush original description:
In Eden Windows we treat all the files as regular files and don't have a
concept of symlinks and executable files. Fixing the TreeEntryType::getType()
to return REGULAR_FILE for executable file and symlink.
Reviewed By: wez
Differential Revision: D20481051
fbshipit-source-id: 0b0c4d7aea28134383ef45aeafc02930b420286b
Summary:
These 3 tests compile without issues on Windows. The RocksDB one is weird,
while it compiles with no hickups, I simply cannot run the resulting test
binary, and I'm not sure how to debug this. The local store one fails in folly.
Reviewed By: chadaustin
Differential Revision: D21393724
fbshipit-source-id: db90bf20a9d116bc8aa493703997c5e8da76eb1f
Summary:
vs2017 is not able to compile the static assertion in KeySpace.cpp.
Previously we thought that this would be resolved in a later release of vs2017
but now that is here it is clear that it hasn't been fixed.
This commit pushes the version requirement to vs2019 (see
https://dev.to/yumetodo/list-of-mscver-and-mscfullver-8nd for a mapping between
product versions and compiler versions), but we cannot build with vs2019
because folly and rangev3 don't compile with vs2019, so this assertion (heh!)
has literally not been tested.
This commit also fixes up an oversight in the gating logic: the intent is that
we perform the assertion on all systems except known broken MSVC. We were
accidentally restricting it to later versions of MSVC.
Reviewed By: simpkins
Differential Revision: D21432890
fbshipit-source-id: e11ffccc53bf8dffdf2db45ad4f3cf199b1cc70d
Summary: Instead of having `HgBackingStore` fetch blobs separately, we now try to read from hgcache and fetch from `HgImporter` as batches.
Reviewed By: chadaustin
Differential Revision: D20903245
fbshipit-source-id: d8e404d6765f1bcbacbf2a39f83eab0a351a3fe0
Summary: This diff makes `HgImportRequestQueue` to be able to return multiple requests in the queue at once.
Reviewed By: chadaustin
Differential Revision: D20197070
fbshipit-source-id: 8cff1780d6e56321a756d30ac0e9b9d5d319c049
Summary: The life of a request is only able to be finished with one of the method. So we can instead having the tracker destroyed when the future is resolved.
Reviewed By: chadaustin
Differential Revision: D20995819
fbshipit-source-id: 5dac2f762513b5d0bcacaab7d0669fc8fdb61e80
Summary:
Readability improvements.
`HgQueuedBackingStore` will need to call these functions individually.
Reviewed By: chadaustin
Differential Revision: D20683321
fbshipit-source-id: 9a9bd766c34559048bd0971f17304090abbb2774
Summary:
During a graceful restart, while the fuse mount is not handling
requests, avoid a possible deadlock between the edenfs, hg
debugedenimporthelper, and watchman processes.
See the comment in HgImporter.cpp.
Reviewed By: fanzeyi
Differential Revision: D21342275
fbshipit-source-id: df8fb5df5d5cd1490e88b42054b34cbb2acdb692
Summary: So the linter will stop nagging at my changes to this file.
Reviewed By: wez
Differential Revision: D21396011
fbshipit-source-id: dbc3479637cca83aa1a11ff94a9033dfa42fc2a6
Summary:
Talked with xavierd last week and we can use LocalStore's `get_missing` to determine if a blob is present locally. In this way we can prevent the backingstore crate from accidentally asking EdenAPI for a blob, so better control at EdenFS level.
With this change, we can use this function at the time where a blob import request is created with confidence that this should be short cheap call.
This diff should not change any behavior or performance.
Reviewed By: xavierd
Differential Revision: D21391959
fbshipit-source-id: fd31687da1e048262cb4eae2974cab6d8915a76d
Summary:
In the initial stages of the windows port we had
problems building rocksdb on windows, so we disabled it.
These days we're able to build it and detect it--we even
require it in the cmake code, but hadn't gotten around
to telling the rest of the code that we can use it.
This commit re-enables it in the build but leaves sqlite
as the default engine until we're able to perform some
benchmarking.
Rocksdb itself has some build issues on Windows; it doesn't
use cmake to locate dependencies, so even though we built
snappy it doesn't know how to find it without modifying the
source:
https://github.com/facebook/rocksdb/blob/master/thirdparty.inc#L4
For that reason, we disable the use of Snappy in the Windows build.
However, in the version of rocksdb that we were using, it would
default to trying to use Snappy even though it wasn't compiled in
and throw an exception.
I've upgraded to a newer version of rocksdb that will simply not
use compression if no compression was enabled at build time.
Given that we mostly store relatively small objects, I'm assuming
that the lack of compression is fine for now.
Reviewed By: xavierd
Differential Revision: D21319896
fbshipit-source-id: 2a2d06d4bd5382706e9220f9b4a2de99dc18311d
Summary:
Move the `UserInfo` code from `fuse/privhelper` to `utils`, and also unify the
POSIX and Windows implementations of this class.
This code was originally put in the `fuse/privhelper` directory since it was
written at the same time as the privhelper. However, it's really a
lower-level library that doesn't depend on FUSE or any of the other code in
the `fuse/` subdirectory.
Reviewed By: wez
Differential Revision: D21296594
fbshipit-source-id: f58682f6ce86bba0328472c491bb4c0dc3370319
Summary:
This sets up the counters that will allow us to expose the number
of pending FUSE requests in Eden top.
As D20846826 mentions adding metrics for FUSE request gives
visibility into fuse requests and overall health of eden.
This provides more insight beyond the metrics for live FUSE requests
since it shows the kernels view of FUSE requests. Looking at the difference
between the number of pending and live request, can identify issues
that arise at the interface between eden and FUSE and monitor how
quickly fuse workers are processing requests.
**note**: this is only for linux since macos has no equivalent to
`/sys/fs/fuse/connections`
Reviewed By: chadaustin
Differential Revision: D21074489
fbshipit-source-id: c0951f0dfd4fa764be28d8686d08cd0dd807db37
Summary: On Windows we are seeing frequent hang issues while using telemetry hg binary. Switching to using the hg.real for all the platforms.
Reviewed By: simpkins
Differential Revision: D20480875
fbshipit-source-id: 82892da7f8d46f2413442c37251243400180f9db
Summary: Close the Pipe handles that are inherited in the child and are not needed in the parent process. This will also make sure the pipe is closed when the child process ends.
Reviewed By: simpkins
Differential Revision: D20923588
fbshipit-source-id: aa58e99148d2f5c3bf666a3237266eac725be563
Summary:
This exposes metrics for the live FUSE requests (the duration
of the longest outstanding request and the number of outstanding
requests).
Because FUSE is the interface through which the user mostly interacts
with the file system they provide good metrics to judge if the perfomance
of eden is normal, or there may be an issue.
Exposing these counters this way will send them to ods, so it will not only
allow for debuging current issues, but can be used to look back at previous
problems. This data could also be used for alerting or more proactive
remediation.
Metrics are exposed per checkout to allow seeing which checkout was
having issues. This data will aggregated in `eden top` to be used as
an overall health indicator, but should more information be needed it
will be logged in ods.
Reviewed By: chadaustin
Differential Revision: D20922194
fbshipit-source-id: 16208883417acb77b62bf712cfdd9068c5420303
Summary: It's a little weird to have get batch returns a pair of `RelativePath` and `Hash`.
Reviewed By: chadaustin
Differential Revision: D20844545
fbshipit-source-id: c0c823e0aca6ea2326cb637226dc0de43fd4f3cd
Summary: This diff makes `HgProxyHash` moveable & copyable so we can return it from `getBatch` method instead returning a pair of `RelativePath` & `Hash`. See next diff for usage.
Reviewed By: chadaustin
Differential Revision: D20841434
fbshipit-source-id: c082c217f75d53bcbfb4853e2055d1458a040188
Summary: gcc warned that these functions did not return with a value on all code paths.
Reviewed By: genevievehelsel
Differential Revision: D21004224
fbshipit-source-id: 3ccd38b19bb4f4c0dbe0b75b4d11f351b6690591
Summary:
This code isn't really working and is giving out corrupted file content. Let's remove it for reading blobs.
We still cannot remove the C++ mercurial code entirely as we rely on it to import manifests. I tested on switching to the Rust manifest implementation but it wasn't working due to new pack file scanning.
Reviewed By: chadaustin
Differential Revision: D20938001
fbshipit-source-id: 9cc7b1ed9cd8228afd6fc5374cda001d1309a379
Summary:
Since imports are now queued in `HgQueuedBackingStore`
the metrics for queued imports were broken
the queueing use to happen in `HgBackingStore`, but
now the queue happens before the imports get here,
in `HgQueuedBackingStore`, thus these metrics were
only measuring live imports
=> this moves the metrics for queued imports into
`HgQueuedBackingStore` so that they capture both
live and queued imports
**note**: since prefetching in `HgQueuedBackingStore`
is in progress, these metrics are only set-up for when
prefetching this is finished
Reviewed By: chadaustin
Differential Revision: D20714494
fbshipit-source-id: 392337aaa60b44b75184889e6c92cb5d2c2577cc
Summary:
As mentioned in D20629833, adding metrics for live
imports in `eden top` gives more transparency to the
imports process and makes identifying import related
issues easier. This is set up to expose metrics for live
imports like those for pending imports in `eden top`.
Similar to D20611728 exposing this via these counters
will log this data. Having this data persisted will allow
tracking the performance of imports, and does the set
up for more pro-active fixing of issues. Further we can
look back to see issues that are no longer occurring, but
still of interest.
This also refactors the registration code so that it requires
no copy pasting to add a new counter. Avoiding copy paste
errors when adding more counters and making it easier to
maintain.
Reviewed By: chadaustin
Differential Revision: D20630813
fbshipit-source-id: 8a7a2a0135c7b7a5cde960b84dcb434c6c99eaeb
Summary: This is actually missing from `HgQueuedBackingStore`. This diff fixes this and assign low priority to these prefetch requests.
Reviewed By: chadaustin
Differential Revision: D20655681
fbshipit-source-id: f3c92b358e16e980390ac7adcae27d41ae5a7277
Summary: Ergonomics improvements. This allows us to have type safety instead of dynamically check for types at runtime.
Reviewed By: chadaustin
Differential Revision: D20740318
fbshipit-source-id: d0a3255105bef22dd236695be9fc9c4220547b83
Summary:
Previously `eden top` shows metrics for imports that are queued
and fetching data (though there is a bug here fixed above), we
want to provide more granularity to help with debugging:
This explicitly separates "current" and "pending" imports:
- "live": The import is live, ie it is currently importing data
- having this metric allows debugging the case that there
is a problem fetching data
- "pending": The import is queued, live, or getting data from the
cache
- having this metric allows debugging the case that there is a
problem initiating the request, for example a request is being
starved on the queue
**note**: I moved the watches closer to the import function call
instead of renaming the currently broken pending import watches
to make it more clear in the code what these are suppose to be
timing
Reviewed By: chadaustin
Differential Revision: D20629833
fbshipit-source-id: 84ef75057149a648a51418a5cc93be87e3b3d6b5
Summary:
Migration from Future-returning executor-erasing collectX forms to
SemiFuture-returning forms, that are less risky in particular with coroutines.
Earlier changes added collectXSemiFuture and collectXUnsafe as a migration
path. We then migrated collectX callsites to collectXSemiFuture or
collectXUnsafe and switched the implementation of collectX to the SemiFuture
form.
This diff renames uses of collectXSemiFuture to collectX, as the two are
identical.
Reviewed By: simpkins
Differential Revision: D20833464
fbshipit-source-id: c14aa40f8cce121632ff922300ef271721c1d695
Summary:
The primary purpose of exposing counters for the maximum length of duration is to add this to `eden top`.
As discussed in D20611704, the duration of time that imports are queued for is a strong indicator of the health of the import process. Adding it to `eden top` will help give users insight into what eden is doing when it hangs for a long time or when there is an issue, indicate whether that issue is due to the import process.
Additionally we choose to use these counters to expose this data because we want to log this data. This data can be used to measure performance of the queueing and import process. In the future this data could be used to set up alerting for regressions and allow more proactive fixes.
Reviewed By: chadaustin
Differential Revision: D20611728
fbshipit-source-id: 9307c1ad749ac5fe356ba9eaf868de41b1a8a3a7
Summary:
Here we are tracking the duration of each of the current imports in `HgBackingStore`. This changes is setup to display the duration of the longest outstanding import in `eden top`.
Displaying the duration of the longest import in eden top will help debug where the problem is when eden hangs. These specifically help determine if eden is hanging because of a queueing issue or not being able to fetch an object from the server.
This is a stronger signal to the user that there is an issue in importing over just a count of imports (the previous addition) because many pending imports is not always abnormal. For example a program may prefetch a big batch of files to read, and this could cause a large number of pending imports. A single import should not hang due to a big prefetch and other normal uses.
Thus we want to display this in addition to the number of imports and use this as our warning indicator. (i.e. a high duration of import will mark the metrics as yellow or red in `eden top` to give a warning to the user)
*code changes*:
- adding lists to store watches to time the duration of each of the imports in `HgBackingStore`
- removing the counters previously used to count the number of imports in `HgBackingStore` in favor of using the size of these lists as this is constant time, so there is no benefit to tracking this value ourselves
Reviewed By: chadaustin
Differential Revision: D20611704
fbshipit-source-id: 8bbe0aed8d10688f7bfb7ecfa2fa5ac181945a76
Summary:
fixes up some functions to avoid narrowing warnings
and others to adopt `folly::to_narrow` to accept them.
This reduces the number of warnings in the windows build.
Reviewed By: chadaustin
Differential Revision: D20562026
fbshipit-source-id: f18ba50f914e142415d1efefebc6297e8c68d38e
Summary:
This is a rough pass that resolves a linker issue on MSVC by
switching to inline static member functions.
Reviewed By: chadaustin
Differential Revision: D20529163
fbshipit-source-id: 578ed440758c685091d3e039e261638e027db17a
Summary:
s/CURLE_SSL_CONNECT_ERROR/CURLE_RECV_ERROR/.
Note CURLE_SSL_CONNECT_ERROR is not explicitly checked in
code in the eden hierarchy (or anywhere else significant in fbcode).
Reviewed By: meyering
Differential Revision: D20490844
fbshipit-source-id: be4a66f49eb4a9eabaf73785f9a203f0aa6905a4
Summary: This diff bumps the open call from FUSE to High priority (which is higher than any other blob open request atm). This has shown improvement on the user experience of EdenFS when it's importing many other things from other channels (thrift, etc.)
Reviewed By: chadaustin
Differential Revision: D20287389
fbshipit-source-id: 319bc44ef8be5c904d7cf0db7cc2f8be28b4760a
Summary: This diff implements `HgQueuedBackingStore` that uses `HgImportRequestQueue` to provide SCM data importing with priorities. This can allow us to customize how we import things and batch importing.
Reviewed By: chadaustin
Differential Revision: D19184826
fbshipit-source-id: da579b5bbff0b1449e9689e2c0159d4a3a475a83
Summary: This diff adds `HgImportRequestQueue` that is responsible for managing incoming requests by their priorities. This queue is later used in the `HgQueuedBackingStore` to prioritize works to the workers of the backing store.
Reviewed By: chadaustin
Differential Revision: D20197069
fbshipit-source-id: 246bbc086054a8021226e9ba6ab26d3bf0cfb7a3
Summary:
This class is used to represent an import request that will be used later in the queue implementation.
When the EdenFS needs to import a blob, it creates an instance of this request and send it to the worker. Then it waits for the promise associated with the request.
In the future, we should be able to change the owned `Promise` into a non-owned `SemiFuture` to a `Promise` somewhere else for merging repetitive import requests.
Reviewed By: chadaustin
Differential Revision: D19184824
fbshipit-source-id: 823aabbed1156acf6306b7aefc76580a540d310d
Summary: This diff adds `Priority` added in the previous diff to the `BackingStore` interface with the default value set to `Priority::Normal`.
Reviewed By: chadaustin
Differential Revision: D20197071
fbshipit-source-id: a92f1b49bb82e3478042e5e3b79b047d834755ea
Summary:
This diff introduces a `Priority` type for EdenFS. This type is used to pass along the priority of a request.
The priority class itself contains two parts, `kind` and `offset`. `kind` uses the first 4-bytes and the reset 12-bytes are used to store offset. The idea is that we can roughly assign a priority kind to most of the requests and offset is used to dynamically tweak the priority of some particular requests. For example, when we saw a process is generate millions of requests we can use this to express "normal priority but less important than other process's normal priority".
Reviewed By: chadaustin
Differential Revision: D20287652
fbshipit-source-id: 9a849fb6cc6ba5e443fea978d5b4dc3ab8ca906a
Summary:
Migration from Future-returning executor-erasing collectX forms to
SemiFuture-returning forms, that are less risky in particular with coroutines.
Earlier diffs added SemiFuture and Unsafe versions. This codemod migrates
collect versions to the Unsafe versions to allow the basic collect versions to
be made safe.
Reviewed By: simpkins
Differential Revision: D20331206
fbshipit-source-id: efc8dff487d45f7d53ee55e8c4696bd3eed0e6da
Summary: expose the counters for number of pending imports (blobs, trees, prefetches) to allow use in tooling
Reviewed By: chadaustin
Differential Revision: D20269853
fbshipit-source-id: d2b7e2110520290751699c4a891d41ebd5b374cf
