Summary:
When unwrapping Try it is convenient to be able to build an ImmediateFuture
directly from the .exception() call.
Reviewed By: chadaustin
Differential Revision: D32333774
fbshipit-source-id: 0066ca7eddc4a57010f60974c234b57691b73d77
Summary:
For some reason, in some cases the compiler wants to use the folly::collectAll
function instead of the ImmediateFuture one. Adding the namespace solves this
issue.
Reviewed By: chadaustin
Differential Revision: D32329807
fbshipit-source-id: 22ee9b20cf33e5718c9bb0270dd93f3f857b0967
Summary:
We already had a collectAllSafe for argument packs, let's have a similar
function when the argument is a vector.
Reviewed By: chadaustin
Differential Revision: D32323573
fbshipit-source-id: 2cfa3da565a7d9bb88746e3cc6b10d1d8890c05e
Summary:
We should all be migrating to platform010, for the improved performance of its
generated code and for its improved diagnostics/portability.
add an explicit constructor, now required for C++20, see https://en.cppreference.com/w/cpp/language/default_constructor
This avoids the following errors:
eden/fs/utils/test/ImmediateFutureTest.cpp:335:17: error: no matching constructor for initialization of 'Uncopyable'
# Context:
The cost of delaying a platform upgrade is now estimated at about $3M/week.
For the preceding one, it was measured at [**$1M/week**](https://fburl.com/jfmpsqmy).
Differential Revision: D32250064
fbshipit-source-id: 8c6990489cafb09629c03e56bdbae92395d6f85f
Summary:
C++20 changed the rules around char8_t. Cast to char* as
necessary. Also, fix references to some removed std::allocator
associated types.
Reviewed By: fanzeyi, meyering
Differential Revision: D32155762
fbshipit-source-id: dc72e5bc58e8c08612ea1a1eb2568861701287c6
Summary:
Takeover data needs to include fuse data for fuse
mounts and NFS data for NFS mounts.
Fuse related data is already wrapped up in
FuseChannelData. we are making an NfsChannelData to
mirror this for NFS. These structs will be used in
TakeoverData in a later commit.
Reviewed By: xavierd
Differential Revision: D31740475
fbshipit-source-id: f64367a5881175b9e20d261e43ec7570c498f380
Summary:
Some compiler (msvc, old gcc) can't seem to figure out that these switch cases
are exhaustive and thus warn about them. Let's help it to silence the warnings.
Reviewed By: chadaustin
Differential Revision: D32022198
fbshipit-source-id: 67f89d52211db45a09e8a2fc0ed64da667b8cbbc
Summary:
To avoid any risk of SSLKEYLOGFILE being used to unexpectedly write
files as root, ensure the privhelper does not even link against
OpenSSL.
Reviewed By: fanzeyi
Differential Revision: D31944839
fbshipit-source-id: 3280657a10f4a468ddb63a874d38198bd333e01c
Summary:
Ignoring the return value of this method is equivalent to dropping the future
entirely, which isn't what is usually desired. Let's make sure the compiler
helps us in finding these cases as code is converted.
Reviewed By: chadaustin
Differential Revision: D31945362
fbshipit-source-id: dd646365e00d3ebb04ed9b6803c561fc11ff4a14
Summary:
This mimics the one eden/fs/utils/Future.h behavior where all the Try are
unwrapped to simply return a tuple.
Reviewed By: kmancini
Differential Revision: D31668773
fbshipit-source-id: f4b9b6b2564cc1e2c43aa5b03bfb06048f0a39c1
Summary:
In some places, EdenFS uses folly::collectAll passing futures to it. In order
to replace it, we need an equivalent for ImmediateFuture.
The implementation relies on the fact that ImmediateFuture's constructor will
extract the value from a SemiFuture if it is ready, and thus the code doesn't
have to special case when all the passed in ImmediateFuture are immediate
values.
Reviewed By: chadaustin
Differential Revision: D31667897
fbshipit-source-id: ea680e1c74528d1eb34210cacb58efda8a26fee0
Summary:
This will cut some of the boiler plate of returning an ImmediateFuture in the
error case.
Reviewed By: chadaustin
Differential Revision: D31769752
fbshipit-source-id: 42be026aee6a6071bbb2c1f8af918b107d593eab
Summary:
Exception cases should be rare, but EdenFS has a couple of spots where
exceptions are somewhat expected. One of which being a lookup failure due to a
non-existing file. Due to the overhead of these lookup failures, EdenFS is
actually spending as much time resolving these, than resolving successful
lookups.
The reason for the high overhead is due to the ImmediateFuture::thenValue just
relying on an exception being thrown when accessing the stored Try value. We
can do better than this by testing the Try before hand, and bailing out in that
case.
Reviewed By: fanzeyi
Differential Revision: D31655223
fbshipit-source-id: 29da7ef38a19f09c91c931e7e8e080e5ed6af269
Summary:
When dealing with a large amount of futures, one of the common pattern is to
store all the pending one into a vector and collect them all at once. Let's add
a similar function for ImmediateFuture to allow converting code that make use
of this pattern.
Reviewed By: chadaustin
Differential Revision: D31452296
fbshipit-source-id: 04aee8f358727e02b4202d09d78fafc0917b8dec
Summary:
In a meeting today, the question arose "when should one use Future,
SemiFuture, or ImmediateFuture?" To help clarify the situation,
reframe some of ImmediateFuture's doc comments in terms of their
semantics.
Simultaneously, replace hasImmediate with isReady, which returns true
when a value can be retrieved without blocking.
Also, allow converting a completed SemiFuture into an ImmediateFuture
which is also isReady.
Reviewed By: xavierd
Differential Revision: D31563622
fbshipit-source-id: b942031c15d46948b1f60ade9f8043fa19442b65
Summary:
This is fairly mechanical diff that finalizes split of Hash into ObjectId and Hash20.
More specifically this diff does two things:
* Replaces `Hash` with `Hash20`
* Removes alias `using Hash = Hash20`
Reviewed By: chadaustin
Differential Revision: D31324202
fbshipit-source-id: 780b6d2a422ddf6d0f3cfc91e3e70ad10ebaa8b4
Summary:
While std::variant is convenient, they are both slow to compile, and the
compiler cannot optimize it as well as a manually written tagged union. Since
ImmediateFuture is performance critical for EdenFS, let's use a tagged union
and speed them up by an additional 40%.
Reviewed By: chadaustin
Differential Revision: D31272296
fbshipit-source-id: e34be4489a596d3577b3bd900a1f20d6c7d8b693
Summary:
The max duration would cause UBSAN failures due to folly's SemiFuture code
multiplying the value which understandably cannot be represented. Splitting the
function is easy and avoids the problem entirely.
Reviewed By: genevievehelsel
Differential Revision: D31272297
fbshipit-source-id: c15ca70ad771c11b4f68bb9974422c0986d4928b
Summary:
folly:format is deprecated in lieu of fmt and std::format. Migrate
most of EdenFS to fmt instead.
Differential Revision: D31025948
fbshipit-source-id: 82ed674d5e255ac129995b56bc8b9731a5fbf82e
Summary: This is a similar diff to D30915090, but for EdenFS.
Differential Revision: D30915126
fbshipit-source-id: 9a718e47237924ebe20176c522a1b1193224236c
Summary:
During a diff operation, files that have a case change should not reported as
changed for case insensitive mount. This is a follow up to D29150552 (37ccaa9231) where the
TreeInode code was taught to ignore case changes for case insensitive mounts.
Reviewed By: kmancini
Differential Revision: D29952464
fbshipit-source-id: e5fa8eccc1c1984a680838fd8589a0278989d9d0
Summary:
TSAN complains that pipes_ is read and written in different threads without
proper synchronization. To avoid this, we can simply close the FileDescriptor
without removing it from the pipes map as this achieve the same result: it
notifies the reader that the endpoint is closed.
Differential Revision: D29924043
fbshipit-source-id: be92630799bb5c78089dbe85f9c2f02f937300bc
Summary:
This adds debug commands for ActivityRecorder:
```
eden debug start_recording --output-dir <DIR>
* stdout: the id of the profile
eden debug stop_recording --unique <ID>
* stdout: the output file path
```
Users can record multiple profiles concurrently. Each profile is identified by the timestamp when it started.
Reviewed By: genevievehelsel
Differential Revision: D29666359
fbshipit-source-id: 487ca67de77378a8141bc4ac46b9abd1375ffd23
Summary: This adds counters for memory and disk counts in addition to import count so that we can understand cache hit rates during local investigation or output this in ActivityRecorder.
Reviewed By: genevievehelsel
Differential Revision: D29805637
fbshipit-source-id: 34261f91c33d6bd4bcb4b85b17d2e68360410896
Summary: NfsTaskQueue can be made more generic to be shared across the codebase, so this makes it its own target in `eden/fs/utils` w/ the name EdenTaskQueue.
Reviewed By: xavierd
Differential Revision: D29244762
fbshipit-source-id: 78348f2ff8fa66bc801aefe7d6b3905e0da278e8
Summary:
Using a PathMap to hold the list of mounts gives us case sensitivity for free
without needing to have #ifdef _WIN32, we can also avoid copying the paths just
to normalize them.
Reviewed By: fanzeyi
Differential Revision: D28969795
fbshipit-source-id: 859cdc3881e36db120913099ce3f451940fa48a4
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: fanzeyi
Differential Revision: D28663227
fbshipit-source-id: cee18ec28283c0aef18151465a3c13e9be604d7d
Summary: FileDescriptor.cp throws some warning on comparison of signed long and unsigned long. So fix these warning and follow how ```folly/portability/SysUio.cpp:doVecOperation``` did
Reviewed By: fanzeyi
Differential Revision: D28608581
fbshipit-source-id: 3b6e35e9764548ce470634f16a1f6eec5118d7ed
Summary: This will be used to replace calls to folly::makeFutureWith.
Reviewed By: chadaustin
Differential Revision: D28515786
fbshipit-source-id: 2c2c542392e8e57b8f865173d6878cb9d00ba376
Summary:
When the caller asks to not perform sanity checking, this should be honored as
the caller can for instance tell that a path is guaranteed to be valid and
shouldn't be checked.
When running the newly added glob benchmark, EdenFS was spending 7.5% of the
time validating a path that was constructed via the operator+.
Reviewed By: fanzeyi
Differential Revision: D28612665
fbshipit-source-id: 282155d0415ab6458bd9307c89a24e9e090bf09d
Summary:
Lambda returning no values would cause the compiler to try to instantiate an
ImmediateFuture<void>, which doesn't compile. We could try special casing the
void type, but it's easier if we simply consider these lambda to return unit.
Reviewed By: genevievehelsel
Differential Revision: D28372253
fbshipit-source-id: 1368ae5dc5e2d4d6a5c3e31bc87ed7d230027c3a
Summary:
This is to mimic the folly::Future API which allows the `get` and `getTry` to
throw if the Future isn't ready. One difference is that the ImmediateFuture API
has a default argument of a max duration instead of having a separate method.
Since chrono durations are expressed as intmax_t (64-bits on systems supported
by EdenFS) a max duration is virtually infinite.
Reviewed By: chadaustin
Differential Revision: D28424053
fbshipit-source-id: 319493174f31367184dbe0aa811a97145b0310cf
Summary:
This allows the macro to be used for both folly::Future, but also with
ImmediateFuture, reducing the cost of migrating to the latter.
Reviewed By: chadaustin
Differential Revision: D28302478
fbshipit-source-id: d8470428ecaa6aaccf2aa884437ca97d4b45806f
Summary:
Since lambda are default capturing by const, Func was also captured that way,
which made it impossible to call if Func was actually a mutable lambda.
Reviewed By: chadaustin
Differential Revision: D28297423
fbshipit-source-id: b9c6bdcf024c2e219ec0f8f5be2379e51df24db0
Summary:
This code initially was for thenValue, but when I converted it for thenTry I
forgot to also remove the hasException case. We were never calling the callback
on a Try that had an exception.
Reviewed By: chadaustin
Differential Revision: D28302477
fbshipit-source-id: 755fb2c8928627fbe1883f3863cafc360e34a038
Summary:
When the passed in callback returns an ImmediateFuture, the code had a small
typing bug where we would try to return an ImmediateFuture<ImmediateFuture<T>>
while the function signature expected an ImmediateFuture<T>. This is due to the
SemiFuture code not coalescing a SemiFuture<ImmediatureFuture<T>> into a
SemiFuture<T>.
Reviewed By: chadaustin
Differential Revision: D28293942
fbshipit-source-id: 1bc8f58d387766f29e573499fb37402ff9b49c83
Summary:
gtest includes some windows headers that will have conflicts with the
folly portability versions. This caused some issues in my in-memory tree
cache diffs (D27050310 (8a1a529fcc)).
We should probably generally be using the folly portable gtests so we can
avoid such issues in the future.
see here for more details: bd600cd4e8/folly/portability/GTest.h (L19)
I ran this with codemod yes to all
- convert all the includes with quotes:
`codemod -d eden/fs --extensions cpp,h '\#include\ "gtest/gtest\.h"' '#include <folly/portability/GTest.h>'`
- convert all the includes with brackets
`codemod -d eden/fs --extensions cpp,h '\#include\ <gtest/gtest\.h>' '#include <folly/portability/GTest.h>'`
- convert the test template
`codemod -d eden/facebook --extensions template '\#include\ <gtest/gtest\.h>' '#include <folly/portability/GTest.h>'`
then used `arc lint` to clean up all the targets files
Reviewed By: genevievehelsel, xavierd
Differential Revision: D28035146
fbshipit-source-id: c3b88df5d4e7cdf4d1e51d9689987ce039f47fde
Summary:
Due to NFS being designed as a network filesystem, it default to binding on a
TCP socket. For EdenFS, since we're not expecting to mount an actual remote
filesystem, we bind these sockets to localhost. Unfortunately, TCP sockets have
some inherent overhead due to being designed to be reliable over a non-reliable
medium.
On macOS, Apple provides a way to mount an NFS server that is listening on a
unix domain socket. Thanks for unix socket being reliable, the TCP overhead
goes away leading to some higher throughput and lower latency for the NFS
server. For EdenFS, timing `rg foobar` over a directory containing 27k files gives:
NFS over TCP:
rg foobar > /dev/null 0.80s user 5.44s system 567% cpu 1.100 total
NFS over UDS:
rg foobar > /dev/null 0.77s user 5.27s system 679% cpu 0.888 total
osxfuse:
rg foobar > /dev/null 0.87s user 5.59s system 662% cpu 0.975 total
The main drawback of going through a unix socket is that D27717945 (bcf6aa465c) appears to
no longer be effective due to
8f02f2a044/bsd/nfs/nfs_vfsops.c (L3739)
Reviewed By: kmancini
Differential Revision: D28261422
fbshipit-source-id: 25dc1dc78cdb50d6c6550a86ef01ea2c894c110f
Summary:
While ImmediateFuture are expected to be used on values that are mostly
immediate, there are cases where it won't be. In these cases we need a way to
wait for the computation/IO to complete. In order to achieve this, we need to
transform an ImmediateFuture onto a SemiFuture.
Reviewed By: fanzeyi
Differential Revision: D28293941
fbshipit-source-id: 227c0acf1e22e4f23a948ca03f2c92ccc160c862
Summary:
When a T can be default constructed, make an ImmediateFuture default
constructible.
Reviewed By: fanzeyi
Differential Revision: D28292874
fbshipit-source-id: 4c239cc9c3f448652b2bcdc103ea1a81ace46402
Summary: This should help the compiler generate even better code.
Reviewed By: chadaustin
Differential Revision: D28153979
fbshipit-source-id: b1d84c92af4fa760c92624c53d7f57330d7706fa
Summary:
In very hot code path, EdenFS is spending a very large amount of time creating
and destroying folly::Future objects. This is due to the required memory
allocation, as well as the handful of atomics that are happening at creation
time, and these are showing up in EdenFS profiles.
In the steady state, EdenFS actually doesn't need futures, as it often times is
able to service requests from its in-memory caches, in which case we should
ideally just return the value itself and not wrap it in a folly::Future. The
added ImmediateFuture is a step in this direction, as it can hold either an
immediate value, or a folly::SemiFuture, and allow the same API to be used
transparently between these 2.
Reviewed By: genevievehelsel
Differential Revision: D28006802
fbshipit-source-id: 89eaa32e7fa82c44844c4b23c4cb30dbeea46ca8
Summary:
This diff defines symlink type in `DirType`.
Even though it is not directly used in the FSCK diff. This will allow us to support symlink in EdenFS Windows in the future.
Reviewed By: genevievehelsel
Differential Revision: D28016305
fbshipit-source-id: 67c1aa22e39198f9c91845129695f27b8303a5f1
Summary:
These methods will be used in my later Windows fsck diff as it will need to scan disk state to find changes.
It is a bit unfortunate that we'll need to stick with boost for now. However this should be a fairly easy migration to `std::filesystem` once that is available.
Reviewed By: kmancini
Differential Revision: D27872828
fbshipit-source-id: f6b27a171026aeaaea3db9f17b8f43cfa25004e4
Summary:
Copy from Watchman.
This allows us to show stack trace when EdenFS terminates on Windows.
Reviewed By: chadaustin
Differential Revision: D27896966
fbshipit-source-id: f3238a37a1176f879d5e6bc051ec97031c9a7096
Summary:
We started getting the message
```stderr: eden/fs/utils/SpawnedProcess.cpp:798:21: error: 'getIOExecutor' is deprecated: getIOExecutor is deprecated. To use the global mutable executor use getUnsafeMutableGlobalIOExecutor. For a better solution use getGlobalIOExecutor. [-Werror,-Wdeprecated-declarations]
```
I don't see why we would need a mutable executor here so I chose `getGlobalIOExecutor` over `getUnsafeMutableGlobalIOExecutor`.
Reviewed By: kmancini
Differential Revision: D27912276
fbshipit-source-id: 95b1053f72c2b4eb2746e3c40c0cf76b69d90d6e
Summary: `CaseSensitivity::Sensitive` is better than a mere `true` out of nowhere.
Reviewed By: kmancini
Differential Revision: D27867180
fbshipit-source-id: 39d21d3cc3b70c78c6984c3ddbd65b53520770be
