Summary:
After printing the exception backtrace, EdenFS terminates manually.
Unfortunately, this means that Windows can't detect that the process crashed as
it thinks it terminated correctly. In turn, this prevents minidumps from being
created which makes debugging the crash impossible.
Fixing this is no easy task as returning from the exception handler would
always trigger an infinite recursion of unwind somewhere in the CRT code. This
would produce a minidump, but it wouldn't be handy to use. The only solution I
found is to reset the exception handler and invoke one as by default, this
creates a minidump. After that, we can exit EdenFS in the same way as before,
being careful to not call `std::abort` or `std::terminate`.
Reviewed By: chadaustin
Differential Revision: D41362303
fbshipit-source-id: 42422db66a5f8033796ed8cb5b3ccf1a235f1da5
Summary:
SemiFuture is a pointer-sized, noncopyable type. We can assume it's
always nothrow-move, so assert that fact and remove some noexcept
checks.
Reviewed By: xavierd
Differential Revision: D41052209
fbshipit-source-id: d9507bcc7ef4dd4f9c1d621aa75a157ed0ba1c5f
Summary:
Moving a PathPiece, AbsolutePath, RelativePath, or PathComponent
should never throw. Add static_asserts to indicate that.
Reviewed By: kmancini
Differential Revision: D41272812
fbshipit-source-id: 0b4b10f8430f31aafe6229f88995108251e52de7
Summary:
std::string_view has noexcept accessors and folly::Range doesn't, so
this allows us to make Path and PathPiece noexcept.
Reviewed By: kmancini
Differential Revision: D41145426
fbshipit-source-id: 046f6f6a532d8d0da8508ccf7896c914e19a25ec
Summary:
Requiring that callers ensure that ObjectFetchContext outlives the
asynchronous result is a burden and a memory-safety risk. Without
Rust's borrow checker, we can't feel confident this is safe under all
error and timeout conditions. It also requires we use
folly::collectAll and collectAllSafe, which means that an error during
fanout has to wait until the fanout completes before we can report the
error to our caller.
Moreover, tying the ObjectFetchContext's lifetime to the
FUSE/NFS/Thrift/etc. request's lifetime gives us incorrect statistics
about how long the request takes. I consider this an API design
mistake (on my part).
Correct the situation by decoupling ObjectFetchContext from
RequestContext, and begin reference-counting ObjectFetchContext.
This does increase the number of atomic RMW operations, but we can
alleviate those where they matter by manually checking
*Future::isReady() before calling .then*.
Reviewed By: xavierd
Differential Revision: D40744706
fbshipit-source-id: 5baf654f6d65790ed0cafb45c0bc7f2aaa8ecec2
Summary:
AbsolutePathPiece does not guarantee null termination. Therefore, we
must allocate a null-terminated string every time.
Reviewed By: xavierd
Differential Revision: D41099085
fbshipit-source-id: 3b8cb1b2630b9e2c190f8efbaa694e7868723e1b
Summary:
At API boundaries, it's common to want to convert a Path or PathPiece
to std::string. Previously, code would call
`.stringPiece().str()`. Add an `asString()` to make this explicit.
Reviewed By: xavierd
Differential Revision: D41140757
fbshipit-source-id: 72003dc9a353cdfa4632a7d1ab433122211186e0
Summary:
RVO only works when the local and the return type are exact
matches. Here, we're invoking the constructor, so we need to
std::move.
Reviewed By: xavierd
Differential Revision: D41097721
fbshipit-source-id: 52e4425a960ce0aa8ae59a827ddde4e9b58a7f92
Summary:
We won't be on C++20 for a while and we'd like to cut dependencies on
folly::Range where possible. starts_with and ends_with are too useful,
so add a polyfill.
Reviewed By: xavierd
Differential Revision: D41097519
fbshipit-source-id: 298f65b15b8beb6f56ab1b6b9896fc6b1225f0fb
Summary:
On Windows, absolute paths are of multiple shape, with all of them having pros
and cons. In particular, device paths (C:/) that EdenFS is using is limited to
260 characters. To bypass this limitation, UNC paths (\\?\) can be used, but no
normalization is done automatically, specifically '/' aren't converted to '\'.
Since UNC paths allow us to support long paths, and would also bring more
safety to EdenFS by enforcing that all AbsolutePath are of the UNC form, let's
use them.
One of the other downside of UNC is that applications may not support them or
may misbehave when passed to them. For this reason, EdenFS doesn't expose UNC
paths in its Thrift API, changing the path to a device path, applications
relying on Thrift are expected to re-UNC them.
Reviewed By: chadaustin
Differential Revision: D40818724
fbshipit-source-id: 260b8732df04bbffee6b2fe52f0ceda222d3941b
Summary:
We can't use std::source_location just yet, but I'd like to switch
ThriftLogHelper to it. Therefore, introduce a polyfill for the near
term.
Reviewed By: genevievehelsel
Differential Revision: D40701594
fbshipit-source-id: 4a39644439df15300df4a5bc303b7f2e87d0bf15
Summary:
Add a silly microbenchmark that compares the innate overheads of
unique_ptr, shared_ptr, and RefPtr.
It's primarily a malloc() benchmark, and the results have the same
caveats that all microbenchmarks do, but we can illustrate that
RefPtr's single-threaded use case is on par with unique_ptr.
Reviewed By: genevievehelsel
Differential Revision: D40355895
fbshipit-source-id: f1d620df14a028749df8ae8893eaf86e25eb5797
Summary:
In cases where a function takes a `const Ptr<Base>&` but we have a
`Ptr<Derived>`, allow access to the `Ptr<Derived>` through a `const
Ptr<Base>&` without bumping the reference count.
Reviewed By: kmancini
Differential Revision: D40351346
fbshipit-source-id: 534607f74264ed1c18557dc28a6688e7a9c992a4
Summary:
Windows exception (like access violation) come with parameters. These
parameters tell you whether a read or a write was attempted, and the
faulting pointer value. It's very helpful when debugging crashes, so
log all exception parameters too.
Reviewed By: xavierd
Differential Revision: D40780025
fbshipit-source-id: bc221555b4f615c3146c7c3a100d898046ad9733
Summary:
I plan to improve the safety of how ObjectFetchContext is passed
through EdenFS by making it reference-counted. First, let's introduce
some lightweight reference-counting that avoids any atomic RMWs at all
in the case that an object is never shared.
Reviewed By: kmancini
Differential Revision: D39978716
fbshipit-source-id: d42dbbf4d2bae5a55b67b40149b62dc461badf78
Summary:
The checkout code was designed with the assumption that concurrent reads/writes
to the working copy would block on locks that checkout holds, thus preventing
races. On Windows, this assumption is slightly broken as reads to the working
copy are served by the tree hierarchy not by the inode hierarchy. Reads are
then reflected back to the inode hierarchy in a background manner to increment
the FS refcount. If, somehow a directory is read at the same time as the
corresponding directory entry for it is being invalidated in checkout, checkout
will attempt to invalidate the directory, but since the directory isn't empty
the invalidation would fail, failing checkout.
To solve this, we could simply pretend that the directory is loaded and
recursively run checkout on it.
Reviewed By: kmancini
Differential Revision: D39596117
fbshipit-source-id: 95a42553b8779d904f57f9d168197398f5d45436
Summary:
As chadaustin pointed, logical not-equal are easier to read than xors. Let's
use the former.
Reviewed By: genevievehelsel
Differential Revision: D40366550
fbshipit-source-id: b9c0f23e47ba40ddc16bd578b303008987a41b95
Summary:
In the case where the promise associated with a SemiFuture is set after the
ImmediateFuture construction, EdenFS will still attempt to push the SemiFuture
to an executor. Since this is quite expensive, let's handle this case by
considering the ImmediateFuture as ready.
In very high fanout computation (like glob), this is expected to reduce the
amount of futures pushed to the Thrift executor. This is expected to help
reduce the amount of Queue Timeout in `glob(**/*)` cases.
Reviewed By: genevievehelsel
Differential Revision: D40312497
fbshipit-source-id: e48efac2e4f2640b6a7ad50957ee7c723772ec81
Summary:
In the following change, ImmediateFuture contains an additional optimization to
treat ready SemiFuture as an immediate value. Thus makeNotReadyImmediateFuture
can no longer be written with the help of an initially unset Promise. To
achieve the same result, we can make the ImmediateFuture class be aware of
never-ready SemiFuture.
Reviewed By: genevievehelsel
Differential Revision: D40322423
fbshipit-source-id: 4faa0c513d1e6afa26951dd00bb8015cc45ac510
Summary: It is no longer necessary to pass a reference to the caught exception along with `std::current_exception()`.
Reviewed By: xavierd
Differential Revision: D40252106
fbshipit-source-id: 08bfbea675c60097050f3f026af308fcf0a240df
Summary:
EdenError is another header file that pulls in folly/Conv.h. It also
uses folly::to's concatenation capabilities. Migrate it to fmt just
like Throw.h.
Reviewed By: genevievehelsel
Differential Revision: D39992358
fbshipit-source-id: 09d1d3446f664ec55b91e5b37ee9299ef253e991
Summary:
fmt never takes ownership of its arguments. Therefore, we can always
pass by const reference.
Reviewed By: genevievehelsel
Differential Revision: D39992308
fbshipit-source-id: 03713338530cbc1d775c6a0089bce140cf0ea030
Summary:
The main non-mechanical conversion from folly/Conv to fmt is replacing
folly::to<std:string>'s variadic concatenation operation with
fmt::join. Now that that's solved, we can slowly migrate the rest.
Reviewed By: genevievehelsel
Differential Revision: D39992102
fbshipit-source-id: 81df8e805c78ea5ca80ad5977f20b19e5a7467a3
Summary:
I don't know the difference between our prior magic incantations and
the new magic incantations, but the old ones crazy errors when writing
variadic wrappers that eventually call into fmt::format or fmt::join.
Reviewed By: vitaut
Differential Revision: D39991902
fbshipit-source-id: 8ca3215267912b3bb0ddf586681d3eb5294f52ff
Summary:
I'm having some confusing conflicts between fmt and our PathPieces. To
at least make the error messages clearer, migrate them to
std::string_view.
Reviewed By: genevievehelsel
Differential Revision: D39991418
fbshipit-source-id: 34d0f1ee23618da4595ee1d072e9cdf4f51736b6
Summary:
The build was broken due to incompatible types being used. Clang doesn't
complain about these, but msvc does.
Reviewed By: fanzeyi
Differential Revision: D39561417
fbshipit-source-id: 605e8cb5537defc31c21b54e67317ceeb8935674
Summary:
While EdenFS now properly handle socket files written to the working copy, the
FSCK code was not, thinking socket files are placeholders. For socket files
already present in the working copy this means that `eden doctor` would
complain about them.
Reviewed By: fanzeyi
Differential Revision: D39442575
fbshipit-source-id: 6dfb8fefe67d80dab32960d39c0b91f8148e2345
Summary:
In some cases, it's desirable to be able to construct an ImmediateFuture<T>
from another type, like in case of inheritence where T is the base class. In
this case, a static_cast is required for the compiler to not error out, instead
we can slightly change ImmediateFuture constructor to allow this conversion.
A couple of manual ImmediateFuture constructors were no longer necessary and
thus removed as a result.
Reviewed By: genevievehelsel
Differential Revision: D39158019
fbshipit-source-id: eb3452ba33d941251d9b4efaee28f82fd2c2dbd3
Summary:
It's not possible to differentiate NFS m1 and intel users in edenfs_events
right now. We are currently rolling out to intel users and it would be good
to understand the real number of intel users on NFS vs fuse.
Let's log the cpu architecture so that we can differentiate them.
Reviewed By: mshroyer
Differential Revision: D39139996
fbshipit-source-id: 582804707a7816c41aa6c69594f7198d3ce5dfe1
Summary:
There are cases where the code doesn't wish to execute the ImmediateFuture
continuation inline. For instance, if the code wants to spawn many CPU
intensive futures to be executed in background threads and then collected,
executing these inline would make the code slower.
One easy way to solve this would be to simply use SemiFuture and push them to
background threads, but the rest of the code would need to be ported to use
SemiFuture all over the place. The use of SemiFuture would also slow down the
callers that wish to execute the continuation inline, which isn't desired.
Thankfully, we can achieve both by using ImmediateFuture, the caller can just
decide if it wants the code to execute inline (a regular ImmediateFuture), or
not (by building a not ready ImmediateFuture).
Reviewed By: chadaustin
Differential Revision: D37330741
fbshipit-source-id: 911827f12725d7a0d64986737d02026386fe459e
Summary:
Due to their eager nature, an ImmediateFuture will execute future continuation
immediately when constructed with either an immediate value, or a ready
SemiFuture. One of the drawback of doing that is that tests often manipulate
data that is already ready, and thus we may rarely execute the not-ready code
path. This can lead to use after free when the capture list of a continuation
is not properly set.
To avoid this, we can make ImmediateFuture always be a wrapper on a SemiFuture,
even when constructed from an immediate value. This way, the use after free
described above will always occur in debug/sanitized builds, and
AddressSanitizer will tell us when these occur.
Reviewed By: kmancini
Differential Revision: D37729622
fbshipit-source-id: bcd858f821cc39d63fb446b0891a067bee370a22
Summary:
Looking at procexp, EdenFS appears to be holding handles of dead processes.
After forcing a process to be spawned by SpawnProcess and killing it, I can see
that the handle doesn't go away. This is due to missing CloseHandle of the
process itself.
Reviewed By: chadaustin
Differential Revision: D38472410
fbshipit-source-id: b9d22ec52e7e9f1d33ff923d47cba49a3135067c
Summary:
Fixes https://github.com/facebookexperimental/edencommon/issues/3
`utils` is a bit too generic and Gentoo seems to be building edencommon into a shared library.
Reviewed By: chadaustin
Differential Revision: D38719753
fbshipit-source-id: fb46b6a7c9d3bcc3034765cb47e997a80c646b3d
Summary: This file handle was being opened but never closed. Let's expose the FileUtils.cpp FileHandle helper so we can use it to close the handle when it goes out of scope.
Reviewed By: xavierd
Differential Revision: D38556231
fbshipit-source-id: f1038eb1b802ebb193116edb4e537945ff4ac737
Summary:
From the folly format docs
> Use fmt::format instead of folly::format for better performance, build times and compatibility with std::format
eden build times have gotten a bit high, cutting out folly format will help reduce build time, so let's start
by banishing it from eden
see the last change in this group for the difference
Reviewed By: xavierd
Differential Revision: D38250999
fbshipit-source-id: bfa6debfb9cc9abb8634df69701658308645e6f1
Summary:
From the folly format docs
> Use fmt::format instead of folly::format for better performance, build times and compatibility with std::format
eden build times have gotten a bit high, cutting out folly format will help reduce build time, so let's start
by banishing it from eden
see the last change in this group for the difference
Reviewed By: xavierd
Differential Revision: D38251000
fbshipit-source-id: a5aee79f7f0acbf969a351c3aa5ac32bb6c01b2a
Summary:
Replace `codePointToUtf8` with `appendCodePointToUtf8` (D38305991)
where it makes sense.
Reviewed By: luciang
Differential Revision: D38306295
fbshipit-source-id: 6203f3509cc5bedd8cf8d484305ac8be583c16d6
Summary:
I was iterating on a single unit test and noticed our compilation times have
crossed the threshold from bad to horrific. I was seeing 30+ seconds per unit
test .cpp file.
After playing around with -ftime-trace, I found some obvious low-hanging fruit:
- forward declaration opportunities
- pImpl
- moving some implementation to cpp files
Some bigger opportunities remain:
The folly/portability/GTest.h and folly/portability/Windows.h header situation
isn't great. They pull in winsock2.h which alone takes two seconds to compile.
We also probably don't need the folly/portability/Unistd.h compatibility
wrappers in EdenFS or Watchman.
Also, folly/Format.h is quite expensive, and there are other dependencies that
pull in Boost MPL.
Reviewed By: xavierd
Differential Revision: D38195736
fbshipit-source-id: 9c64bab5ff5851d5a896674712aec71d6780c79a
Summary:
In D37422971 (3d9636fe8c), it was shown that moving a path will invalidate the pieces on the
moved path. Unfortunately, this can only be observed when the path is short
enough to fit in the SSO of std::string. To force this to be true even without
SSO, we can pessimize moves into copies and leverage the use of ASAN to tell us
if the code is misbehaving.
Reviewed By: kmancini
Differential Revision: D37705144
fbshipit-source-id: a4446632b0e5d3b422226e85c324059c5b8cf379
Summary: moves `void` `TreeOverlay` functions to the `BufferedTreeOverlay`'s worker thread and waits on non-`void` functions. This maintains ordering with a `flush` which puts a no-op wait onto the `BufferedTreeOverlay`'s worker thread.
Reviewed By: chadaustin
Differential Revision: D35634643
fbshipit-source-id: fe080b4940e17fc6328f7246179185dcc7d16f70
Summary:
Currently the eden trace inode live and retroactive commands only display inode numbers in order to identify the inode whose event is being referenced. In order to help users more easily debug Eden, it would be helpful to also provide Inode names and/or inode paths to these commands. While computing inode paths can potentially be an expensive operation, we get over this by computing these paths outside of the critical path when inode events get published into tracebus and instead compute the paths in the thrift service handler for all inodes as they are needed. These paths are computed using the EdenMount's InodeMap.
However, as a consequence this might mean path information is not always available, namely during a start event when creating a new file, directory, or symlink while the live `eden trace inode` command is being run, or after an inode has been removed after its materialization event has occurred in the `eden trace indeed --retroactive` command. In these cases, an empty string is shown for the path, though a follow up diff is planned that will add file name information (not full paths) to each inode event as they occur that will be shown when full path information is unavailable.
Reviewed By: MichaelCuevas
Differential Revision: D37961108
fbshipit-source-id: 88d5d303eb5afc616ff46bc083cd96b55c572cfa
Summary:
To simplify the build and fall in line with Folly's conventions,
remove several conditional includes. Instead, headers conditionally
define symbols.
Reviewed By: xavierd
Differential Revision: D37897850
fbshipit-source-id: 8b52d5310f5cd8cb17fdc013117c271ae09dd5d6
Summary:
In most cases, the type held by an ImmediateFuture is moveable in a noexcept
fashion. However, several debugability improvements are being added to both
PathFuncs and ImmediateFuture which makes this no longer true in
debug/sanitized builds. Thus let's go the more complex route of properly
specifying the noexcept qualifier.
Reviewed By: kmancini
Differential Revision: D37840356
fbshipit-source-id: 452dd2b65ae813ce24617753d5f51ccfe00eb184
Summary:
When the ImmediateFuture is holding a SemiFuture, the compiler complains about
the use of `std::forward`, let's fix that and add few tests to verify that it
now compiles well.
Reviewed By: fanzeyi
Differential Revision: D37861191
fbshipit-source-id: 590f22f1fdbfda7e7106730a7d77ddfd5b08d815
Summary:
I want to test how Eden recovers from an ungraceful process exit. Let's
enable FaultInjector to kill the process.
Reviewed By: xavierd
Differential Revision: D36986502
fbshipit-source-id: 3a92a16e7ed11a8afbe2dc81f84f43666d5bd73f
Summary:
While using thenError in the following diff, I realized that the thenError code
had a bug that failed compiling. The added test is an example of a similar code
that would fail to compile despite being perfectly valid.
As a bonus, this also makes thenValue slightly more efficient when dealing with
exceptions in the SemiFuture case.
Reviewed By: chadaustin
Differential Revision: D37824174
fbshipit-source-id: 2f759a65ebffce456f8f4f13ea4792ce078ae926
Summary:
I've always been annoyed at needing to include folly/Conv.h or fmt to
throw a formatted exception. And while exceptions are rarely in the
hot path, it's still worth having the option to optimize the
formatting and exception allocation step at some point in the future.
I'm sure I missed some, but this diff migrates most of our
`throw T{fmt::format(...)}` patterns to `throwf` and
`throw T{folly::to<std::string>(...)}` to `throw_`.
Reviewed By: genevievehelsel
Differential Revision: D37229822
fbshipit-source-id: c053b5cdaed99e58c485afd2a99be98828f07657
Summary:
In order to avoid copying paths and allocation, this code cleverly uses path
piece on a path that is moved and will live for longer than the path pieces.
Unfortunately, the assumption on which this optimization was built is not
always true. To understand more, we need to dig onto the implementation details
of RelativePath and of std::string.
A RelativePath is merely a wrapper around an std::string, and a
RelativePathPiece is also just a wrapper on top of a folly::StringPiece (which
has similar semantic as std::string_view), that is, a RelativePathPiece merely
holds a view of to the underlying std::string buffer. When a RelativePath is
being moved, the held std::string is also moved, which in most cases prevents
copying and re-allocating a string, this makes the move operation fairly cheap
and since the RelativePathPiece were a view on that first string memory
allocation, these are still viewing valid and allocated memory.
However, std::string have an optimization where small strings aren't heap
allocated, but are stored in the std::string object itself, this is called SSO
for small string optimization. In this case, a folly::StringPiece is no longer
a view on the heap allocated memory, but on that SSO memory. What this means is
that moving a SSO std::string will make the folly::StringPiece invalid as it
would no longer point to valid memory!
As mentioned above, RelativePath and RelativePathPiece are just wrappers on top
of std::string and folly::StringView, and thus the same caveat exist. This
means that the optimization performed in handleNotPresentFileNotification is
correct only for RelativePath long enough to not trigger the SSO.
Since this pitfall is extremely easy to fall into, follow up changes will
attempt to reduce the likelyhood of falling into them.
Reviewed By: genevievehelsel
Differential Revision: D37422971
fbshipit-source-id: 610bcf955d53498710b79f12754cd2b1344bc001
Summary:
Intended as a more lighteight version of thenTry when only an exception needs
to be handled.
Reviewed By: chadaustin
Differential Revision: D37476671
fbshipit-source-id: 6d8e1a24cb8eaccf60bd9715abaebf81c7b367d9
Summary:
This change finishes the implementation for the `eden trace inode --retroactive` subcommand. This subcommand allows users to retroactively see information about past inode events, which will help with retroactively debugging EdenFS when things don't go as expected. Currently only start and finish times for inode materialization events are shown, though we intend to generalize this with other types of inode state transitions (i.e. loading inodes) as well.
This change specifically builds off of D37288805 (13ab48d2ab), and utilizes the thrift endpoint built there for retrieving retroactive inode events in order to properly display them.
Example of command output
{F746522409}
Reviewed By: genevievehelsel
Differential Revision: D37356739
fbshipit-source-id: 7d46a69057329e9fc7f205253b822053fa9505e2
Summary:
GCC provides some useful warnings and errors when compiling EdenFS,
and I noticed our open source build was failing. Fix some of the
issues.
Reviewed By: xavierd
Differential Revision: D37157074
fbshipit-source-id: de5705ec2eac18cd269143739289031ca5d478fc
Summary:
On Windows, system error may be using the Win32ErrorCategory, and not the
generic system_category or generic_category. Thus we also need to test for that
category as well as the Windows error codes.
Reading a bit on the internet how Win32 error codes are supposed to be handled,
in theory, using the system_category should be the preferred method. However,
this would also mean that printing exceptions would also contain a \r at the
end, which feels suboptimal. Maybe xlog should just be fixed to never try to
print \r characters to avoid this issue?
Reviewed By: genevievehelsel
Differential Revision: D37189378
fbshipit-source-id: 03ca35c38a05bc35c8a184a6069b1ab41b6852c0
Summary:
When the PathMap is case insensitive, its ordering must be the same as if only
lower cased elements were inserted into it. Unfortunately, due to the use of a
case sensitive std::lower_bound, this property is violated. To fix this, and
simplify the code a bit, the comparator passed to std::lower_bound must be case
insensitive so searching and inserting are looking in the same location.
To achieve this, a new generic and centralized way of comparing path is
introduced. Since A couple of places are interested in comparing paths, and
doing so with cross platform and case insensitively in mind can be tricky.
Let's add this code to the PathFuncs code directly so it's available everywhere
for every paths types.
Reviewed By: chadaustin
Differential Revision: D36119091
fbshipit-source-id: 3f7f6349c00d14bb9cb424d33328024f2962d7c8
Summary:
One of the unchecked and not documentation assumption of Tree is that they must
be sorted the same way as a TreeInode. Unfortunately, there is no guarantee
that the TreeEntry in a Tree are sorted, and the only place where this
guarantee is upheld... is in the testharness test library.
To solve this, the best way is to use the same underlying datastructure in both
TreeInode and Tree: a PathMap. This is guarantee to be sorted, and it properly
takes into account case sensistivity.
Reviewed By: chadaustin
Differential Revision: D36257252
fbshipit-source-id: 0fadd3c84d8e95631a8293c6797544ea94046a10
Summary:
Add some `FaultInjector` methods to allow passing in multiple arguments to
construct the fault key value.
These APIs make it easy for the caller to dynamically construct the key, but
skip doing key construction entirely when fault injection is disabled.
Reviewed By: xavierd
Differential Revision: D36435174
fbshipit-source-id: 4d37a18610ba95e90b86b49c9342de5e782362ed
Summary:
Add a couple more comments documenting the usage of the FaultInjector class.
Also improve the comments for check() and checkAsync(). The previous comments
for check() incorrectly stated that it did not support faults that block or
delay execution.
Reviewed By: genevievehelsel
Differential Revision: D36435175
fbshipit-source-id: eb95f00562cc3c24feb130807e5e5621b16b6595
Summary:
Update FaultInjector to use F14NodeMap directly rather than the
older StringKeyedUnorderedMap class from folly/experimental. These days
StringKeyedUnorderedMap is just a wrapper around F14NodeMap, and it is better
to just use F14NodeMap directly.
Reviewed By: genevievehelsel
Differential Revision: D36435176
fbshipit-source-id: 3d9425bf885282f497729b28a5253c27454b86bd
Summary: Update the FaultInjector code to use the C++17 std::variant class.
Reviewed By: genevievehelsel
Differential Revision: D36435177
fbshipit-source-id: 2cc9087f6bd202339b7d6409ca895c9aa18e191c
Summary: Allow setting creation flags on Windows. This is used in the next diff in the stack.
Reviewed By: chadaustin
Differential Revision: D35948068
fbshipit-source-id: 8fce0c86c81ea53c191dffa02d772d8adbfa23e0
Summary:
On Windows, dbghelp.dll does not look at the executable path for debug symbol files by default. This has caused us to not be able to see stack traces when iterating / debugging EdenFS on Windows.
This diff fixes it by manually adding the binary directory to the search list, and explicitly load it when reporting a crash.
This will also trigger loading from the debug symbol server **if it is configured in the environment variable**.
Reviewed By: xavierd
Differential Revision: D35623865
fbshipit-source-id: 72822bf38af93632fbd3a20757863d6bea4b078a
Summary: This diff makes SpawnedProcess to be Unicode-compatible.
Reviewed By: xavierd
Differential Revision: D35267985
fbshipit-source-id: 627035bf5170b2f300ede8553dff08b4ab7385a3
Summary:
TL;DR: File invalidation after checkout is broken on NFS macOS. This proposes a
fix.
Previously, to invalidate things on NFS we opened and closed all the parent
directories of any files/directories that changed during a checkout operation.
This worked on Linux because all open calls result in some request into the
EdenFS daemon (usually a getattr I think). The returned response from this
would show that the directory had update timestamps. So the open would see the
parent directories in their updated state. This would trigger the NFS client to
clear it's caches for that directory and their recursive children to preserve
CTO. CTO or close-to-open consistency guarantees that if on process observed a
file in state A and closed the file, then another process opened that same file
, it will see the file in state A or a later state.
macOS does not seem to do CTO.
It seems that most of the "read" filesystem calls can basically be served from
cache on NFS. Only writes seem to be guaranteed to make it into eden.
So instead of using a "read" filesystem call to trigger invalidation, we need to
use a write one.
I tried opening things in write mode, but the problem is that we need to
invalidate directories (to ensure the entry information is updated) and
directories can not be opened in write mode.
I tried writing 0 bytes to the files themselves that have changed, but this
empty write is short circuited somewhere in the kernel.
The only filesystem call that can be a noop, and seems to trigger a call into
eden is chmod. We are not really working off any guarantees any more, but
it seems to work on both Linux and macOS in my testing so far and is better
than our current broken state of invalidation.
So now to invalidate we chmod parent directories that have changed with their
current mode. Note that this could get extra tricky if we were mixing updating
directory permissions with invalidating them. We would need to ensure we chmod
with the most up to date mode bits. However, because we do not update
permissions anywhere that we also invalidate (checkout does not know how to
update directory permissions) things are a little simpler.
It's important that the chmod can not complete until it seems an updated view of
the parent directory. So we have to be careful about locking here. (Although
that hasn't really changed since we switched from open to chmod.)
One thing that does change is that since chmod is technically a write, it could
cause extra materialization that would be bad for checkout and status
performance. To prevent these problems I have updated setattr to not materialize
a directory when the attribute change is a noop which it should be in our
invalidation technique unless another client modified the directory in the
meantime in which case the directory should be modified anyways. This would
mean that we could end up wiping out clients changes to permissions in the
working copy during checkout. but this matches non eden checkout behavior. So I
think this is ok.
Reviewed By: xavierd
Differential Revision: D35435764
fbshipit-source-id: 196c4995b130b595f9582204237e726e5212993f
Summary: Eden's `getEntryInformation` API currently loads Inodes for all paths queried, force-materializing the path component's Inodes. This often isn't required, and eden could be fetching non-loaded data from the object store instead.
Reviewed By: chadaustin
Differential Revision: D35372760
fbshipit-source-id: e31a450a20b09249f03339dcd1aeca2eb363046e
Summary:
Stack context:
To get a build setup for edencommon I picked an eden library to move over
to edencommon.
I picked process name cache. I would like to have this library in watchman
so that I can log spawning process command lines as well as client command
lines.
This diff:
finally actually moves over process name cache
Reviewed By: chadaustin
Differential Revision: D34218020
fbshipit-source-id: 1e51ec6524d1e67a1dfd4d03a8189f882f5d3ff1
Summary:
Stack context:
To get a build setup for edencommon I picked an eden library to move over
to edencommon.
I picked process name cache. I would like to have this library in watchman
so that I can log spawning process command lines as well as client command
lines.
This diff:
process name cache has some benchmark tests so I am moving benchharness over.
Reviewed By: chadaustin
Differential Revision: D34218024
fbshipit-source-id: 260730adb7e5d4a3131e72fa2f03cc4c5066c075
Summary:
Stack context:
To get a build setup for edencommon I picked an eden library to move over
to edencommon.
I picked process name cache. I would like to have this library in watchman
so that I can log spawning process command lines as well as client command
lines.
This diff:
process name cache depends on StringConv.h so I am moving that over.
Reviewed By: chadaustin
Differential Revision: D34218023
fbshipit-source-id: d928768a6b5f737a1c546c6f3c73acc5535fc44c
Summary:
Stack context:
To get a build setup for edencommon I picked an eden library to move over
to edencommon.
I picked process name cache. I would like to have this library in watchman
so that I can log spawning process command lines as well as client command
lines.
This Diff:
process name cache depends on WinError.h so I am moving that over.
Reviewed By: chadaustin
Differential Revision: D34218026
fbshipit-source-id: 064f5bfc50c4ba120c887b01ff1f0a3e0543d498
Summary:
To get a build setup for edencommon I picked an eden library to move over
to edencommon.
I picked process name cache. I would like to have this library in watchman
so that I can log spawning process command lines as well as client command
lines.
Diff:
process name cache depends on Handle.h so I am moving that over.
Reviewed By: chadaustin
Differential Revision: D34218022
fbshipit-source-id: 05c42eb0fa8be78fd001249c0c09ed7fa515ea44
Summary:
Stack context:
To get a build setup for edencommon I picked an eden library to move over
to edencommon.
I picked process name cache. I would like to have this library in watchman
so that I can log spawning process command lines as well as client command
lines.
Diff:
process name cache depends on Synchronized so I am moving that over in this diff.
Reviewed By: chadaustin
Differential Revision: D34218021
fbshipit-source-id: 528e7b4b12476b33a294a342e6e4ee13c92c7361
Summary:
This opens the directory and reads all the names from it, excluding "." and
"..". On Windows, this function is written in such a way that it allows
concurrent deletion/renaming of the directory while this is running.
Reviewed By: chadaustin
Differential Revision: D35067144
fbshipit-source-id: 20fb8f9ac8a49970cb360eb3307e86d846760e98
Summary:
The PathFuncs.h header is included pretty much everywhere in EdenFS, and thus
its compile speed affects the entirety of EdenFS. Unfortunately, the
boost::filesystem dependency is pretty large and is thus slowing down the
compilation.
Since the few methods that rely on Boost are only used in one place, let's
instead inline the use of Boost in these places.
Reviewed By: chadaustin, fanzeyi
Differential Revision: D35058316
fbshipit-source-id: b80a39f1575acc0fa4b9e64eb5375b95b4b421da
Summary:
Our use of RCU is fundamentally incompatible with N4895 since it allows a read
section to be entered in one thread and closed in another one. We use this
feature in Prjfs to prevent a mount to be unmounted while requests are ongoing
on it.
Let's instead make use of a folly::AtomicReadMostlyMainPtr which is equivalent
to a folly::Synchronized<std::shared_ptr<>> without requiring as many atomics
in the read path.
Reviewed By: Decave
Differential Revision: D34982722
fbshipit-source-id: 299f25fc8c8aa72fb89cc9fc880860780ea0517d
Summary:
Section 5.3.5 in the new C++ Concurrency Technical Specification 2
(http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/n4895.pdf) specifies
the following function:
rcu_domain& rcu_default_domain() noexcept;
The folly RCU header file currently exposes a very similar rcu_default_domain()
function that returns a pointer to an rcu_domain rather than a reference. This
change updates the folly RCU implementation to instead return a reference, and
updates the few callers in fbcode that were invoking it to instead expect a
reference.
Note that in doing so, we also update rcu_barrier(), rcu_retire(), and
rcu_obj_base::retire() to adhere to the new standard.
Reviewed By: yfeldblum
Differential Revision: D34622786
fbshipit-source-id: 4460ce563f9e9ef4d65cdd517707297c23091c80
Summary:
With Facebook having been renamed Meta Platforms, we need to change the license
headers.
Reviewed By: fanzeyi
Differential Revision: D33407812
fbshipit-source-id: b11bfbbf13a48873f0cea75f212cc7b07a68fb2e
Summary:
Like it says in the title. In Buck2 this doesn't quite work as `__FILE__` is
relative to fbsource but now tests are relative to fbcode.
This might be something we have to fix in some way, but until then let's just
fix the test.
Reviewed By: xavierd
Differential Revision: D33092380
fbshipit-source-id: f54fabe795f909111cc7f2269359a4117232e630
Summary:
static_assert sizeof doesn't show the actual size in clang or gcc, so
implement a CheckSize template.
Reviewed By: xavierd
Differential Revision: D32998926
fbshipit-source-id: 432fbf19eae5b4572e5a4908f9007b7242a3ae69
Summary: This will be used in a subsequent diff.
Reviewed By: chadaustin
Differential Revision: D32837579
fbshipit-source-id: 9a32be6a9c912faaf43064c98f61c9fbec17c7ba
Summary:
Now that we have NFS mounts we need to support takeover for those mounts.
This commit teaches the takeover server and client to serialize and deserialize
NFS information.
NFS information includes:
- the mountd socket. Mountd is used to register mount points, and should keep
the same socket across graceful restarts.
- a connected socket for each of the NFS mount points.
- All the rest of the mount point information that we send for fuse mounts (with
the exception of the fuse negotiated parameters.
Future commits will teach the NFS components to graceful stop and yield this info
as well as restart with this info.
Reviewed By: xavierd
Differential Revision: D31802787
fbshipit-source-id: 667625589b77eed79f7e17390013a818d4be7068
Summary: adds FileInode::getBlobMetadata() method. Instead of fetching sha1 and filesize separately, we can use this method to get them both at the same time
Reviewed By: xavierd
Differential Revision: D32399852
fbshipit-source-id: 6b4e271136492a7fa836a49c3794347a5ebab9ef
Summary:
We should all be migrating to platform010, for the improved performance of its
generated code and for its improved diagnostics/portability.
reinterpret_cast another u8"..." literal to const char*
This avoids the following errors:
```
stderr: In file included from eden/fs/utils/test/Utf8Test.cpp:9:
In file included from folly/portability/GTest.h:32:
third-party-buck/platform010/build/googletest/include/gtest/gtest.h:1545:11: error: invalid operands to binary expression ('char8_t const[4]' and 'const std::basic_string<char>')
if (lhs == rhs) {
~~~ ^ ~~~
third-party-buck/platform010/build/googletest/include/gtest/gtest.h:1572:12: note: in instantiation of function template specialization 'testing::internal::CmpHelperEQ<char8_t [4], std::basic_string<char>>' requested here
return CmpHelperEQ(lhs_expression, rhs_expression, lhs, rhs);
^
eden/fs/utils/test/Utf8Test.cpp:43:3: note: in instantiation of function template specialization 'testing::internal::EqHelper::Compare<char8_t [4], std::basic_string<char>, nullptr>' requested here
EXPECT_EQ(u8"\uFFFD", ensureValidUtf8("\xff"));
```
Differential Revision: D32484896
fbshipit-source-id: b271292eb121e0e183bec56ab51fc6acb791b052
Summary: Have constructor do init() instead. I can't think of any valid reason to have this API other than confusing everyone who decides to use it.
Reviewed By: praihan
Differential Revision: D32302228
fbshipit-source-id: 136e0d2919291acd60be953337309cd006bd674b
Summary:
When unwrapping Try it is convenient to be able to build an ImmediateFuture
directly from the .exception() call.
Reviewed By: chadaustin
Differential Revision: D32333774
fbshipit-source-id: 0066ca7eddc4a57010f60974c234b57691b73d77
Summary:
For some reason, in some cases the compiler wants to use the folly::collectAll
function instead of the ImmediateFuture one. Adding the namespace solves this
issue.
Reviewed By: chadaustin
Differential Revision: D32329807
fbshipit-source-id: 22ee9b20cf33e5718c9bb0270dd93f3f857b0967
Summary:
We already had a collectAllSafe for argument packs, let's have a similar
function when the argument is a vector.
Reviewed By: chadaustin
Differential Revision: D32323573
fbshipit-source-id: 2cfa3da565a7d9bb88746e3cc6b10d1d8890c05e
Summary:
We should all be migrating to platform010, for the improved performance of its
generated code and for its improved diagnostics/portability.
add an explicit constructor, now required for C++20, see https://en.cppreference.com/w/cpp/language/default_constructor
This avoids the following errors:
eden/fs/utils/test/ImmediateFutureTest.cpp:335:17: error: no matching constructor for initialization of 'Uncopyable'
# Context:
The cost of delaying a platform upgrade is now estimated at about $3M/week.
For the preceding one, it was measured at [**$1M/week**](https://fburl.com/jfmpsqmy).
Differential Revision: D32250064
fbshipit-source-id: 8c6990489cafb09629c03e56bdbae92395d6f85f
Summary:
C++20 changed the rules around char8_t. Cast to char* as
necessary. Also, fix references to some removed std::allocator
associated types.
Reviewed By: fanzeyi, meyering
Differential Revision: D32155762
fbshipit-source-id: dc72e5bc58e8c08612ea1a1eb2568861701287c6
Summary:
Takeover data needs to include fuse data for fuse
mounts and NFS data for NFS mounts.
Fuse related data is already wrapped up in
FuseChannelData. we are making an NfsChannelData to
mirror this for NFS. These structs will be used in
TakeoverData in a later commit.
Reviewed By: xavierd
Differential Revision: D31740475
fbshipit-source-id: f64367a5881175b9e20d261e43ec7570c498f380
Summary:
Some compiler (msvc, old gcc) can't seem to figure out that these switch cases
are exhaustive and thus warn about them. Let's help it to silence the warnings.
Reviewed By: chadaustin
Differential Revision: D32022198
fbshipit-source-id: 67f89d52211db45a09e8a2fc0ed64da667b8cbbc
Summary:
To avoid any risk of SSLKEYLOGFILE being used to unexpectedly write
files as root, ensure the privhelper does not even link against
OpenSSL.
Reviewed By: fanzeyi
Differential Revision: D31944839
fbshipit-source-id: 3280657a10f4a468ddb63a874d38198bd333e01c
Summary:
Ignoring the return value of this method is equivalent to dropping the future
entirely, which isn't what is usually desired. Let's make sure the compiler
helps us in finding these cases as code is converted.
Reviewed By: chadaustin
Differential Revision: D31945362
fbshipit-source-id: dd646365e00d3ebb04ed9b6803c561fc11ff4a14
Summary:
This mimics the one eden/fs/utils/Future.h behavior where all the Try are
unwrapped to simply return a tuple.
Reviewed By: kmancini
Differential Revision: D31668773
fbshipit-source-id: f4b9b6b2564cc1e2c43aa5b03bfb06048f0a39c1
Summary:
In some places, EdenFS uses folly::collectAll passing futures to it. In order
to replace it, we need an equivalent for ImmediateFuture.
The implementation relies on the fact that ImmediateFuture's constructor will
extract the value from a SemiFuture if it is ready, and thus the code doesn't
have to special case when all the passed in ImmediateFuture are immediate
values.
Reviewed By: chadaustin
Differential Revision: D31667897
fbshipit-source-id: ea680e1c74528d1eb34210cacb58efda8a26fee0
Summary:
This will cut some of the boiler plate of returning an ImmediateFuture in the
error case.
Reviewed By: chadaustin
Differential Revision: D31769752
fbshipit-source-id: 42be026aee6a6071bbb2c1f8af918b107d593eab
Summary:
Exception cases should be rare, but EdenFS has a couple of spots where
exceptions are somewhat expected. One of which being a lookup failure due to a
non-existing file. Due to the overhead of these lookup failures, EdenFS is
actually spending as much time resolving these, than resolving successful
lookups.
The reason for the high overhead is due to the ImmediateFuture::thenValue just
relying on an exception being thrown when accessing the stored Try value. We
can do better than this by testing the Try before hand, and bailing out in that
case.
Reviewed By: fanzeyi
Differential Revision: D31655223
fbshipit-source-id: 29da7ef38a19f09c91c931e7e8e080e5ed6af269
Summary:
When dealing with a large amount of futures, one of the common pattern is to
store all the pending one into a vector and collect them all at once. Let's add
a similar function for ImmediateFuture to allow converting code that make use
of this pattern.
Reviewed By: chadaustin
Differential Revision: D31452296
fbshipit-source-id: 04aee8f358727e02b4202d09d78fafc0917b8dec
Summary:
In a meeting today, the question arose "when should one use Future,
SemiFuture, or ImmediateFuture?" To help clarify the situation,
reframe some of ImmediateFuture's doc comments in terms of their
semantics.
Simultaneously, replace hasImmediate with isReady, which returns true
when a value can be retrieved without blocking.
Also, allow converting a completed SemiFuture into an ImmediateFuture
which is also isReady.
Reviewed By: xavierd
Differential Revision: D31563622
fbshipit-source-id: b942031c15d46948b1f60ade9f8043fa19442b65
Summary:
This is fairly mechanical diff that finalizes split of Hash into ObjectId and Hash20.
More specifically this diff does two things:
* Replaces `Hash` with `Hash20`
* Removes alias `using Hash = Hash20`
Reviewed By: chadaustin
Differential Revision: D31324202
fbshipit-source-id: 780b6d2a422ddf6d0f3cfc91e3e70ad10ebaa8b4
Summary:
While std::variant is convenient, they are both slow to compile, and the
compiler cannot optimize it as well as a manually written tagged union. Since
ImmediateFuture is performance critical for EdenFS, let's use a tagged union
and speed them up by an additional 40%.
Reviewed By: chadaustin
Differential Revision: D31272296
fbshipit-source-id: e34be4489a596d3577b3bd900a1f20d6c7d8b693
Summary:
The max duration would cause UBSAN failures due to folly's SemiFuture code
multiplying the value which understandably cannot be represented. Splitting the
function is easy and avoids the problem entirely.
Reviewed By: genevievehelsel
Differential Revision: D31272297
fbshipit-source-id: c15ca70ad771c11b4f68bb9974422c0986d4928b
Summary:
folly:format is deprecated in lieu of fmt and std::format. Migrate
most of EdenFS to fmt instead.
Differential Revision: D31025948
fbshipit-source-id: 82ed674d5e255ac129995b56bc8b9731a5fbf82e
Summary: This is a similar diff to D30915090, but for EdenFS.
Differential Revision: D30915126
fbshipit-source-id: 9a718e47237924ebe20176c522a1b1193224236c
Summary:
During a diff operation, files that have a case change should not reported as
changed for case insensitive mount. This is a follow up to D29150552 (37ccaa9231) where the
TreeInode code was taught to ignore case changes for case insensitive mounts.
Reviewed By: kmancini
Differential Revision: D29952464
fbshipit-source-id: e5fa8eccc1c1984a680838fd8589a0278989d9d0
Summary:
TSAN complains that pipes_ is read and written in different threads without
proper synchronization. To avoid this, we can simply close the FileDescriptor
without removing it from the pipes map as this achieve the same result: it
notifies the reader that the endpoint is closed.
Differential Revision: D29924043
fbshipit-source-id: be92630799bb5c78089dbe85f9c2f02f937300bc
Summary:
This adds debug commands for ActivityRecorder:
```
eden debug start_recording --output-dir <DIR>
* stdout: the id of the profile
eden debug stop_recording --unique <ID>
* stdout: the output file path
```
Users can record multiple profiles concurrently. Each profile is identified by the timestamp when it started.
Reviewed By: genevievehelsel
Differential Revision: D29666359
fbshipit-source-id: 487ca67de77378a8141bc4ac46b9abd1375ffd23
Summary: This adds counters for memory and disk counts in addition to import count so that we can understand cache hit rates during local investigation or output this in ActivityRecorder.
Reviewed By: genevievehelsel
Differential Revision: D29805637
fbshipit-source-id: 34261f91c33d6bd4bcb4b85b17d2e68360410896
Summary: NfsTaskQueue can be made more generic to be shared across the codebase, so this makes it its own target in `eden/fs/utils` w/ the name EdenTaskQueue.
Reviewed By: xavierd
Differential Revision: D29244762
fbshipit-source-id: 78348f2ff8fa66bc801aefe7d6b3905e0da278e8
Summary:
Using a PathMap to hold the list of mounts gives us case sensitivity for free
without needing to have #ifdef _WIN32, we can also avoid copying the paths just
to normalize them.
Reviewed By: fanzeyi
Differential Revision: D28969795
fbshipit-source-id: 859cdc3881e36db120913099ce3f451940fa48a4
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: fanzeyi
Differential Revision: D28663227
fbshipit-source-id: cee18ec28283c0aef18151465a3c13e9be604d7d
Summary: FileDescriptor.cp throws some warning on comparison of signed long and unsigned long. So fix these warning and follow how ```folly/portability/SysUio.cpp:doVecOperation``` did
Reviewed By: fanzeyi
Differential Revision: D28608581
fbshipit-source-id: 3b6e35e9764548ce470634f16a1f6eec5118d7ed
Summary: This will be used to replace calls to folly::makeFutureWith.
Reviewed By: chadaustin
Differential Revision: D28515786
fbshipit-source-id: 2c2c542392e8e57b8f865173d6878cb9d00ba376
Summary:
When the caller asks to not perform sanity checking, this should be honored as
the caller can for instance tell that a path is guaranteed to be valid and
shouldn't be checked.
When running the newly added glob benchmark, EdenFS was spending 7.5% of the
time validating a path that was constructed via the operator+.
Reviewed By: fanzeyi
Differential Revision: D28612665
fbshipit-source-id: 282155d0415ab6458bd9307c89a24e9e090bf09d
Summary:
Lambda returning no values would cause the compiler to try to instantiate an
ImmediateFuture<void>, which doesn't compile. We could try special casing the
void type, but it's easier if we simply consider these lambda to return unit.
Reviewed By: genevievehelsel
Differential Revision: D28372253
fbshipit-source-id: 1368ae5dc5e2d4d6a5c3e31bc87ed7d230027c3a
Summary:
This is to mimic the folly::Future API which allows the `get` and `getTry` to
throw if the Future isn't ready. One difference is that the ImmediateFuture API
has a default argument of a max duration instead of having a separate method.
Since chrono durations are expressed as intmax_t (64-bits on systems supported
by EdenFS) a max duration is virtually infinite.
Reviewed By: chadaustin
Differential Revision: D28424053
fbshipit-source-id: 319493174f31367184dbe0aa811a97145b0310cf
Summary:
This allows the macro to be used for both folly::Future, but also with
ImmediateFuture, reducing the cost of migrating to the latter.
Reviewed By: chadaustin
Differential Revision: D28302478
fbshipit-source-id: d8470428ecaa6aaccf2aa884437ca97d4b45806f
Summary:
Since lambda are default capturing by const, Func was also captured that way,
which made it impossible to call if Func was actually a mutable lambda.
Reviewed By: chadaustin
Differential Revision: D28297423
fbshipit-source-id: b9c6bdcf024c2e219ec0f8f5be2379e51df24db0
Summary:
This code initially was for thenValue, but when I converted it for thenTry I
forgot to also remove the hasException case. We were never calling the callback
on a Try that had an exception.
Reviewed By: chadaustin
Differential Revision: D28302477
fbshipit-source-id: 755fb2c8928627fbe1883f3863cafc360e34a038
Summary:
When the passed in callback returns an ImmediateFuture, the code had a small
typing bug where we would try to return an ImmediateFuture<ImmediateFuture<T>>
while the function signature expected an ImmediateFuture<T>. This is due to the
SemiFuture code not coalescing a SemiFuture<ImmediatureFuture<T>> into a
SemiFuture<T>.
Reviewed By: chadaustin
Differential Revision: D28293942
fbshipit-source-id: 1bc8f58d387766f29e573499fb37402ff9b49c83
Summary:
gtest includes some windows headers that will have conflicts with the
folly portability versions. This caused some issues in my in-memory tree
cache diffs (D27050310 (8a1a529fcc)).
We should probably generally be using the folly portable gtests so we can
avoid such issues in the future.
see here for more details: bd600cd4e8/folly/portability/GTest.h (L19)
I ran this with codemod yes to all
- convert all the includes with quotes:
`codemod -d eden/fs --extensions cpp,h '\#include\ "gtest/gtest\.h"' '#include <folly/portability/GTest.h>'`
- convert all the includes with brackets
`codemod -d eden/fs --extensions cpp,h '\#include\ <gtest/gtest\.h>' '#include <folly/portability/GTest.h>'`
- convert the test template
`codemod -d eden/facebook --extensions template '\#include\ <gtest/gtest\.h>' '#include <folly/portability/GTest.h>'`
then used `arc lint` to clean up all the targets files
Reviewed By: genevievehelsel, xavierd
Differential Revision: D28035146
fbshipit-source-id: c3b88df5d4e7cdf4d1e51d9689987ce039f47fde
Summary:
Due to NFS being designed as a network filesystem, it default to binding on a
TCP socket. For EdenFS, since we're not expecting to mount an actual remote
filesystem, we bind these sockets to localhost. Unfortunately, TCP sockets have
some inherent overhead due to being designed to be reliable over a non-reliable
medium.
On macOS, Apple provides a way to mount an NFS server that is listening on a
unix domain socket. Thanks for unix socket being reliable, the TCP overhead
goes away leading to some higher throughput and lower latency for the NFS
server. For EdenFS, timing `rg foobar` over a directory containing 27k files gives:
NFS over TCP:
rg foobar > /dev/null 0.80s user 5.44s system 567% cpu 1.100 total
NFS over UDS:
rg foobar > /dev/null 0.77s user 5.27s system 679% cpu 0.888 total
osxfuse:
rg foobar > /dev/null 0.87s user 5.59s system 662% cpu 0.975 total
The main drawback of going through a unix socket is that D27717945 (bcf6aa465c) appears to
no longer be effective due to
8f02f2a044/bsd/nfs/nfs_vfsops.c (L3739)
Reviewed By: kmancini
Differential Revision: D28261422
fbshipit-source-id: 25dc1dc78cdb50d6c6550a86ef01ea2c894c110f
Summary:
While ImmediateFuture are expected to be used on values that are mostly
immediate, there are cases where it won't be. In these cases we need a way to
wait for the computation/IO to complete. In order to achieve this, we need to
transform an ImmediateFuture onto a SemiFuture.
Reviewed By: fanzeyi
Differential Revision: D28293941
fbshipit-source-id: 227c0acf1e22e4f23a948ca03f2c92ccc160c862
Summary:
When a T can be default constructed, make an ImmediateFuture default
constructible.
Reviewed By: fanzeyi
Differential Revision: D28292874
fbshipit-source-id: 4c239cc9c3f448652b2bcdc103ea1a81ace46402
Summary: This should help the compiler generate even better code.
Reviewed By: chadaustin
Differential Revision: D28153979
fbshipit-source-id: b1d84c92af4fa760c92624c53d7f57330d7706fa
Summary:
In very hot code path, EdenFS is spending a very large amount of time creating
and destroying folly::Future objects. This is due to the required memory
allocation, as well as the handful of atomics that are happening at creation
time, and these are showing up in EdenFS profiles.
In the steady state, EdenFS actually doesn't need futures, as it often times is
able to service requests from its in-memory caches, in which case we should
ideally just return the value itself and not wrap it in a folly::Future. The
added ImmediateFuture is a step in this direction, as it can hold either an
immediate value, or a folly::SemiFuture, and allow the same API to be used
transparently between these 2.
Reviewed By: genevievehelsel
Differential Revision: D28006802
fbshipit-source-id: 89eaa32e7fa82c44844c4b23c4cb30dbeea46ca8
Summary:
This diff defines symlink type in `DirType`.
Even though it is not directly used in the FSCK diff. This will allow us to support symlink in EdenFS Windows in the future.
Reviewed By: genevievehelsel
Differential Revision: D28016305
fbshipit-source-id: 67c1aa22e39198f9c91845129695f27b8303a5f1
Summary:
These methods will be used in my later Windows fsck diff as it will need to scan disk state to find changes.
It is a bit unfortunate that we'll need to stick with boost for now. However this should be a fairly easy migration to `std::filesystem` once that is available.
Reviewed By: kmancini
Differential Revision: D27872828
fbshipit-source-id: f6b27a171026aeaaea3db9f17b8f43cfa25004e4
Summary:
Copy from Watchman.
This allows us to show stack trace when EdenFS terminates on Windows.
Reviewed By: chadaustin
Differential Revision: D27896966
fbshipit-source-id: f3238a37a1176f879d5e6bc051ec97031c9a7096
Summary:
We started getting the message
```stderr: eden/fs/utils/SpawnedProcess.cpp:798:21: error: 'getIOExecutor' is deprecated: getIOExecutor is deprecated. To use the global mutable executor use getUnsafeMutableGlobalIOExecutor. For a better solution use getGlobalIOExecutor. [-Werror,-Wdeprecated-declarations]
```
I don't see why we would need a mutable executor here so I chose `getGlobalIOExecutor` over `getUnsafeMutableGlobalIOExecutor`.
Reviewed By: kmancini
Differential Revision: D27912276
fbshipit-source-id: 95b1053f72c2b4eb2746e3c40c0cf76b69d90d6e
Summary: `CaseSensitivity::Sensitive` is better than a mere `true` out of nowhere.
Reviewed By: kmancini
Differential Revision: D27867180
fbshipit-source-id: 39d21d3cc3b70c78c6984c3ddbd65b53520770be
Summary: Migrate the codebase away from the ad-hoc `folly::uint64ToBufferUnsafe` and to `folly::to_ascii_decimal` which is intended for these cases.
Reviewed By: WillerZ
Differential Revision: D27281875
fbshipit-source-id: 0c98749e4aed9c873853eed2221cf54a89279ff4
Summary:
I want to use EdenError from some code outside of service/, so move
EdenError into utils.
Reviewed By: genevievehelsel
Differential Revision: D25447438
fbshipit-source-id: 2d1ddfa379238369679e84708518a9ba106f76b9
Summary:
The world has moved on utf-8 as the default encoding for files and data, but
EdenFS still accepts non utf-8 filenames to be written to it. In fact, most of
the time when a non utf-8 file is written to the working copy, and even though
EdenFS handles it properly, Mercurial ends up freaking out and crash. In all of
these cases, non-utf8 files were not intentional, and thus refusing to create
them wouldn't be a loss of functionality.
Note that this diff makes the asumption that Mercurial's manifest only accept
utf8 path, and thus we only have to protect against files being created in the
working copy that aren't utf8.
The unfortunate part of this diff is that it makes importing trees a bit more
expensive as testing that a path is utf8 valid is not free.
Reviewed By: chadaustin
Differential Revision: D25442975
fbshipit-source-id: 89341a004272736a61639751da43c2e9c673d5b3
Summary: Update name to match usage of Try as tri-state, since this method also throws if the Try is empty
Reviewed By: yfeldblum
Differential Revision: D25737810
fbshipit-source-id: a4166153362f07353d212216fbaf7105867eef2a
Summary:
Mercurial support files with `\` in their name, which can't be represented on
Windows due to `\` being the path separator. Currently, EdenFS will throw
errors at the user when such file are encountered, let's simply warn, and
continue.
Reviewed By: chadaustin
Differential Revision: D25430523
fbshipit-source-id: 4167b4cd81380226aead8e4f4850a7738087fd95
