Summary: FileDescriptor.cp throws some warning on comparison of signed long and unsigned long. So fix these warning and follow how ```folly/portability/SysUio.cpp:doVecOperation``` did
Reviewed By: fanzeyi
Differential Revision: D28608581
fbshipit-source-id: 3b6e35e9764548ce470634f16a1f6eec5118d7ed
Summary:
Now that makeImmediateFutureWith exists, we can simply use it instead of
constructing a ImmediateFuture<folly::Unit> and calling thenValue on it.
Reviewed By: chadaustin
Differential Revision: D28518059
fbshipit-source-id: 0041cf863fb32efab274f11c77c76109ca9b454f
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: xavierd
Differential Revision: D28665465
fbshipit-source-id: 9281de7fd62f38e09d91435bb53c819bb98fb4ec
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: xavierd
Differential Revision: D28595085
fbshipit-source-id: abc03d210b2e9c5aa19a8925be6d4c426311e826
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: xavierd
Differential Revision: D28594404
fbshipit-source-id: f3dec92403739d67df3ecd091f2d8283a11ea0db
Summary: This will be used to replace calls to folly::makeFutureWith.
Reviewed By: chadaustin
Differential Revision: D28515786
fbshipit-source-id: 2c2c542392e8e57b8f865173d6878cb9d00ba376
Summary: This diff removes the gflag based batch size option and promote it into a EdenFS Configuration so we can experiment with different batch size easily.
Reviewed By: chadaustin
Differential Revision: D28555280
fbshipit-source-id: 6d3a7be3cd880f0aaa3f427c0328222efa2d37ea
Summary:
Copying an InodePtr/shared_ptr adds 2 atomics: one for the copy, and one when
the copy is dropped. In the case where a move can be achieved, we can avoid
these atomics and make the code more efficient.
Reviewed By: fanzeyi
Differential Revision: D28614769
fbshipit-source-id: 9c4be6ce335cc17eb889a5681aa9c13595a7909e
Summary:
Looking at the glob code, I noticed that some paths were copied into lambda and
vectors, while they could have been moved.
Reviewed By: fanzeyi
Differential Revision: D28614211
fbshipit-source-id: 5662586bcc3d9a9c5ca899c59d29b40a4e590adc
Summary:
When the caller asks to not perform sanity checking, this should be honored as
the caller can for instance tell that a path is guaranteed to be valid and
shouldn't be checked.
When running the newly added glob benchmark, EdenFS was spending 7.5% of the
time validating a path that was constructed via the operator+.
Reviewed By: fanzeyi
Differential Revision: D28612665
fbshipit-source-id: 282155d0415ab6458bd9307c89a24e9e090bf09d
Summary:
Querying RocksDB for a blob that we know is not present is expensive, and will
always construct a StoreResult::missing which does allocate memory for that to
simply be dropped. We can revamp the StoreResult a bit, but for now let's
simply not query RocksDB.
In theory, if RocksDB is populated with blobs this can be a loss as more
request will need to go to the hgcache, in practice, RocksDB local caching has
been disabled for so long that we'll always get cache misses.
Reviewed By: fanzeyi
Differential Revision: D28592666
fbshipit-source-id: 49f48097e81eddb4f9c3eba7af774baf018b0821
Summary:
C++ is full of footguns, in this case, the lockContents method either returns a
const std::vector<TreeEntry>&, or an rlock RAII for a folly::Synchronized
object. When assigning the return value of that method to an `auto` variable, a
copy is made in both cases, which isn't exactly what is desired in the first
case. Instead using the reference as-is is what we want.
To achieve the right behavior, a `const auto&` needs to be used: it will either be
a const reference in the first case, or a copy in the second case.
During an `arc focus2` while the Buck parsing phase is ongoing, the copy shows
up at the top of the profiler as one of the most expensive operation. This
significantly reduce its cost.
Reviewed By: chadaustin
Differential Revision: D28591502
fbshipit-source-id: eac1e062ed94442c90ac549ec137d91b4cb42b9c
Summary:
While Buck compiling, when a glob query is running, EdenFS uses tons of CPU,
let's add a small benchmark to profile the code in isolation and keep track of
the progress.
Reviewed By: fanzeyi
Differential Revision: D28611157
fbshipit-source-id: 81df21c8a8aba44fdefec044f9cae387b4cdd15c
Summary:
Add support to our C++ and Python SNAPSHOT parsers for version 2,
which is just an arbitrary binary string after the header.
Don't start writing the format yet. I'd like to roll one release with
read support before writing the new format.
Reviewed By: xavierd
Differential Revision: D28528235
fbshipit-source-id: 0b06596f06cafa26258ab392eaae4a8994f55f1f
Summary: The legacy format was last written in 2017, so we can remove it now.
Reviewed By: xavierd
Differential Revision: D28524663
fbshipit-source-id: 59ed437b0e5ef2e5ee3e3e9944d1049a13d64d9e
Summary:
EdenFS goes out of its way to track the second working copy parent,
but it never uses it. Stop writing it to the SNAPSHOT file.
Reviewed By: genevievehelsel
Differential Revision: D28453213
fbshipit-source-id: d7d36a1c67553f92234bec911051f4f1d4ef1d4a
Summary: This will force future modification to this file to be fully type checked.
Reviewed By: fanzeyi
Differential Revision: D28544422
fbshipit-source-id: 5ecc08a21e49535a3ed4ae9c33f219080365e609
Summary: This will force future changes to this file to be fully type checked.
Reviewed By: fanzeyi
Differential Revision: D28544427
fbshipit-source-id: 0ccfe675818a96ee8fc702317ace85d0b26ccc26
Summary: This will force future changes to this file to be fully type checked.
Reviewed By: fanzeyi
Differential Revision: D28544421
fbshipit-source-id: 67d15e4d6ed13c139adffa7cedf7795856d87c7e
Summary: This will enforce future changes to this file to be fully type checked.
Reviewed By: fanzeyi
Differential Revision: D28544426
fbshipit-source-id: d502de1e0e4c3f7910e0796b2c82ebbb9992be22
Summary:
This adds the few missing types. This will ensure that future modifications of
this file are all type checked.
Reviewed By: fanzeyi
Differential Revision: D28544424
fbshipit-source-id: 4053ede8a2901928e90c484158777aab31937600
Summary: That file was already had good types.
Reviewed By: fanzeyi
Differential Revision: D28544420
fbshipit-source-id: b1b02b7d116ca382001c318fd0626c8d50480013
Summary:
This will enable future modifications of this file to always be properly type
checked.
Reviewed By: fanzeyi
Differential Revision: D28544423
fbshipit-source-id: 73e2fe9ba386c7a0b6165424c383db7d647abcc0
Summary:
This fixes the handful of errors and warnings that Pyre threw when I tried
making this file strict. Enabling pyre strict will force future addition and
modifications to this file to have type checking, increasing the reliability of
the code.
Reviewed By: fanzeyi
Differential Revision: D28543212
fbshipit-source-id: 06b976a37e6f4c3a0bab2eb3ecc4c217497bbbd5
Summary:
On Windows, Buck will spawn one process per test target and all the tests will
run sequentially. What this means is that the inodes tests are running for a
long time due to running single threaded. To remediate this, let's start by
splitting the tests into multiple targets.
Reviewed By: chadaustin
Differential Revision: D28521827
fbshipit-source-id: dc4cd20cb9d1cdc52e84b373bb82f6bae44aa9d4
Summary: For whatever reason some of my debug logs got landed :(
Reviewed By: fanzeyi
Differential Revision: D28510825
fbshipit-source-id: 65bd9165020bfe3afff54109a9f550a460266737
Summary:
This diff implements FSCK for EdenFS Windows.
On Windows, users can still modify EdenFS mounts even when EdenFS is not
running, which may cause mismatch between EdenFS state and on-disk view. This
subsequently may cause issues when EdenFS is running again (such as permission
error, not seeing added entries in `hg status`, etc..).
This diff adds FSCK to EdenFS Windows. It is not exactly same as what fsck
traditionally do on *NIX systems. We are still dubbing it as FSCK since it
works at the same place as eden fsck.
At startup, FSCK will crawl the EdenFS mount to compare the overlay state with
disk state. It then synchronizes the overlay view with what the user has on
disk. Note Windows does not always permit user to modify the mount, it only
allows changes in certain situation. In particular, when the directory is in
Full state, this diff takes advantage of that so we can finish the scanning by
only scans such directories.
One limitation of Windows FSCK is that, it cannot reliably tell if the user
deleted a directory or file from dirty placeholder directories. This is because
ProjectedFS will hide untouched entries under dirty placeholder directory when
EdenFS is not running, and there is no way we can tell if the entry is gone
because of user deletion or hid by ProjectedFS.
This is not perfect but acceptable to some extent. One possible failure scenario is:
1. User creates a directory named `foo`.
2. User writes a file to that directory (`foo/bar`).
3. EdenFS then stops running.
4. User then deletes the entire `foo` directory on disk.
5. EdenFS starts again, `foo` will be recrated with an empty `bar` file. This
will still correctly show in `hg status`, and the user is able to delete
them again.
Reviewed By: xavierd
Differential Revision: D27872753
fbshipit-source-id: c553db568379062ff4504204c1a1785664f87c00
Summary:
We want eden doctor to report a problem about mismatched running and installed version only if the running version is more than 2 weeks older than installed.
Please see task for more context
Reviewed By: fanzeyi
Differential Revision: D28464491
fbshipit-source-id: 34b9b4cf533482f3006100bbf675c89ea7ee6fff
Summary:
While looking at the previous diff, I noticed that a significant amount of
memory was used by the loading promise. Since this loading promise is only used
when an inode is being loaded, its usage is the uncommon case: there should be
significantly more inodes being loaded or unloaded than loading at any given
time in EdenFS's lifetime.
Reviewed By: chadaustin
Differential Revision: D28477391
fbshipit-source-id: e80269506b980d7f217f6bdf59034ef003c4b3fc
Summary:
Similarly to the NFS change, this moves all the folly::Future handling in the
FuseChannel itself instead of inside the dispatcher. This should allow the
inode code to be converted to using ImmediateFuture instead of folly::Future.
Reviewed By: genevievehelsel
Differential Revision: D28372252
fbshipit-source-id: 7aae29d4a32500513c0062dfa42b2c7a3be038db
Summary:
Lambda returning no values would cause the compiler to try to instantiate an
ImmediateFuture<void>, which doesn't compile. We could try special casing the
void type, but it's easier if we simply consider these lambda to return unit.
Reviewed By: genevievehelsel
Differential Revision: D28372253
fbshipit-source-id: 1368ae5dc5e2d4d6a5c3e31bc87ed7d230027c3a
Summary:
The InodeMap can be extremely hot when issuing tons of requests to EdenFS.
Unfortunately, it still needs to do memory allocation due to its use of
folly::Future. By switching to ImmediateFuture we can avoid the memory
allocation, speeding it up slightly.
For the NFS dispatcher, this moves the call to `semi()` inward, allowing us to
target specific inode methods to convert next. For the Fuse dispatcher, I've
simply converted the ImmediateFuture into a Future directly, keeping the rest
of the code unchanged, a subsequent change will convert the dispatcher code to
ImmediateFuture.
Reviewed By: chadaustin
Differential Revision: D28302480
fbshipit-source-id: 4e097a721443f0d52f34a337a96f8a63a9a7cd7c
Summary:
This is to mimic the folly::Future API which allows the `get` and `getTry` to
throw if the Future isn't ready. One difference is that the ImmediateFuture API
has a default argument of a max duration instead of having a separate method.
Since chrono durations are expressed as intmax_t (64-bits on systems supported
by EdenFS) a max duration is virtually infinite.
Reviewed By: chadaustin
Differential Revision: D28424053
fbshipit-source-id: 319493174f31367184dbe0aa811a97145b0310cf
Summary:
DurhamG discovered that setting max_background in fuse_init_out allows
a larger number of concurrent FUSE requests. The documentation
indicates it's intended to affect background requests like readahead,
but empirically you can observe increased live FUSE requests with `rg
-j200` and `eden top`. Default to 1000 because EdenFS can handle a
large amount of concurrency and we want to avoid blob and tree fetches
to block FUSE IO when not necessary.
Reviewed By: xavierd
Differential Revision: D27526032
fbshipit-source-id: 0d3fa383772f719524a9be84b73fa2eb599580d7
Summary:
`getConfig` was ambiguous, so differentiate EdenConfig and
CheckoutConfig across the implementation.
Reviewed By: genevievehelsel
Differential Revision: D28398762
fbshipit-source-id: 9490e4b31c5d1b0570ee5615e5a3197400397993
Summary: To avoid inconsistency on an unimportant degree of freedom, disallow configs with underscores.
Reviewed By: genevievehelsel
Differential Revision: D28398510
fbshipit-source-id: 7720bd79129102d668d229979733a90547e275da
Summary:
EdenFS doesn't have a mononoke backend anymore, so the configs are no
longer necessary.
Reviewed By: genevievehelsel
Differential Revision: D28398369
fbshipit-source-id: 63f352bd82bc14b745535c227cb213e01f15afe8
Summary:
As EdenConfig grows more knobs, some structure is warranted. I'd
prefer something like anonymous structs or maybe even an
EdenSettingGroup type, but for now add some comments and at least put
them near each other.
Reviewed By: genevievehelsel
Differential Revision: D28398306
fbshipit-source-id: 169c4d2e0eeeb36d1812cdb73c9ac61d11652e09
Summary:
The structure had 2 padding of 4 bytes each due to the alignment requirement of
the EdenMount pointer and of the location_ field. Moving the EdenMount pointer
allows the padding to go away.
Reviewed By: fanzeyi
Differential Revision: D28476945
fbshipit-source-id: b521b4184d3924480ef54f840389156faab3988d
Summary:
This allows the macro to be used for both folly::Future, but also with
ImmediateFuture, reducing the cost of migrating to the latter.
Reviewed By: chadaustin
Differential Revision: D28302478
fbshipit-source-id: d8470428ecaa6aaccf2aa884437ca97d4b45806f
Summary:
Now that both implementation are using ImmediateFuture, we can move the
ImmediateFuture one layer up.
Reviewed By: kmancini
Differential Revision: D28302479
fbshipit-source-id: 3c2c164a90ffb42a0e7da8528f976af34fc87315
Summary:
With the Mountd code being converted to ImmediateFuture, we can now convert
Nfsd3 to use ImmediateFuture too. For now, this isn't expected to perform
better due to the InodeMap still using on folly::Future, which forces the
ImmediateFuture code to store a SemiFuture. A future change will look at
switching the InodeMap to ImmediateFuture to solve this.
Reviewed By: kmancini
Differential Revision: D28297422
fbshipit-source-id: 8b85103657e877b0f102130f2117bbe60598ed52
Summary:
Since lambda are default capturing by const, Func was also captured that way,
which made it impossible to call if Func was actually a mutable lambda.
Reviewed By: chadaustin
Differential Revision: D28297423
fbshipit-source-id: b9c6bdcf024c2e219ec0f8f5be2379e51df24db0
Summary:
As a first towards converting the code to use ImmediateFuture more broadly,
let's start with a small self contained part of the code.
This is mostly a sed except for the dispatchRpc method that needs to call
.semi().via().
Reviewed By: kmancini
Differential Revision: D28297421
fbshipit-source-id: e706e91fc8f132d4ef742ae98af9bb8304e0bf36
Summary:
This code initially was for thenValue, but when I converted it for thenTry I
forgot to also remove the hasException case. We were never calling the callback
on a Try that had an exception.
Reviewed By: chadaustin
Differential Revision: D28302477
fbshipit-source-id: 755fb2c8928627fbe1883f3863cafc360e34a038
