Summary:
This will enable future modifications of this file to always be properly type
checked.
Reviewed By: fanzeyi
Differential Revision: D28544423
fbshipit-source-id: 73e2fe9ba386c7a0b6165424c383db7d647abcc0
Summary:
This fixes the handful of errors and warnings that Pyre threw when I tried
making this file strict. Enabling pyre strict will force future addition and
modifications to this file to have type checking, increasing the reliability of
the code.
Reviewed By: fanzeyi
Differential Revision: D28543212
fbshipit-source-id: 06b976a37e6f4c3a0bab2eb3ecc4c217497bbbd5
Summary:
On Windows, Buck will spawn one process per test target and all the tests will
run sequentially. What this means is that the inodes tests are running for a
long time due to running single threaded. To remediate this, let's start by
splitting the tests into multiple targets.
Reviewed By: chadaustin
Differential Revision: D28521827
fbshipit-source-id: dc4cd20cb9d1cdc52e84b373bb82f6bae44aa9d4
Summary: For whatever reason some of my debug logs got landed :(
Reviewed By: fanzeyi
Differential Revision: D28510825
fbshipit-source-id: 65bd9165020bfe3afff54109a9f550a460266737
Summary:
This diff implements FSCK for EdenFS Windows.
On Windows, users can still modify EdenFS mounts even when EdenFS is not
running, which may cause mismatch between EdenFS state and on-disk view. This
subsequently may cause issues when EdenFS is running again (such as permission
error, not seeing added entries in `hg status`, etc..).
This diff adds FSCK to EdenFS Windows. It is not exactly same as what fsck
traditionally do on *NIX systems. We are still dubbing it as FSCK since it
works at the same place as eden fsck.
At startup, FSCK will crawl the EdenFS mount to compare the overlay state with
disk state. It then synchronizes the overlay view with what the user has on
disk. Note Windows does not always permit user to modify the mount, it only
allows changes in certain situation. In particular, when the directory is in
Full state, this diff takes advantage of that so we can finish the scanning by
only scans such directories.
One limitation of Windows FSCK is that, it cannot reliably tell if the user
deleted a directory or file from dirty placeholder directories. This is because
ProjectedFS will hide untouched entries under dirty placeholder directory when
EdenFS is not running, and there is no way we can tell if the entry is gone
because of user deletion or hid by ProjectedFS.
This is not perfect but acceptable to some extent. One possible failure scenario is:
1. User creates a directory named `foo`.
2. User writes a file to that directory (`foo/bar`).
3. EdenFS then stops running.
4. User then deletes the entire `foo` directory on disk.
5. EdenFS starts again, `foo` will be recrated with an empty `bar` file. This
will still correctly show in `hg status`, and the user is able to delete
them again.
Reviewed By: xavierd
Differential Revision: D27872753
fbshipit-source-id: c553db568379062ff4504204c1a1785664f87c00
Summary:
We want eden doctor to report a problem about mismatched running and installed version only if the running version is more than 2 weeks older than installed.
Please see task for more context
Reviewed By: fanzeyi
Differential Revision: D28464491
fbshipit-source-id: 34b9b4cf533482f3006100bbf675c89ea7ee6fff
Summary:
While looking at the previous diff, I noticed that a significant amount of
memory was used by the loading promise. Since this loading promise is only used
when an inode is being loaded, its usage is the uncommon case: there should be
significantly more inodes being loaded or unloaded than loading at any given
time in EdenFS's lifetime.
Reviewed By: chadaustin
Differential Revision: D28477391
fbshipit-source-id: e80269506b980d7f217f6bdf59034ef003c4b3fc
Summary:
Similarly to the NFS change, this moves all the folly::Future handling in the
FuseChannel itself instead of inside the dispatcher. This should allow the
inode code to be converted to using ImmediateFuture instead of folly::Future.
Reviewed By: genevievehelsel
Differential Revision: D28372252
fbshipit-source-id: 7aae29d4a32500513c0062dfa42b2c7a3be038db
Summary:
Lambda returning no values would cause the compiler to try to instantiate an
ImmediateFuture<void>, which doesn't compile. We could try special casing the
void type, but it's easier if we simply consider these lambda to return unit.
Reviewed By: genevievehelsel
Differential Revision: D28372253
fbshipit-source-id: 1368ae5dc5e2d4d6a5c3e31bc87ed7d230027c3a
Summary:
The InodeMap can be extremely hot when issuing tons of requests to EdenFS.
Unfortunately, it still needs to do memory allocation due to its use of
folly::Future. By switching to ImmediateFuture we can avoid the memory
allocation, speeding it up slightly.
For the NFS dispatcher, this moves the call to `semi()` inward, allowing us to
target specific inode methods to convert next. For the Fuse dispatcher, I've
simply converted the ImmediateFuture into a Future directly, keeping the rest
of the code unchanged, a subsequent change will convert the dispatcher code to
ImmediateFuture.
Reviewed By: chadaustin
Differential Revision: D28302480
fbshipit-source-id: 4e097a721443f0d52f34a337a96f8a63a9a7cd7c
Summary:
This is to mimic the folly::Future API which allows the `get` and `getTry` to
throw if the Future isn't ready. One difference is that the ImmediateFuture API
has a default argument of a max duration instead of having a separate method.
Since chrono durations are expressed as intmax_t (64-bits on systems supported
by EdenFS) a max duration is virtually infinite.
Reviewed By: chadaustin
Differential Revision: D28424053
fbshipit-source-id: 319493174f31367184dbe0aa811a97145b0310cf
Summary:
DurhamG discovered that setting max_background in fuse_init_out allows
a larger number of concurrent FUSE requests. The documentation
indicates it's intended to affect background requests like readahead,
but empirically you can observe increased live FUSE requests with `rg
-j200` and `eden top`. Default to 1000 because EdenFS can handle a
large amount of concurrency and we want to avoid blob and tree fetches
to block FUSE IO when not necessary.
Reviewed By: xavierd
Differential Revision: D27526032
fbshipit-source-id: 0d3fa383772f719524a9be84b73fa2eb599580d7
Summary:
`getConfig` was ambiguous, so differentiate EdenConfig and
CheckoutConfig across the implementation.
Reviewed By: genevievehelsel
Differential Revision: D28398762
fbshipit-source-id: 9490e4b31c5d1b0570ee5615e5a3197400397993
Summary: To avoid inconsistency on an unimportant degree of freedom, disallow configs with underscores.
Reviewed By: genevievehelsel
Differential Revision: D28398510
fbshipit-source-id: 7720bd79129102d668d229979733a90547e275da
Summary:
EdenFS doesn't have a mononoke backend anymore, so the configs are no
longer necessary.
Reviewed By: genevievehelsel
Differential Revision: D28398369
fbshipit-source-id: 63f352bd82bc14b745535c227cb213e01f15afe8
Summary:
As EdenConfig grows more knobs, some structure is warranted. I'd
prefer something like anonymous structs or maybe even an
EdenSettingGroup type, but for now add some comments and at least put
them near each other.
Reviewed By: genevievehelsel
Differential Revision: D28398306
fbshipit-source-id: 169c4d2e0eeeb36d1812cdb73c9ac61d11652e09
Summary:
The structure had 2 padding of 4 bytes each due to the alignment requirement of
the EdenMount pointer and of the location_ field. Moving the EdenMount pointer
allows the padding to go away.
Reviewed By: fanzeyi
Differential Revision: D28476945
fbshipit-source-id: b521b4184d3924480ef54f840389156faab3988d
Summary:
This allows the macro to be used for both folly::Future, but also with
ImmediateFuture, reducing the cost of migrating to the latter.
Reviewed By: chadaustin
Differential Revision: D28302478
fbshipit-source-id: d8470428ecaa6aaccf2aa884437ca97d4b45806f
Summary:
Now that both implementation are using ImmediateFuture, we can move the
ImmediateFuture one layer up.
Reviewed By: kmancini
Differential Revision: D28302479
fbshipit-source-id: 3c2c164a90ffb42a0e7da8528f976af34fc87315
Summary:
With the Mountd code being converted to ImmediateFuture, we can now convert
Nfsd3 to use ImmediateFuture too. For now, this isn't expected to perform
better due to the InodeMap still using on folly::Future, which forces the
ImmediateFuture code to store a SemiFuture. A future change will look at
switching the InodeMap to ImmediateFuture to solve this.
Reviewed By: kmancini
Differential Revision: D28297422
fbshipit-source-id: 8b85103657e877b0f102130f2117bbe60598ed52
Summary:
Since lambda are default capturing by const, Func was also captured that way,
which made it impossible to call if Func was actually a mutable lambda.
Reviewed By: chadaustin
Differential Revision: D28297423
fbshipit-source-id: b9c6bdcf024c2e219ec0f8f5be2379e51df24db0
Summary:
As a first towards converting the code to use ImmediateFuture more broadly,
let's start with a small self contained part of the code.
This is mostly a sed except for the dispatchRpc method that needs to call
.semi().via().
Reviewed By: kmancini
Differential Revision: D28297421
fbshipit-source-id: e706e91fc8f132d4ef742ae98af9bb8304e0bf36
Summary:
This code initially was for thenValue, but when I converted it for thenTry I
forgot to also remove the hasException case. We were never calling the callback
on a Try that had an exception.
Reviewed By: chadaustin
Differential Revision: D28302477
fbshipit-source-id: 755fb2c8928627fbe1883f3863cafc360e34a038
Summary:
When the passed in callback returns an ImmediateFuture, the code had a small
typing bug where we would try to return an ImmediateFuture<ImmediateFuture<T>>
while the function signature expected an ImmediateFuture<T>. This is due to the
SemiFuture code not coalescing a SemiFuture<ImmediatureFuture<T>> into a
SemiFuture<T>.
Reviewed By: chadaustin
Differential Revision: D28293942
fbshipit-source-id: 1bc8f58d387766f29e573499fb37402ff9b49c83
Summary:
gtest includes some windows headers that will have conflicts with the
folly portability versions. This caused some issues in my in-memory tree
cache diffs (D27050310 (8a1a529fcc)).
We should probably generally be using the folly portable gtests so we can
avoid such issues in the future.
see here for more details: bd600cd4e8/folly/portability/GTest.h (L19)
I ran this with codemod yes to all
- convert all the includes with quotes:
`codemod -d eden/fs --extensions cpp,h '\#include\ "gtest/gtest\.h"' '#include <folly/portability/GTest.h>'`
- convert all the includes with brackets
`codemod -d eden/fs --extensions cpp,h '\#include\ <gtest/gtest\.h>' '#include <folly/portability/GTest.h>'`
- convert the test template
`codemod -d eden/facebook --extensions template '\#include\ <gtest/gtest\.h>' '#include <folly/portability/GTest.h>'`
then used `arc lint` to clean up all the targets files
Reviewed By: genevievehelsel, xavierd
Differential Revision: D28035146
fbshipit-source-id: c3b88df5d4e7cdf4d1e51d9689987ce039f47fde
Summary:
In order to get Watchman to recognize EdenFS mounted as NFS, we previously set
the f_mntfromname to be "edenfs". Unfortunately, Apple decided that when the
NFS server is accessed via a unix socket it would overwrite that and instead
use the path to the socket.
Digging in the source code, I did find that the f_fstypename can be overwritten
by calling fsctl with the right set of arguments. Thus, let's do just that.
Reviewed By: kmancini
Differential Revision: D28270605
fbshipit-source-id: f6be4e394d814806aa03ec3e82b8bc2faf364ea7
Summary:
Due to NFS being designed as a network filesystem, it default to binding on a
TCP socket. For EdenFS, since we're not expecting to mount an actual remote
filesystem, we bind these sockets to localhost. Unfortunately, TCP sockets have
some inherent overhead due to being designed to be reliable over a non-reliable
medium.
On macOS, Apple provides a way to mount an NFS server that is listening on a
unix domain socket. Thanks for unix socket being reliable, the TCP overhead
goes away leading to some higher throughput and lower latency for the NFS
server. For EdenFS, timing `rg foobar` over a directory containing 27k files gives:
NFS over TCP:
rg foobar > /dev/null 0.80s user 5.44s system 567% cpu 1.100 total
NFS over UDS:
rg foobar > /dev/null 0.77s user 5.27s system 679% cpu 0.888 total
osxfuse:
rg foobar > /dev/null 0.87s user 5.59s system 662% cpu 0.975 total
The main drawback of going through a unix socket is that D27717945 (bcf6aa465c) appears to
no longer be effective due to
8f02f2a044/bsd/nfs/nfs_vfsops.c (L3739)
Reviewed By: kmancini
Differential Revision: D28261422
fbshipit-source-id: 25dc1dc78cdb50d6c6550a86ef01ea2c894c110f
Summary:
macOS supports NFS servers that can be reached via a unix socket as a way to
improve performance by reducing the TCP cost. To support this, let's first
allow the socket to bind to to be passed to the RpcServer, and then pass it
through to the privhelper code.
Reviewed By: kmancini
Differential Revision: D28261423
fbshipit-source-id: 78c60aac26353d1da76a67897429b964332df8b3
Summary: Looks like this is causing files to not be immediately updated after an update.
Reviewed By: kmancini
Differential Revision: D28384151
fbshipit-source-id: 61159db64a9f3c9c1f9e54ee0462ca870ff2aecc
Summary:
While ImmediateFuture are expected to be used on values that are mostly
immediate, there are cases where it won't be. In these cases we need a way to
wait for the computation/IO to complete. In order to achieve this, we need to
transform an ImmediateFuture onto a SemiFuture.
Reviewed By: fanzeyi
Differential Revision: D28293941
fbshipit-source-id: 227c0acf1e22e4f23a948ca03f2c92ccc160c862
Summary:
When a T can be default constructed, make an ImmediateFuture default
constructible.
Reviewed By: fanzeyi
Differential Revision: D28292874
fbshipit-source-id: 4c239cc9c3f448652b2bcdc103ea1a81ace46402
Summary: This should help the compiler generate even better code.
Reviewed By: chadaustin
Differential Revision: D28153979
fbshipit-source-id: b1d84c92af4fa760c92624c53d7f57330d7706fa
Summary: This is the same change as D27137328 (a9a1b73418) but for macFUSE.
Reviewed By: kmancini
Differential Revision: D28328029
fbshipit-source-id: c58e146dba2e7e3bdb320f2b5e80946e4a7b3afe
Summary: With the addition of the ability to "background" the prefetches in the daemon itself, we can remove the subprocess backgrounding in the python layer and just depend on the internal backgrounding.
Reviewed By: chadaustin
Differential Revision: D27825274
fbshipit-source-id: aa01dc24c870704272186476be34d668dfff6de5
Summary: getTreeForManifest is no longer called, so remove it.
Reviewed By: genevievehelsel
Differential Revision: D28306796
fbshipit-source-id: e51a32fa7d75c54b2e3525e88c162247b4496560
Summary:
In an NFS mount, the InodeNumber are sent to the client as the unique
identifier for a file. For caching purposes, the client will issue a GETATTR
call on that InodeNumber prior to opening it, to see if the file changed and
thus whether its cache needs to be invalidated.
In EdenFS, the checkout process does unfortunately replace file inodes
entirely, causing new InodeNumber to be created, and thus after an update, an
NFS client would not realize that the content changed, and would thus return
the old content to the application. To solve this, we could approach it in 2
different ways:
- Build a different kind of handle to hand over to the NFS client
- Keep InodeNumber constant during checkout.
After trying the first option, it became clear that this would effectively need
to duplicate a lot of functionality from the InodeMap, but with added memory
consumption. This diff attempts to do the second one.
Reviewed By: chadaustin
Differential Revision: D28132721
fbshipit-source-id: 94d470e33174bb9ffd7db00e1b37924096aac8e9
Summary:
In very hot code path, EdenFS is spending a very large amount of time creating
and destroying folly::Future objects. This is due to the required memory
allocation, as well as the handful of atomics that are happening at creation
time, and these are showing up in EdenFS profiles.
In the steady state, EdenFS actually doesn't need futures, as it often times is
able to service requests from its in-memory caches, in which case we should
ideally just return the value itself and not wrap it in a folly::Future. The
added ImmediateFuture is a step in this direction, as it can hold either an
immediate value, or a folly::SemiFuture, and allow the same API to be used
transparently between these 2.
Reviewed By: genevievehelsel
Differential Revision: D28006802
fbshipit-source-id: 89eaa32e7fa82c44844c4b23c4cb30dbeea46ca8
Summary:
Currently running `edenfsctl top` will crash on Windows:
```
Traceback (most recent call last):
File "C:\Python38\Lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Python38\Lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\tools\eden\bin\edenfsctl.exe\__main__.py", line 3, in <module>
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2253, in zipapp_main
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2236, in main
File "C:\Python38\Lib\asyncio\runners.py", line 43, in run
return loop.run_until_complete(main)
File "C:\Python38\Lib\asyncio\base_events.py", line 616, in run_until_complete
return future.result()
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2212, in async_main
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 1059, in run
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\top.py", line 395, in __init__
File "C:\Python38\Lib\curses\__init__.py", line 13, in <module>
from _curses import *
ModuleNotFoundError: No module named '_curses'
```
This diff will let it prints an error message.
Reviewed By: xavierd
Differential Revision: D28207330
fbshipit-source-id: a465fe5941b469f4a1ef964f1d4dc8a593639e7c
Summary:
It looks like argparse's exit was not able to handle asyncio event loop well,
causing edenfsctl to generate a long ugly stack trace when the command line
flag does not parse.
Let's just move the arguments parsing outside the asyncio runloop to avoid this
problem as a whole. In theory it should improve our `--help` time a little bit.
Reviewed By: chadaustin
Differential Revision: D28206622
fbshipit-source-id: 881eefaea73b244eadff0165965085e64dad935f
Summary:
Some user reported to see `edenfsctl restart` crashes due to this call to
os.getuid() since it does not available on Windows. P410914264
https://docs.python.org/3.9/library/os.html#os.getuid
Reviewed By: chadaustin
Differential Revision: D28204262
fbshipit-source-id: 077bf207d8b1b6c014fface63ea93e66057629cd
Summary:
This applies the formatting changes from black v21.4b2 to all covered
projects in fbsource. Most changes are to single line docstrings, as black
will now remove leading and trailing whitespace to match PEP8. Any other
formatting changes are likely due to files that landed without formatting,
or files that previously triggered errors in black.
Any changes to code should be AST identical. Any test failures are likely
due to bad tests, or testing against the output of pyfmt.
Reviewed By: thatch
Differential Revision: D28204910
fbshipit-source-id: 804725bcd14f763e90c5ddff1d0418117c15809a
Summary:
Context: https://fb.workplace.com/groups/edenfswindows/permalink/828914994691047/
Even with D27872753 it doesn't really make sense to have `eden fsck` running on Windows since it requires EdenFS repository to **be unmounted**.
This diff changes it to generate a warning to redirect users to run `eden doctor` instead (which is likely what they need).
Reviewed By: kmancini
Differential Revision: D28203778
fbshipit-source-id: ae105678876903bcf6514252bf07189775f9b187
Summary: This just makes it more obvious _where_ `--force` should be passed.
Reviewed By: genevievehelsel
Differential Revision: D28119590
fbshipit-source-id: 1fbdb4428e9b89e7b66c959f874067485a91d534
Summary:
From what I can see, this was added when EdenFS had a Mononoke store, which is
now long gone, thus we should be able to remove the Curl dependency altogether.
Reviewed By: fanzeyi
Differential Revision: D28037816
fbshipit-source-id: 834f7db64bab5dda1748ad2f033c27a2854b0ba4
