Summary:
In D29079762 (ea2e2f8bbd), globbing was fixed to not match the recursive glob (**) against
the entire path, as this would lead some paths to be matched while they
shouldn't. It however introduced another bug: in some cases, recursive globs
would no longer match paths that should be matched.
To fix both, a partial revert of the original diff is done with a small tweak:
the path that is matched against no longer starts at the root of the
repository. This will prevent `a/b/**/b/c.txt` to match `a/b/c.txt` as
`**/b/c.txt` would only be matched against `c.txt`, and not `a/b/c.txt` like it
was previously.
Reviewed By: fanzeyi
Differential Revision: D29175333
fbshipit-source-id: 1a4137d6f64f6cb77c4be09bd143f72630aa58d5
Summary:
When buck kill fails, eden rm will also fail. This has caused some checkouts
to not be removed when they could be. Stopping aux processes is a nice thing
to do before we unmount. It ensures these processes close file handles in the
mount, but we force unmount anyways so open file handles should not be able to
block the umount call.
Reviewed By: xavierd
Differential Revision: D29205962
fbshipit-source-id: a899940efa5cc1d960cd14a775b7053c34f5d6f2
Summary: This is more confusing than really helpful, thus let's remove the log.
Reviewed By: rkjfb
Differential Revision: D29317007
fbshipit-source-id: 3aba1ab8de7906e193946938aa69b32a09b8e5de
Summary:
Buck v2 builds from the root of the repo, not the current cell. This means that
the inferred logger name ends up being different.
We're going to need to fix this generally because otherwise it'll change logger
names for everyone (I'm tracking this in T93776519), but in the interest of not
having one Eden test arbitrarily failing on Buck v2 let's update this
with a workaround for now.
Reviewed By: genevievehelsel
Differential Revision: D29270388
fbshipit-source-id: 6968d9b6195a5eed7bd4018b161e12d88f78a421
Summary:
I'm suspecting that it is causing some broken pipe errors that aren't
resolving themself.
Reviewed By: kmancini
Differential Revision: D29279304
fbshipit-source-id: cfbf2261f2ac7dd7ec8b3311d1e27a0b9e160de4
Summary:
Invalidation on Windows is tricky, and I got it wrong in subtle ways
previously. The main obvious issue is that when the on-disk invalidation fails,
the refcount shouldn't be decremented as the placeholder/file is still present
on disk. This could cause weird issues in later checkout. The second one is how
invalidating a directory doesn't remove a placeholder (it actually adds one),
and thus we shouldn't decrement the FS refcount. And lastly, the refcount
should be decremented regardless of whether the inode is loaded or unloaded. As
long as it is known by the InodeMap it needs invalidation.
Reviewed By: fanzeyi
Differential Revision: D28970899
fbshipit-source-id: 0d64cadae01fcd4e028c53de9357ece7d648cdd4
Summary:
A user may have some undesirable environment variables when calling `edenfsctl start`,
which we do not want to propagate to edenfs as this may affect EdenFS's
ability to run properly. Having PYTHONPATH set to inside a repository may for
instance lead to a deadlock when EdenFS is trying to setup redirections.
To avoid this, we need to sanitize the environment before calling edenfs. This
functionality already exist but was bypassed on Windows.
Reviewed By: chadaustin
Differential Revision: D29244358
fbshipit-source-id: bc96698732e71412296ed5e28842b59b2c758699
Summary: The original commit broke globbing more than it fixed it. D29175333 will fully fix it, but in the meantime, let's revert the change to get a release out.
Reviewed By: singhsrb
Differential Revision: D29231954
fbshipit-source-id: 7a42e980c6fc4de09bee713a3a4141d52272b6d1
Summary:
Now that the InodeMap is pre-populated with the Overlay, we no longer need the
`folly::kIsWindows` test during checkout as inodes will enter this condition
when unloaded (IsInodeRemembered).
Reviewed By: fanzeyi
Differential Revision: D25106153
fbshipit-source-id: 511d795ae947651e1eaf3c54b8f1ab83c77f5cc4
Summary:
On a non-EdenFS mount, a case change of a file or directory isn't reported as
the file can still be accessed with the same path, and it has the same content
as before. EdenFS had a different behavior whereas renaming a file to a
different case would report a missing file and an untracked one, with different
casing.
This can be surprising to users as the behavior is different from Mercurial,
and it's also hard to actually fix as 2 renames would need to happen, a single
one would not work due to the case insensitivity nature of the filesystem.
While I do believe that reporting the case change might be more desirable to
allow users to actually commit changes the case of files, Mercurial might be
broken in subtle ways today if we allow this, thus it's best to make EdenFS
behave similarly to Mercurial.
Reviewed By: genevievehelsel
Differential Revision: D29150552
fbshipit-source-id: 6cceaa4c9fa61c03f35fcd91a9c01554da252222
Summary:
On case insensitive mounts, updating between commits that have a different
casing for a file/directory would lead to the update failing due to EdenFS
believing that an untracked file is present in the mount. That conflict is
however bogus and EdenFS simply gets confused in
TreeInode::processCheckoutEntry about the entry with the different casing.
To fix this, we should avoid comparing paths in a case sensitive manner and
instead compare then in a case insensitive fashion. This allows the rest of the
checkout code to update the directory/file in place.
On Windows, there is one more subtlety: we can't change the casing of a
file/directory that is already a placeholder and thus we need to force the
entire hierarchy to be removed, this will also make the checkout fail in case
of untracked files in the hierarchy, which is also the behavior on case
sensitive systems.
Reviewed By: genevievehelsel
Differential Revision: D29121741
fbshipit-source-id: 3d2cdacf296a3d061fc828cd6d04d249542cb63f
Summary:
## This diff
* We are migrating `edenfsctl strace` to cpp due to segfaulting from Thrift-py3 (similar to {D25515377 (a152fa4585)}).
* This diff implements new subcmd `edenfsctl trace strace` for this cpp migration.
Reviewed By: xavierd
Differential Revision: D29164534
fbshipit-source-id: 4d8ed23393004f394159c36f71e0c78c077c7c73
Summary:
When running tests, we need a way for Eden to find its privhelper. Normally, it
finds it by looking next to the EdenFS binary itself, but while that works in
Buck 1, it does not work in Buck 2, where the binaries are in separate
directories
Compared to the previous diffs, this requires some extra care because on a real
system, EdenFS is running as a setuid binary, and we don't want to let people
just run whatever binary they want the EdenFS privhelper!
To that end, we reject this argument entirely if the binary is setuid.
Reviewed By: xavierd
Differential Revision: D29061439
fbshipit-source-id: bf9427211d4209cf0bea805b9ea3a53270ec455b
Summary:
In Buck v1 the path we receive for hgPath is absolute so we can use it across
chdirs and the likes.
However, in Buck v2, it isn't. This means that using `--hgPath` doesn't work if
we go and chdir later, which we do because we run from the repo directory.
This updates the unit tests to set the path to an absolute value when they
otherwise look for their hg binary (which they do for operations in HgRepo). If
possible I would have preferred to thread this through to the HgImporter
constructor, but we have quite a few overloaded constructors already and this
turned out to be very messy.
Finally, this also updates the tests themselves to set `hgPath` to `false` so
that if we try to actually run it without configuring it, it won't work.
Reviewed By: xavierd
Differential Revision: D29060828
fbshipit-source-id: 17ad615d7ff98e9ce8278386f018167ec589c336
Summary:
I'd like to be able to override this from unit tests. The underlying goal is
to let us make this path absolute in unit tests, so that we can make Eden unit
tests that rely on hgPath work on Buck v2, where paths are relative.
xavierd suggested overriding the glfag would be the easiest course of action
so that's what this does.
Reviewed By: xavierd
Differential Revision: D29060827
fbshipit-source-id: c4084072e3eef9cc5c9d404d10b3e19028f177ea
Summary:
When matching against the ** pattern, EdenFS would match against the full path
instead of against just the current entry. This means that what was matched by
the glob prior to ** will be rechecked against by the pattern that follows **,
which isn't what is expected.
As a bonus, the `candidateName` variable will be constructed less than before
leading to less memory allocations.
Reviewed By: chadaustin
Differential Revision: D29079762
fbshipit-source-id: af15ecf229ce7119100dd375df23269bb7cdb1c0
Summary:
When using `eden glob` at the root of the repository, the `searchRoot` is
initialized with ".", which is an invalid PathComponent. Let's special case
this in edenfs itself as python's Path appear to always want to be "." and thus
it's easier to fix in EdenFS.
Reviewed By: chadaustin
Differential Revision: D29072478
fbshipit-source-id: 0108294e407784832cd35b3dd6aaceacf4bd7dd9
Summary:
These tests rely on the wall clock and fail somewhat regularly on CI
and on my local machine, so remove them.
Reviewed By: fanzeyi
Differential Revision: D29047847
fbshipit-source-id: 4008d079fac55f52827be534ab1b18b93c1d2f2f
Summary:
Whenever a file is modified in an EdenFS mount and a watchman subscription is
active, watchman will be nodified and will issue a getFilesChangedSince Thrift
call. In order to do that, Watchman ends up always re-creating a new connection
to EdenFS, causing the .eden/socket or .eden/config to be re-read in order to
find EdenFS's socket.
For workloads with heavy write traffic to EdenFS, this readlink/read can add
up. On Windows, writing ~2.5GB worth of data lead Watchman to read over 650MB
worth of data from the .eden/config!
Reviewed By: chadaustin
Differential Revision: D28912258
fbshipit-source-id: 549d57592672ef9cbdcccc213d0612d019677923
Summary:
## This diff
* Implement `getCauseDetail` for Fuse fetch context so that we log which Fuse call triggers the fetch.
Reviewed By: chadaustin, kmancini
Differential Revision: D29092690
fbshipit-source-id: 230ef193332a90a01ca9b89f5b8d8f20cdf4f321
Summary: The new name is `LockedPtr::as_lock`.
Reviewed By: aary
Differential Revision: D28987868
fbshipit-source-id: 8abd6a69a1b9c884adf137f06c24fe0df9ddd089
Summary: Buck has not relied on the .buckversion file for a while now. Trying to clean up the number of configs at the root of the cell for buck. This diff attempts to remove .buckversion code referecnes. Instead of calling `cat .buckversion` to get the buckversion hash, you can call `buck --fast-version` which parses the `buck-java11` file without downloading buck. Alternatively, you can also do something like ` cat .buck-java11 | grep -o -E -e "[0-9a-f]{40}" | head -1` to get the buckversion hash.
Reviewed By: DrMarcII
Differential Revision: D28553844
fbshipit-source-id: ab90fa6a5e7467f6d5c11ffa9d0e10a03433975f
Summary:
Using a PathMap to hold the list of mounts gives us case sensitivity for free
without needing to have #ifdef _WIN32, we can also avoid copying the paths just
to normalize them.
Reviewed By: fanzeyi
Differential Revision: D28969795
fbshipit-source-id: 859cdc3881e36db120913099ce3f451940fa48a4
Summary:
We're planning on rolling out NFS to our users soon, let's make sure NFS is
enabled.
Reviewed By: singhsrb
Differential Revision: D29002395
fbshipit-source-id: 8351fa45c9bd5e68e162baa80e20bd34564b5ece
Summary:
The InodeMap is fully functional on Windows, there is no need to ifdef out its
counters.
Reviewed By: genevievehelsel
Differential Revision: D28970898
fbshipit-source-id: adeb2db19c70d5ff4a3a162fe34c32195fc2a1e0
Summary:
On Windows, the working copy doesn't go away on unmount, instead placeholders
and full files[0] are still present on disk. For this reason, EdenFS needs to
either overly invalidate files and directories at update times, or need to
remember what is present on disk so the state can be recovered.
For now, this diff simply focus on reloading all the inodes that are present on
disk in the InodeMap.
[0]: https://docs.microsoft.com/en-us/windows/win32/projfs/cache-state
Reviewed By: chadaustin
Differential Revision: D28889082
fbshipit-source-id: 90170c1291da563bea455c8032dc8282a093c9b3
Summary: This will implement getTree API of RE-CAS
Reviewed By: chadaustin
Differential Revision: D28157535
fbshipit-source-id: 6b09338f0635aa948497d934a8351a9ba362f1ab
Summary: As we need to debugging a IO related performance issue. We want to see if writes in overlay has caused any issues. This diff adds an option to allow us to create in-memory overlays.
Reviewed By: chadaustin
Differential Revision: D28951905
fbshipit-source-id: 9ef1da183771ee69b855b49bbabc2ef02a7ec9ac
Summary:
Like it says in the title. Right now this dependency is invisible to Buck so
we rely on reading it from the checkout. However, while this works on Buck 1
(I guess we run the tests from the repo root), it doesn't work Buck 2, and it
won't work if we try to run those tests remotely.
Let's make the dependency explicit!
Reviewed By: xavierd
Differential Revision: D28936482
fbshipit-source-id: 71c4a492c33f586abe6b721b90898f5424c06e94
Summary:
Backing stores differentiate between individual tree objects and the
root of a checkout. For example, Git and Mercurial roots are commit
hashes. Allow EdenFS to track variable-width roots to better support
arbitrary backing stores.
Reviewed By: genevievehelsel
Differential Revision: D28619584
fbshipit-source-id: d94f1ecd21a0c416c1b4933341c70deabf386496
Summary:
Making it the default should make cloning using EdenFS easier and will remove
the need to manually pass the `--nfs` command line to `eden clone`.
Reviewed By: kmancini
Differential Revision: D28913818
fbshipit-source-id: 742c35c950cb5edf34fd9769fb78bf26095af0d5
Summary:
WAL is known to be significantly faster than the default DELETE journaling
mode, let's enable it.
Reviewed By: fanzeyi
Differential Revision: D28915718
fbshipit-source-id: 3ee10d10dbaf07a01a23101a6266874a41784cfc
Summary:
These appear to be unecessary, thus there is no need to keep them in place as
it makes the code harder to follow.
Reviewed By: chadaustin
Differential Revision: D28850241
fbshipit-source-id: 129b853e88f0291bb637c9d24de4872a222a5af4
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
A few final write calls need to be threaded as well.
Reviewed By: xavierd
Differential Revision: D28844011
fbshipit-source-id: 214853eea7fa65263fe6415e1ae8b76ca21512d6
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
Some thrift methods used by watchman to get metadata implicitly are not threaded.
These can cause fetches, so let's thread the fetch context here too.
Reviewed By: genevievehelsel
Differential Revision: D28842300
fbshipit-source-id: b1e4b3aea879d6ed7b92afa26184616dedad5935
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
This change is for symlink
Reviewed By: genevievehelsel
Differential Revision: D28841453
fbshipit-source-id: 080eb62f0b562f8e0995c34e9a8302238fc59ed8
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
Most of rmdir is already threaded, not sure if this case can actuall cause
fetches in production, but might as well thread.
Reviewed By: genevievehelsel
Differential Revision: D28840211
fbshipit-source-id: 8dea08e775be470dd1730e2d32750a6912650ee0
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
this change is for rename
Reviewed By: genevievehelsel
Differential Revision: D23467437
fbshipit-source-id: e9d79c65fb5c4d686f0597550e43a0e87c4792cb
Summary:
we now want macFUSE to be installed instead of osxfuse. eden doctor will spew
errors if osxfuse is not installed or loaded, so it perpetually complains.
Update these checks for macFUSE now.
Reviewed By: xavierd
Differential Revision: D28751766
fbshipit-source-id: 4bc61349e33492aebe888a4e869ef7620c74768e
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
This change is for mkdir
Reviewed By: genevievehelsel
Differential Revision: D23458622
fbshipit-source-id: f3914a4f692490434882143664a5d5f1701e93ba
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
This change is for create
Reviewed By: genevievehelsel
Differential Revision: D23457862
fbshipit-source-id: d4c9cc658c26b3119b2b2a1da061e299eaf510c9
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
Most of lookup is already threaded. This finishes the threading for lookup.
Reviewed By: xavierd
Differential Revision: D23456910
fbshipit-source-id: fab7397caeee19f921d8fba1fb6528baa5cf2960
Summary:
BackingStore and LocalStore are no longer tied at the hip, so decouple
FakeBackingStore from LocalStore.
Reviewed By: kmancini
Differential Revision: D28615431
fbshipit-source-id: ee6bc807da6de4ed8fba8ab6d52ff5aeff34e8ae
Summary:
The meaning of the root ID is defined by the BackingStore, so move
parsing and rendering into the BackingStore interface.
Reviewed By: xavierd
Differential Revision: D28560426
fbshipit-source-id: 7cfed4870d48016811b604348742754f6cdbd842
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
this change is for mknod
Reviewed By: chadaustin
Differential Revision: D23452153
fbshipit-source-id: 7b9bc6b624fbe81b91770bc65a0d27bc9d397032
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
this change is for getxattr
Reviewed By: chadaustin
Differential Revision: D23451954
fbshipit-source-id: bae73878754d59661cddf7c0b001e506bbc88d13
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
this change is for readlink
Reviewed By: chadaustin
Differential Revision: D23451821
fbshipit-source-id: 1f8ee369a992ab3489a9366f9a972f67461970de
Summary:
Tree metadata prefetching is a second step after a normal fetch. Let's
add it to the batch tree fetching path. Ideally we'd unify this with the
non-batch tree fetching path a bit. Or just get rid of the non-batch path
entirely.
Reviewed By: chadaustin
Differential Revision: D28427267
fbshipit-source-id: e23fc667f609925e8671a71876165ad2cea161b9
Summary:
Let's move the tree metadata fetching logic to a function so it can be
reused for tree batch fetching.
Reviewed By: chadaustin
Differential Revision: D28719443
fbshipit-source-id: 08197c63918334283e29c8327e244114d33ddb4c
Summary:
Now that the backing store has a batch tree API, let's make Eden take
advantage of it.
This is largely just a copy of how blob batch fetching works. The next diff will
add tree metadata prefetching to this fetch path as well.
Reviewed By: chadaustin
Differential Revision: D28426788
fbshipit-source-id: 0dbd2d37950af88a470ef9d8996e9b912988d363
Summary:
In the eden benchmark, EdenFS didn't return the file list, while Watchman would
always do it, thus let's always return the file list.
I've also removed the calls to SetItemsProcessed as the output is completely
misleading. The "Time" column is what needs to be looked at and not the
"items_per_second" that SetItemsProcessed shows.
Reviewed By: kmancini
Differential Revision: D28664475
fbshipit-source-id: e920b311464667a720d9ab00286e094829e292f7
Summary:
Previously if you `sudo umount -f fbsource-nfs` then try to
`eden mount fbsource-nfs`, the mount will fail because the EdenMount already
exists and is still running.
Let's properly unmount our selfs on a force unmount like we do for fuse.
There are two potential ways to detect a fource unmount: the UMNT call to the
mount deamon or the socket to the nfsd closing. The UMNT call is unreliable
(on Linux we do not get the UNMT call on `umount -l`), so this diff pursues the
socket closing option.
When the nfsd socket is closed we trigger the EdenMount unmounting process if
this has not already started.
Reviewed By: xavierd
Differential Revision: D28329482
fbshipit-source-id: 5df8f3eb818a92536095195f1b3a9e412394fbf6
Summary:
The evaluateRecursiveComponentImpl function calls itself recursively, but would
always check for every invocation whether the recursiveChildren_ was empty. We
know that if evaluateRecursiveComponentImpl was called that recursiveChildren_
wasn't empty, so we can remove the check and move it to its caller.
Reviewed By: chadaustin
Differential Revision: D28724574
fbshipit-source-id: b1e07393d2f1cdb605550eca5f0b0b01591a785d
Summary:
Running the glob benchmark, creating the future causes EdenFS's CPU usage to
explode to 1000% for a single threaded glob query. Removing this significantly
reduce the CPU usage to 100% while not affecting performance, despite what the
comment claimed.
Reviewed By: kmancini
Differential Revision: D28617400
fbshipit-source-id: 42ff3401ff1da9d7a7f63bd1c91c8b5b30b182fd
Summary:
This is the last piece that allows both FUSE and NFS to not allocate futures
when replying to getattr calls.
Reviewed By: chadaustin
Differential Revision: D28396819
fbshipit-source-id: 9a4f8d5d03991dbacc4c56116bd56a38773dcd63
Summary:
Since Buck issues glob requests via Watchman, I was curious to understand the
overhead that Watchman adds to see if that is worth optimizing or bypassing.
From the results shown below, it looks like talking to EdenFS directly would
yield a ~4x globbing improvement.
Reviewed By: chadaustin
Differential Revision: D28644517
fbshipit-source-id: 1aa4c43d45248626adde31491fe40836ad6c436a
Summary:
For non-UTF8 names, the PathComponent constructor would raise an exception, and
since that exception wasn't caught by the handler itself, it would bubble up to
the RPC server and a generic "server IO error" would be sent back to the
client. Since non-UTF8 names aren't a server error, but an invalid argument, we
should instead return a different error.
Unfortunately, EILSEQ isn't an error that an NFS server can return, instead
let's use EINVAL as the argument is clearly invalid.
Reviewed By: chadaustin
Differential Revision: D28482032
fbshipit-source-id: b59044f1a76f7eac79e2df07356a0aeafa22e3c5
Summary:
The modeToNfsMode simply didn't consider all the mode bits to be translated to
the proper NFS mode bits. It now does.
Reviewed By: chadaustin
Differential Revision: D28459428
fbshipit-source-id: d879fb1be2085e44110ba552bc47d2770637fc86
Summary:
An NFS client caches the attributes of files to avoid having to request these
very frequently. What this means is that a file changed by another client (or
by the server itself) may take some time to be reflected on the client, that
time depends on the attribute caching configuration of the mount point.
For EdenFS, files can changed in 2 ways:
- Either it is changed by the user via the mount point,
- Or the user runs an `hg update`
For the first one, the client will simply update its attributes appropriately,
but for the second one, the cached attributes will only be updated when the
user does opens the file, any calls to stat prior will return the old
attributes. Since EdenFS runs on the same host, we can force the attributes
caches to be discarded by simply issuing an open call on the file that changed.
Reviewed By: chadaustin
Differential Revision: D28456482
fbshipit-source-id: 91022d35a33e436c47d94403d0c139992f880cf9
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: fanzeyi
Differential Revision: D28663227
fbshipit-source-id: cee18ec28283c0aef18151465a3c13e9be604d7d
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: xavierd
Differential Revision: D28610639
fbshipit-source-id: c85535669f84695217125af5de7cc4848fba6b2d
Summary: FileDescriptor.cp throws some warning on comparison of signed long and unsigned long. So fix these warning and follow how ```folly/portability/SysUio.cpp:doVecOperation``` did
Reviewed By: fanzeyi
Differential Revision: D28608581
fbshipit-source-id: 3b6e35e9764548ce470634f16a1f6eec5118d7ed
Summary:
Now that makeImmediateFutureWith exists, we can simply use it instead of
constructing a ImmediateFuture<folly::Unit> and calling thenValue on it.
Reviewed By: chadaustin
Differential Revision: D28518059
fbshipit-source-id: 0041cf863fb32efab274f11c77c76109ca9b454f
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: xavierd
Differential Revision: D28665465
fbshipit-source-id: 9281de7fd62f38e09d91435bb53c819bb98fb4ec
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: xavierd
Differential Revision: D28595085
fbshipit-source-id: abc03d210b2e9c5aa19a8925be6d4c426311e826
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: xavierd
Differential Revision: D28594404
fbshipit-source-id: f3dec92403739d67df3ecd091f2d8283a11ea0db
Summary: This will be used to replace calls to folly::makeFutureWith.
Reviewed By: chadaustin
Differential Revision: D28515786
fbshipit-source-id: 2c2c542392e8e57b8f865173d6878cb9d00ba376
Summary: This diff removes the gflag based batch size option and promote it into a EdenFS Configuration so we can experiment with different batch size easily.
Reviewed By: chadaustin
Differential Revision: D28555280
fbshipit-source-id: 6d3a7be3cd880f0aaa3f427c0328222efa2d37ea
Summary:
Copying an InodePtr/shared_ptr adds 2 atomics: one for the copy, and one when
the copy is dropped. In the case where a move can be achieved, we can avoid
these atomics and make the code more efficient.
Reviewed By: fanzeyi
Differential Revision: D28614769
fbshipit-source-id: 9c4be6ce335cc17eb889a5681aa9c13595a7909e
Summary:
Looking at the glob code, I noticed that some paths were copied into lambda and
vectors, while they could have been moved.
Reviewed By: fanzeyi
Differential Revision: D28614211
fbshipit-source-id: 5662586bcc3d9a9c5ca899c59d29b40a4e590adc
Summary:
When the caller asks to not perform sanity checking, this should be honored as
the caller can for instance tell that a path is guaranteed to be valid and
shouldn't be checked.
When running the newly added glob benchmark, EdenFS was spending 7.5% of the
time validating a path that was constructed via the operator+.
Reviewed By: fanzeyi
Differential Revision: D28612665
fbshipit-source-id: 282155d0415ab6458bd9307c89a24e9e090bf09d
Summary:
Querying RocksDB for a blob that we know is not present is expensive, and will
always construct a StoreResult::missing which does allocate memory for that to
simply be dropped. We can revamp the StoreResult a bit, but for now let's
simply not query RocksDB.
In theory, if RocksDB is populated with blobs this can be a loss as more
request will need to go to the hgcache, in practice, RocksDB local caching has
been disabled for so long that we'll always get cache misses.
Reviewed By: fanzeyi
Differential Revision: D28592666
fbshipit-source-id: 49f48097e81eddb4f9c3eba7af774baf018b0821
Summary:
C++ is full of footguns, in this case, the lockContents method either returns a
const std::vector<TreeEntry>&, or an rlock RAII for a folly::Synchronized
object. When assigning the return value of that method to an `auto` variable, a
copy is made in both cases, which isn't exactly what is desired in the first
case. Instead using the reference as-is is what we want.
To achieve the right behavior, a `const auto&` needs to be used: it will either be
a const reference in the first case, or a copy in the second case.
During an `arc focus2` while the Buck parsing phase is ongoing, the copy shows
up at the top of the profiler as one of the most expensive operation. This
significantly reduce its cost.
Reviewed By: chadaustin
Differential Revision: D28591502
fbshipit-source-id: eac1e062ed94442c90ac549ec137d91b4cb42b9c
Summary:
While Buck compiling, when a glob query is running, EdenFS uses tons of CPU,
let's add a small benchmark to profile the code in isolation and keep track of
the progress.
Reviewed By: fanzeyi
Differential Revision: D28611157
fbshipit-source-id: 81df21c8a8aba44fdefec044f9cae387b4cdd15c
Summary:
Add support to our C++ and Python SNAPSHOT parsers for version 2,
which is just an arbitrary binary string after the header.
Don't start writing the format yet. I'd like to roll one release with
read support before writing the new format.
Reviewed By: xavierd
Differential Revision: D28528235
fbshipit-source-id: 0b06596f06cafa26258ab392eaae4a8994f55f1f
Summary: The legacy format was last written in 2017, so we can remove it now.
Reviewed By: xavierd
Differential Revision: D28524663
fbshipit-source-id: 59ed437b0e5ef2e5ee3e3e9944d1049a13d64d9e
Summary:
EdenFS goes out of its way to track the second working copy parent,
but it never uses it. Stop writing it to the SNAPSHOT file.
Reviewed By: genevievehelsel
Differential Revision: D28453213
fbshipit-source-id: d7d36a1c67553f92234bec911051f4f1d4ef1d4a
Summary: This will force future modification to this file to be fully type checked.
Reviewed By: fanzeyi
Differential Revision: D28544422
fbshipit-source-id: 5ecc08a21e49535a3ed4ae9c33f219080365e609
Summary: This will force future changes to this file to be fully type checked.
Reviewed By: fanzeyi
Differential Revision: D28544427
fbshipit-source-id: 0ccfe675818a96ee8fc702317ace85d0b26ccc26
Summary: This will force future changes to this file to be fully type checked.
Reviewed By: fanzeyi
Differential Revision: D28544421
fbshipit-source-id: 67d15e4d6ed13c139adffa7cedf7795856d87c7e
Summary: This will enforce future changes to this file to be fully type checked.
Reviewed By: fanzeyi
Differential Revision: D28544426
fbshipit-source-id: d502de1e0e4c3f7910e0796b2c82ebbb9992be22
Summary:
This adds the few missing types. This will ensure that future modifications of
this file are all type checked.
Reviewed By: fanzeyi
Differential Revision: D28544424
fbshipit-source-id: 4053ede8a2901928e90c484158777aab31937600
Summary: That file was already had good types.
Reviewed By: fanzeyi
Differential Revision: D28544420
fbshipit-source-id: b1b02b7d116ca382001c318fd0626c8d50480013
Summary:
This will enable future modifications of this file to always be properly type
checked.
Reviewed By: fanzeyi
Differential Revision: D28544423
fbshipit-source-id: 73e2fe9ba386c7a0b6165424c383db7d647abcc0
Summary:
This fixes the handful of errors and warnings that Pyre threw when I tried
making this file strict. Enabling pyre strict will force future addition and
modifications to this file to have type checking, increasing the reliability of
the code.
Reviewed By: fanzeyi
Differential Revision: D28543212
fbshipit-source-id: 06b976a37e6f4c3a0bab2eb3ecc4c217497bbbd5
Summary:
On Windows, Buck will spawn one process per test target and all the tests will
run sequentially. What this means is that the inodes tests are running for a
long time due to running single threaded. To remediate this, let's start by
splitting the tests into multiple targets.
Reviewed By: chadaustin
Differential Revision: D28521827
fbshipit-source-id: dc4cd20cb9d1cdc52e84b373bb82f6bae44aa9d4
Summary: For whatever reason some of my debug logs got landed :(
Reviewed By: fanzeyi
Differential Revision: D28510825
fbshipit-source-id: 65bd9165020bfe3afff54109a9f550a460266737
Summary:
This diff implements FSCK for EdenFS Windows.
On Windows, users can still modify EdenFS mounts even when EdenFS is not
running, which may cause mismatch between EdenFS state and on-disk view. This
subsequently may cause issues when EdenFS is running again (such as permission
error, not seeing added entries in `hg status`, etc..).
This diff adds FSCK to EdenFS Windows. It is not exactly same as what fsck
traditionally do on *NIX systems. We are still dubbing it as FSCK since it
works at the same place as eden fsck.
At startup, FSCK will crawl the EdenFS mount to compare the overlay state with
disk state. It then synchronizes the overlay view with what the user has on
disk. Note Windows does not always permit user to modify the mount, it only
allows changes in certain situation. In particular, when the directory is in
Full state, this diff takes advantage of that so we can finish the scanning by
only scans such directories.
One limitation of Windows FSCK is that, it cannot reliably tell if the user
deleted a directory or file from dirty placeholder directories. This is because
ProjectedFS will hide untouched entries under dirty placeholder directory when
EdenFS is not running, and there is no way we can tell if the entry is gone
because of user deletion or hid by ProjectedFS.
This is not perfect but acceptable to some extent. One possible failure scenario is:
1. User creates a directory named `foo`.
2. User writes a file to that directory (`foo/bar`).
3. EdenFS then stops running.
4. User then deletes the entire `foo` directory on disk.
5. EdenFS starts again, `foo` will be recrated with an empty `bar` file. This
will still correctly show in `hg status`, and the user is able to delete
them again.
Reviewed By: xavierd
Differential Revision: D27872753
fbshipit-source-id: c553db568379062ff4504204c1a1785664f87c00
Summary:
We want eden doctor to report a problem about mismatched running and installed version only if the running version is more than 2 weeks older than installed.
Please see task for more context
Reviewed By: fanzeyi
Differential Revision: D28464491
fbshipit-source-id: 34b9b4cf533482f3006100bbf675c89ea7ee6fff
Summary:
While looking at the previous diff, I noticed that a significant amount of
memory was used by the loading promise. Since this loading promise is only used
when an inode is being loaded, its usage is the uncommon case: there should be
significantly more inodes being loaded or unloaded than loading at any given
time in EdenFS's lifetime.
Reviewed By: chadaustin
Differential Revision: D28477391
fbshipit-source-id: e80269506b980d7f217f6bdf59034ef003c4b3fc
Summary:
Similarly to the NFS change, this moves all the folly::Future handling in the
FuseChannel itself instead of inside the dispatcher. This should allow the
inode code to be converted to using ImmediateFuture instead of folly::Future.
Reviewed By: genevievehelsel
Differential Revision: D28372252
fbshipit-source-id: 7aae29d4a32500513c0062dfa42b2c7a3be038db
Summary:
Lambda returning no values would cause the compiler to try to instantiate an
ImmediateFuture<void>, which doesn't compile. We could try special casing the
void type, but it's easier if we simply consider these lambda to return unit.
Reviewed By: genevievehelsel
Differential Revision: D28372253
fbshipit-source-id: 1368ae5dc5e2d4d6a5c3e31bc87ed7d230027c3a
Summary:
The InodeMap can be extremely hot when issuing tons of requests to EdenFS.
Unfortunately, it still needs to do memory allocation due to its use of
folly::Future. By switching to ImmediateFuture we can avoid the memory
allocation, speeding it up slightly.
For the NFS dispatcher, this moves the call to `semi()` inward, allowing us to
target specific inode methods to convert next. For the Fuse dispatcher, I've
simply converted the ImmediateFuture into a Future directly, keeping the rest
of the code unchanged, a subsequent change will convert the dispatcher code to
ImmediateFuture.
Reviewed By: chadaustin
Differential Revision: D28302480
fbshipit-source-id: 4e097a721443f0d52f34a337a96f8a63a9a7cd7c
Summary:
This is to mimic the folly::Future API which allows the `get` and `getTry` to
throw if the Future isn't ready. One difference is that the ImmediateFuture API
has a default argument of a max duration instead of having a separate method.
Since chrono durations are expressed as intmax_t (64-bits on systems supported
by EdenFS) a max duration is virtually infinite.
Reviewed By: chadaustin
Differential Revision: D28424053
fbshipit-source-id: 319493174f31367184dbe0aa811a97145b0310cf
Summary:
DurhamG discovered that setting max_background in fuse_init_out allows
a larger number of concurrent FUSE requests. The documentation
indicates it's intended to affect background requests like readahead,
but empirically you can observe increased live FUSE requests with `rg
-j200` and `eden top`. Default to 1000 because EdenFS can handle a
large amount of concurrency and we want to avoid blob and tree fetches
to block FUSE IO when not necessary.
Reviewed By: xavierd
Differential Revision: D27526032
fbshipit-source-id: 0d3fa383772f719524a9be84b73fa2eb599580d7
Summary:
`getConfig` was ambiguous, so differentiate EdenConfig and
CheckoutConfig across the implementation.
Reviewed By: genevievehelsel
Differential Revision: D28398762
fbshipit-source-id: 9490e4b31c5d1b0570ee5615e5a3197400397993
Summary: To avoid inconsistency on an unimportant degree of freedom, disallow configs with underscores.
Reviewed By: genevievehelsel
Differential Revision: D28398510
fbshipit-source-id: 7720bd79129102d668d229979733a90547e275da
Summary:
EdenFS doesn't have a mononoke backend anymore, so the configs are no
longer necessary.
Reviewed By: genevievehelsel
Differential Revision: D28398369
fbshipit-source-id: 63f352bd82bc14b745535c227cb213e01f15afe8
Summary:
As EdenConfig grows more knobs, some structure is warranted. I'd
prefer something like anonymous structs or maybe even an
EdenSettingGroup type, but for now add some comments and at least put
them near each other.
Reviewed By: genevievehelsel
Differential Revision: D28398306
fbshipit-source-id: 169c4d2e0eeeb36d1812cdb73c9ac61d11652e09
Summary:
The structure had 2 padding of 4 bytes each due to the alignment requirement of
the EdenMount pointer and of the location_ field. Moving the EdenMount pointer
allows the padding to go away.
Reviewed By: fanzeyi
Differential Revision: D28476945
fbshipit-source-id: b521b4184d3924480ef54f840389156faab3988d
Summary:
This allows the macro to be used for both folly::Future, but also with
ImmediateFuture, reducing the cost of migrating to the latter.
Reviewed By: chadaustin
Differential Revision: D28302478
fbshipit-source-id: d8470428ecaa6aaccf2aa884437ca97d4b45806f
Summary:
Now that both implementation are using ImmediateFuture, we can move the
ImmediateFuture one layer up.
Reviewed By: kmancini
Differential Revision: D28302479
fbshipit-source-id: 3c2c164a90ffb42a0e7da8528f976af34fc87315
Summary:
With the Mountd code being converted to ImmediateFuture, we can now convert
Nfsd3 to use ImmediateFuture too. For now, this isn't expected to perform
better due to the InodeMap still using on folly::Future, which forces the
ImmediateFuture code to store a SemiFuture. A future change will look at
switching the InodeMap to ImmediateFuture to solve this.
Reviewed By: kmancini
Differential Revision: D28297422
fbshipit-source-id: 8b85103657e877b0f102130f2117bbe60598ed52
Summary:
Since lambda are default capturing by const, Func was also captured that way,
which made it impossible to call if Func was actually a mutable lambda.
Reviewed By: chadaustin
Differential Revision: D28297423
fbshipit-source-id: b9c6bdcf024c2e219ec0f8f5be2379e51df24db0
Summary:
As a first towards converting the code to use ImmediateFuture more broadly,
let's start with a small self contained part of the code.
This is mostly a sed except for the dispatchRpc method that needs to call
.semi().via().
Reviewed By: kmancini
Differential Revision: D28297421
fbshipit-source-id: e706e91fc8f132d4ef742ae98af9bb8304e0bf36
Summary:
This code initially was for thenValue, but when I converted it for thenTry I
forgot to also remove the hasException case. We were never calling the callback
on a Try that had an exception.
Reviewed By: chadaustin
Differential Revision: D28302477
fbshipit-source-id: 755fb2c8928627fbe1883f3863cafc360e34a038
Summary:
When the passed in callback returns an ImmediateFuture, the code had a small
typing bug where we would try to return an ImmediateFuture<ImmediateFuture<T>>
while the function signature expected an ImmediateFuture<T>. This is due to the
SemiFuture code not coalescing a SemiFuture<ImmediatureFuture<T>> into a
SemiFuture<T>.
Reviewed By: chadaustin
Differential Revision: D28293942
fbshipit-source-id: 1bc8f58d387766f29e573499fb37402ff9b49c83
Summary:
gtest includes some windows headers that will have conflicts with the
folly portability versions. This caused some issues in my in-memory tree
cache diffs (D27050310 (8a1a529fcc)).
We should probably generally be using the folly portable gtests so we can
avoid such issues in the future.
see here for more details: bd600cd4e8/folly/portability/GTest.h (L19)
I ran this with codemod yes to all
- convert all the includes with quotes:
`codemod -d eden/fs --extensions cpp,h '\#include\ "gtest/gtest\.h"' '#include <folly/portability/GTest.h>'`
- convert all the includes with brackets
`codemod -d eden/fs --extensions cpp,h '\#include\ <gtest/gtest\.h>' '#include <folly/portability/GTest.h>'`
- convert the test template
`codemod -d eden/facebook --extensions template '\#include\ <gtest/gtest\.h>' '#include <folly/portability/GTest.h>'`
then used `arc lint` to clean up all the targets files
Reviewed By: genevievehelsel, xavierd
Differential Revision: D28035146
fbshipit-source-id: c3b88df5d4e7cdf4d1e51d9689987ce039f47fde
Summary:
In order to get Watchman to recognize EdenFS mounted as NFS, we previously set
the f_mntfromname to be "edenfs". Unfortunately, Apple decided that when the
NFS server is accessed via a unix socket it would overwrite that and instead
use the path to the socket.
Digging in the source code, I did find that the f_fstypename can be overwritten
by calling fsctl with the right set of arguments. Thus, let's do just that.
Reviewed By: kmancini
Differential Revision: D28270605
fbshipit-source-id: f6be4e394d814806aa03ec3e82b8bc2faf364ea7
Summary:
Due to NFS being designed as a network filesystem, it default to binding on a
TCP socket. For EdenFS, since we're not expecting to mount an actual remote
filesystem, we bind these sockets to localhost. Unfortunately, TCP sockets have
some inherent overhead due to being designed to be reliable over a non-reliable
medium.
On macOS, Apple provides a way to mount an NFS server that is listening on a
unix domain socket. Thanks for unix socket being reliable, the TCP overhead
goes away leading to some higher throughput and lower latency for the NFS
server. For EdenFS, timing `rg foobar` over a directory containing 27k files gives:
NFS over TCP:
rg foobar > /dev/null 0.80s user 5.44s system 567% cpu 1.100 total
NFS over UDS:
rg foobar > /dev/null 0.77s user 5.27s system 679% cpu 0.888 total
osxfuse:
rg foobar > /dev/null 0.87s user 5.59s system 662% cpu 0.975 total
The main drawback of going through a unix socket is that D27717945 (bcf6aa465c) appears to
no longer be effective due to
8f02f2a044/bsd/nfs/nfs_vfsops.c (L3739)
Reviewed By: kmancini
Differential Revision: D28261422
fbshipit-source-id: 25dc1dc78cdb50d6c6550a86ef01ea2c894c110f
Summary:
macOS supports NFS servers that can be reached via a unix socket as a way to
improve performance by reducing the TCP cost. To support this, let's first
allow the socket to bind to to be passed to the RpcServer, and then pass it
through to the privhelper code.
Reviewed By: kmancini
Differential Revision: D28261423
fbshipit-source-id: 78c60aac26353d1da76a67897429b964332df8b3
Summary: Looks like this is causing files to not be immediately updated after an update.
Reviewed By: kmancini
Differential Revision: D28384151
fbshipit-source-id: 61159db64a9f3c9c1f9e54ee0462ca870ff2aecc
Summary:
While ImmediateFuture are expected to be used on values that are mostly
immediate, there are cases where it won't be. In these cases we need a way to
wait for the computation/IO to complete. In order to achieve this, we need to
transform an ImmediateFuture onto a SemiFuture.
Reviewed By: fanzeyi
Differential Revision: D28293941
fbshipit-source-id: 227c0acf1e22e4f23a948ca03f2c92ccc160c862
Summary:
When a T can be default constructed, make an ImmediateFuture default
constructible.
Reviewed By: fanzeyi
Differential Revision: D28292874
fbshipit-source-id: 4c239cc9c3f448652b2bcdc103ea1a81ace46402
Summary: This should help the compiler generate even better code.
Reviewed By: chadaustin
Differential Revision: D28153979
fbshipit-source-id: b1d84c92af4fa760c92624c53d7f57330d7706fa
Summary: This is the same change as D27137328 (a9a1b73418) but for macFUSE.
Reviewed By: kmancini
Differential Revision: D28328029
fbshipit-source-id: c58e146dba2e7e3bdb320f2b5e80946e4a7b3afe
Summary: With the addition of the ability to "background" the prefetches in the daemon itself, we can remove the subprocess backgrounding in the python layer and just depend on the internal backgrounding.
Reviewed By: chadaustin
Differential Revision: D27825274
fbshipit-source-id: aa01dc24c870704272186476be34d668dfff6de5
Summary: getTreeForManifest is no longer called, so remove it.
Reviewed By: genevievehelsel
Differential Revision: D28306796
fbshipit-source-id: e51a32fa7d75c54b2e3525e88c162247b4496560
Summary:
In an NFS mount, the InodeNumber are sent to the client as the unique
identifier for a file. For caching purposes, the client will issue a GETATTR
call on that InodeNumber prior to opening it, to see if the file changed and
thus whether its cache needs to be invalidated.
In EdenFS, the checkout process does unfortunately replace file inodes
entirely, causing new InodeNumber to be created, and thus after an update, an
NFS client would not realize that the content changed, and would thus return
the old content to the application. To solve this, we could approach it in 2
different ways:
- Build a different kind of handle to hand over to the NFS client
- Keep InodeNumber constant during checkout.
After trying the first option, it became clear that this would effectively need
to duplicate a lot of functionality from the InodeMap, but with added memory
consumption. This diff attempts to do the second one.
Reviewed By: chadaustin
Differential Revision: D28132721
fbshipit-source-id: 94d470e33174bb9ffd7db00e1b37924096aac8e9
Summary:
In very hot code path, EdenFS is spending a very large amount of time creating
and destroying folly::Future objects. This is due to the required memory
allocation, as well as the handful of atomics that are happening at creation
time, and these are showing up in EdenFS profiles.
In the steady state, EdenFS actually doesn't need futures, as it often times is
able to service requests from its in-memory caches, in which case we should
ideally just return the value itself and not wrap it in a folly::Future. The
added ImmediateFuture is a step in this direction, as it can hold either an
immediate value, or a folly::SemiFuture, and allow the same API to be used
transparently between these 2.
Reviewed By: genevievehelsel
Differential Revision: D28006802
fbshipit-source-id: 89eaa32e7fa82c44844c4b23c4cb30dbeea46ca8
Summary:
Currently running `edenfsctl top` will crash on Windows:
```
Traceback (most recent call last):
File "C:\Python38\Lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Python38\Lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\tools\eden\bin\edenfsctl.exe\__main__.py", line 3, in <module>
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2253, in zipapp_main
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2236, in main
File "C:\Python38\Lib\asyncio\runners.py", line 43, in run
return loop.run_until_complete(main)
File "C:\Python38\Lib\asyncio\base_events.py", line 616, in run_until_complete
return future.result()
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 2212, in async_main
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\main.py", line 1059, in run
File "C:\tools\eden\bin\edenfsctl.exe\eden\fs\cli\top.py", line 395, in __init__
File "C:\Python38\Lib\curses\__init__.py", line 13, in <module>
from _curses import *
ModuleNotFoundError: No module named '_curses'
```
This diff will let it prints an error message.
Reviewed By: xavierd
Differential Revision: D28207330
fbshipit-source-id: a465fe5941b469f4a1ef964f1d4dc8a593639e7c
Summary:
It looks like argparse's exit was not able to handle asyncio event loop well,
causing edenfsctl to generate a long ugly stack trace when the command line
flag does not parse.
Let's just move the arguments parsing outside the asyncio runloop to avoid this
problem as a whole. In theory it should improve our `--help` time a little bit.
Reviewed By: chadaustin
Differential Revision: D28206622
fbshipit-source-id: 881eefaea73b244eadff0165965085e64dad935f
Summary:
Some user reported to see `edenfsctl restart` crashes due to this call to
os.getuid() since it does not available on Windows. P410914264
https://docs.python.org/3.9/library/os.html#os.getuid
Reviewed By: chadaustin
Differential Revision: D28204262
fbshipit-source-id: 077bf207d8b1b6c014fface63ea93e66057629cd
Summary:
This applies the formatting changes from black v21.4b2 to all covered
projects in fbsource. Most changes are to single line docstrings, as black
will now remove leading and trailing whitespace to match PEP8. Any other
formatting changes are likely due to files that landed without formatting,
or files that previously triggered errors in black.
Any changes to code should be AST identical. Any test failures are likely
due to bad tests, or testing against the output of pyfmt.
Reviewed By: thatch
Differential Revision: D28204910
fbshipit-source-id: 804725bcd14f763e90c5ddff1d0418117c15809a
Summary:
Context: https://fb.workplace.com/groups/edenfswindows/permalink/828914994691047/
Even with D27872753 it doesn't really make sense to have `eden fsck` running on Windows since it requires EdenFS repository to **be unmounted**.
This diff changes it to generate a warning to redirect users to run `eden doctor` instead (which is likely what they need).
Reviewed By: kmancini
Differential Revision: D28203778
fbshipit-source-id: ae105678876903bcf6514252bf07189775f9b187
Summary: This just makes it more obvious _where_ `--force` should be passed.
Reviewed By: genevievehelsel
Differential Revision: D28119590
fbshipit-source-id: 1fbdb4428e9b89e7b66c959f874067485a91d534
Summary:
From what I can see, this was added when EdenFS had a Mononoke store, which is
now long gone, thus we should be able to remove the Curl dependency altogether.
Reviewed By: fanzeyi
Differential Revision: D28037816
fbshipit-source-id: 834f7db64bab5dda1748ad2f033c27a2854b0ba4
Summary: Looks like these aren't needed since these files are owned by a TARGETS file.
Reviewed By: genevievehelsel
Differential Revision: D28101197
fbshipit-source-id: d790530227641bf25e48bd96c8a95dd31f08a954
Summary:
Now that autodeps knows where to find cpptoml.h, we no longer need these
manual annotation.
Reviewed By: kmancini
Differential Revision: D28100956
fbshipit-source-id: 463b73834c500c1d16a4a769af3655938124d49d
Summary:
This diff defines symlink type in `DirType`.
Even though it is not directly used in the FSCK diff. This will allow us to support symlink in EdenFS Windows in the future.
Reviewed By: genevievehelsel
Differential Revision: D28016305
fbshipit-source-id: 67c1aa22e39198f9c91845129695f27b8303a5f1
Summary:
We were ignoring the return value of runWhileMaterialized, and thus we were
returning to FUSE before fallocate returned.
Reviewed By: fanzeyi
Differential Revision: D28081991
fbshipit-source-id: f398942ddb2432e48e80c148abc8edb7e5ada71d
Summary:
folly::via is a Future API, and thus it creates one, which requires allocating
it and then attaching it to the Executore. Since the code to dispatch a request
isn't Future based, we don't need to use folly::via, and we can simply add the
lambda to the Executor directly. This removes expensive memory allocations from
the EventBase.
Reviewed By: kmancini
Differential Revision: D27976674
fbshipit-source-id: 8fa9724a94ba69b071ab894cdbbad0d33733c098
Summary:
Neither macOS, nor Linux are sending multi-fragment requests to the NFS server.
Since supporting these means calling into memmove, which can be expensive for
large requests, let's just remove support for them for now. If somehow macOS
and/or Linux start sending these, the XCHECK(isLast) will catch this and we can
fix the code by then.
Reviewed By: kmancini
Differential Revision: D27976671
fbshipit-source-id: 77c758b2bb36517d22d5b637e6f0ebf84cc19e5b
Summary:
The EventBase is single threaded, and for heavily concurrent client workflows,
it could see a lot activity, thus every cycle saved can be used to drive more
client requests. The construction of the IOBuf doesn't need to be done while in
the EventBase, thus let's build it outside.
Reviewed By: kmancini
Differential Revision: D27976670
fbshipit-source-id: c6c015ef26df1dcb3fc0c5f179e474bafbd71fac
Summary:
Passing 64 to preallocate means that the AsyncSocket code will issue reads of
64 bytes, even though the IOBufQueue has significantly more space available. We
can thus pass a bigger size to preallocate to reduce both the cost of
allocation, and the syscall cost. For heavily concurrent client code, this will
allow us to read more than one request per syscall.
The careful reader may have noticed that for very small requests the code may
reallocate more often that it should as it will always reallocate when falling
under 4KB. This is likely to not be an issue in practice.
Reviewed By: kmancini
Differential Revision: D27976672
fbshipit-source-id: 4c7e3aecc4763ab20854f3c466ce0872332f9b77
Summary:
These are various cleanups that should make the code easier to read, there is
no behavior changes.
Reviewed By: kmancini
Differential Revision: D27976673
fbshipit-source-id: 470eb628ca75bf1712a93c6e9aa3a27c3f314d01
Summary:
Running `rg foobar` in a loop and profiling EdenFS shows that we're spending a
significant amount of time collecting the size of non-materialized files. Since
this will never change, we can easily cache it for much faster access.
Reviewed By: chadaustin
Differential Revision: D27924804
fbshipit-source-id: 8b8af63dcb82664db2ecd81b3fcdc006a3a52d72
Summary: Create a proxy that stored RECAS-> Eden Hash, similar to SCS and HG proxy.
Reviewed By: chadaustin
Differential Revision: D27873498
fbshipit-source-id: 0b3e50e3a74b8f0914547178789cb6684b780866
Summary: Create a RE-CAS backing store with all APIs unimplemented, and Linux only.
Reviewed By: chadaustin
Differential Revision: D27771047
fbshipit-source-id: de00c6e290f924872eae7290b1945e6b3f40d610
Summary:
Chad first noted that deserializing trees from the local store can be expensive.
From the thrift side EdenFS does not have a copy of trees in memory. This
means for glob files each of the trees that have not been materialized will be
read from the local store. Since reading an deserializing trees from the local
store can be expensive lets add an in memory cache so that some of these
reads can be satisfied from here instead.
We collect some cache statistics already, lets expose them through thrift with
the rest of the stats.
Reviewed By: chadaustin
Differential Revision: D27052265
fbshipit-source-id: d7fdf70260599b8df43824e2442471e332c1b0cf
Summary:
Chad first noted that deserializing trees from the local store can be expensive.
From the thrift side EdenFS does not have a copy of trees in memory. This
means for glob files each of the trees that have not been materialized will be
read from the local store. Since reading an deserializing trees from the local
store can be expensive lets add an in memory cache so that some of these
reads can be satisfied from here instead.
Here we actually start to use the cache!
Reviewed By: chadaustin
Differential Revision: D27050310
fbshipit-source-id: e35db193fea0af7f387b6f44c49b5bcc2a902858
Summary:
This introduces some basic unit tests to ensure correctness of the cache.
We are adding tests to cover the simple methods of the object cache since we
are using that code path here. And adding a few sanity check tests to make sure
the cache works with trees.
Reviewed By: chadaustin
Differential Revision: D27050296
fbshipit-source-id: b5f0577c1662483f732bb962c5b40bca8e1dcb40
Summary:
Chad first noted that deserializing trees from the local store can be expensive.
From the thrift side EdenFS does not have a copy of trees in memory. This
means for glob files each of the trees that have not been materialized will be
read from the local store. Since reading an deserializing trees from the local
store can be expensive lets add an in memory cache so that some of these
reads can be satisfied from here instead.
This introduces the class for the in memory cache and is based on the existing
BlobCache. note that we keep the minimum number of entries functionality from
the blob cache. This is unlikely to be needed as trees are much less likely
than blobs to exceed a reasonable cache size limit, but kept since we already
have it.
Reviewed By: chadaustin
Differential Revision: D27050285
fbshipit-source-id: 9dd46419761d32387b6f55ff508b60105edae3af
Summary:
On all the code paths that matter we always acquire the write lock. Since we are
basically just using a simple lock, distributed mutex is a more efficient implementation
that does fancy tricks with cachelines. This improves performance from testing
globs which cause many concurrent reads from the cache.
Reviewed By: chadaustin
Differential Revision: D27810990
fbshipit-source-id: d22470f3f39e2cd3895f5ea772955b62030d154a
Summary:
Now that Object Cache actually does most the work, I am moving the
BlobCache tests to be ObjectCache tests. I am leaving a few blob cache
tests to sanity check that the cache works with blobs.
Reviewed By: chadaustin
Differential Revision: D27776113
fbshipit-source-id: ef58279d93035588beb162ee19173a42e3ca4e5b
Summary:
We would like to use a limited size LRU cache fore trees as well as blobs,
so I am templatizing this to allow us to use this cache for trees.
Trees will not need to use Interest handles, but in the future we could use
this cache for blob metadata, which might want to use interest handles.
Additionally if we at somepoint change the inode tree scheme that would remove
the tree content from the inodes itself, interest handle might be useful for
trees. We could also use this cache proxy hashes which may or may not use
interest handles. Since some caches may want interest handles and others will
not I am creating get/insert functions that work with and without interest
handles.
Reviewed By: chadaustin
Differential Revision: D27797025
fbshipit-source-id: 6db3e6ade56a9f65f851c01eeea5de734371d8f0
Summary:
Thrift setter API is deprecated since it doesn't bring any value over direct assignment. Removing it can reduce build-time and make our codebase more consistent.
If result of `s.set_foo(bar)` is unused, this diff replaces
s.set_foo(bar);
with
s.foo_ref() = bar;
Otherwise, it replaces
s.set_foo(bar)
with
s.foo_ref().emplace(bar)
Reviewed By: xavierd
Differential Revision: D27986185
fbshipit-source-id: d90aaf27f25f2ecfcbbbe7886e0c0d784f607a87
Summary: Allows us to background a prefetch (similar to how prefetch-profile fetches are backgrounded). A thing to note here is that we do not deduplicate fetches for prefetches, but if there is enough busy work between bulk filesystem accesses and the prefetch finishing, this should not be an issue.
Reviewed By: chadaustin
Differential Revision: D27028428
fbshipit-source-id: 5c528fff76719f42151542eaa3499271f7ab6fa3
Summary:
These methods will be used in my later Windows fsck diff as it will need to scan disk state to find changes.
It is a bit unfortunate that we'll need to stick with boost for now. However this should be a fairly easy migration to `std::filesystem` once that is available.
Reviewed By: kmancini
Differential Revision: D27872828
fbshipit-source-id: f6b27a171026aeaaea3db9f17b8f43cfa25004e4
Summary:
Copy from Watchman.
This allows us to show stack trace when EdenFS terminates on Windows.
Reviewed By: chadaustin
Differential Revision: D27896966
fbshipit-source-id: f3238a37a1176f879d5e6bc051ec97031c9a7096
Summary:
When EdenFS is killed, either due to `eden stop` timing out, or when simply
rebooting the host, the edenfs.log becomes filled with fsck errors, which also
slows down the fsck process.
Since we already print the number of errors per mount, limiting ourself to the
first 50 errors is probably good enough.
Reviewed By: fanzeyi
Differential Revision: D27943618
fbshipit-source-id: 2b3e6e3ae4df648d4b1dccf73c71f8dbbded3892
Summary: Mercurial still needs this to work in Python 2 for a few more weeks.
Reviewed By: quark-zju, xavierd
Differential Revision: D27943521
fbshipit-source-id: 2b5106496fbb523cdc97a3dce3ad0cbfab5c17b7
Summary:
We started getting the message
```stderr: eden/fs/utils/SpawnedProcess.cpp:798:21: error: 'getIOExecutor' is deprecated: getIOExecutor is deprecated. To use the global mutable executor use getUnsafeMutableGlobalIOExecutor. For a better solution use getGlobalIOExecutor. [-Werror,-Wdeprecated-declarations]
```
I don't see why we would need a mutable executor here so I chose `getGlobalIOExecutor` over `getUnsafeMutableGlobalIOExecutor`.
Reviewed By: kmancini
Differential Revision: D27912276
fbshipit-source-id: 95b1053f72c2b4eb2746e3c40c0cf76b69d90d6e
Summary: `CaseSensitivity::Sensitive` is better than a mere `true` out of nowhere.
Reviewed By: kmancini
Differential Revision: D27867180
fbshipit-source-id: 39d21d3cc3b70c78c6984c3ddbd65b53520770be
Summary: This diff makes treeoverlay the default overlay type for Windows users.
Reviewed By: kmancini
Differential Revision: D27247658
fbshipit-source-id: 866eafc794eff1c262eab3061f14eb597bea0a66
Summary: This diff allows EdenFS to create tree overlay based on checkout configuration.
Reviewed By: kmancini
Differential Revision: D27242580
fbshipit-source-id: d0ebe33017c16517c117c1886f62bc9c6fe9f466
Summary:
Now that lastCheckoutTime is a single uint64_t, we no longer need a lock to
protect it, simple atomics are sufficient. Since reading an atomic usually
doesn't require any atomic operation, this will save a handful of atomics when
loading an inode where the last checkout time is read.
Reviewed By: chadaustin
Differential Revision: D27860653
fbshipit-source-id: 464e950c949ca243664d213da99d96ff5d0fcbb8
Summary:
The lastCheckoutTime is mostly used to initialize the timestamps of newly
loaded inodes and since these store an EdenTimestamp, we incur a conversion for
every inode loads. Instead of doing this conversion, we can simply make it an
EdenTimestamp directly.
Similarly, the getNow function was always converted to an EdenTimestamp
(sometimes more than once), we can also make it return the EdenTimestamp
directly.
Reviewed By: chadaustin
Differential Revision: D27860652
fbshipit-source-id: 9ea8fe9a312e6c3d8667b93130bb32a46ab92961
Summary:
I am debugging why some people get vim to pop up during a merge conflict and some do not.
also fixes a few lint issues
Reviewed By: DurhamG
Differential Revision: D27684419
fbshipit-source-id: f636d71c18353a3816d1e442c05790cf4fd7b90b
Summary:
While soft mount are nice as they allow the server (edenfs) to die and the
client applications to not end up in D state, this also force a maximum
(non-configuerable) 60s timeout for all IOs, after which application receive a
ETIMEDOUT. Thus, we need to not make the mount hard, thankfully, since the
mount is INTR, applications should not stay in D state if EdenFS dies.
Reviewed By: genevievehelsel
Differential Revision: D27808311
fbshipit-source-id: 17c30e88e5b236418064d8c309d85fdc6f1ca3e9
Summary:
activate recently got broken when we added the prefetch-metadata flag,
this needs to be on activate as well as fetch
Reviewed By: xavierd
Differential Revision: D27778771
fbshipit-source-id: 052710c2f206e66d8042314773b6b408cff4915c
Summary: This seems to have broken the EdenFS HgPrefetch test.
Reviewed By: xavierd
Differential Revision: D27795192
fbshipit-source-id: 80a748036961aa6a5750182bf65637fb76825341
Summary:
Previously we'd skip dynamicconfigs when there wasn't a repo available.
Now that dynamicconfig can represent the NoRepo state, let's load dynamicconfigs
in that situation.
This also supports the case where there is no user name.
Reviewed By: kulshrax
Differential Revision: D26801059
fbshipit-source-id: 377cfffb6695a7fbe31303868a88862259ebf8c4
Summary: Since HeaderClientChannel now accepts a transport unique_ptr there's no need to have this deleter exposed outside of HeaderClientChannel.
Reviewed By: iahs
Differential Revision: D27729209
fbshipit-source-id: 064b03afdfe567b6df6437348596f0f6f97f6aaf
Summary:
This allows unix sockets to be created in the mount. This will allow Buck to
run properly as it tries to create sockets in the repository.
Reviewed By: kmancini
Differential Revision: D27690406
fbshipit-source-id: 5725d68bdda12f3a5882ce48b6bdd02b14cdece4
Summary: This merely adds the types for the procedure
Reviewed By: kmancini
Differential Revision: D27690405
fbshipit-source-id: b94fb03658cabaece4166c29135c5fdf9a613d3c
Summary:
This is roughly the same logic as the UNLINK one with the only difference being
in the handling of "." and "..".
Reviewed By: kmancini
Differential Revision: D27684716
fbshipit-source-id: 86a95c38e6c783bc3a45c0a8b000d0210b6dd0b8
Summary:
This merely adds the types needed for the RMDIR procedure. Implementation will
follow.
Reviewed By: genevievehelsel
Differential Revision: D27684736
fbshipit-source-id: 84f5a4f3dc805e7893853b0de1dc19cb01c1319f
Summary:
To get the size in bytes, we need to multiply the quantity with the block size,
not with itself.
Reviewed By: genevievehelsel
Differential Revision: D27690857
fbshipit-source-id: 7d7ca767881b1118fc24befed230a63f342bc911
Summary:
MacOS does not have the device field like linux that we can use to mark edenfs
nfs mounts. But there is the `f_mntfromname` field. This field more typically
might have the path which this nfs mount is mirrored from, but it should be fine
to hyjack as the edenfs indicator field.
Reviewed By: xavierd
Differential Revision: D27717945
fbshipit-source-id: 056fb39dc3273b68d79c26487fd045f4e7f93b7b
Summary:
With fuse we report "edenfs:" as the device, let's do the same thing with nfs
so watchman can recognize edenfs nfs mounts similarly.
I think its fine to use the standard "edenfs" as the server name in the mount
call rather than the address, from looking at:
https://www.systutorials.com/docs/linux/man/8-mount.nfs/
Reviewed By: xavierd
Differential Revision: D27630764
fbshipit-source-id: 9e476c90ece90e758b98d140c6bf4067dbca3661
Summary: Running these with tsan appears to run properly, let's try to re-enable them.
Reviewed By: genevievehelsel
Differential Revision: D27723525
fbshipit-source-id: 42e61d26cf478cbe808698a6a0615015832180fa
Summary:
The CMake documentation states:
>By default, the value index of value-parameterized tests is replaced by the
>actual value in the CTest test name. If this behavior is undesirable (e.g.
>because the value strings are unwieldy), this option will suppress this
>behavior.
Which appears to be a decent default, but not when the parameter is a pointer,
in which case the test name will contain some hex values.
Reviewed By: genevievehelsel
Differential Revision: D27713222
fbshipit-source-id: 0f15b24d04817384ff975ad7b07e16b744e1eb2e
Summary:
It looks like nfs_test is tripping the nightly pyre infer job causing it to not
publish diff that increase our typing coverage. Let's manually type it to
unblock the nightly job.
Reviewed By: genevievehelsel
Differential Revision: D27682093
fbshipit-source-id: a32df9f5b8eeaef2006de7d64f5adadb763402e8
Summary: Migrate the codebase away from the ad-hoc `folly::uint64ToBufferUnsafe` and to `folly::to_ascii_decimal` which is intended for these cases.
Reviewed By: WillerZ
Differential Revision: D27281875
fbshipit-source-id: 0c98749e4aed9c873853eed2221cf54a89279ff4
Summary:
We are moving our skelton for eden top a bit further, by creating the view
structure for eden top. This creates the widget for the banner of eden top,
the help page and the main page as well as a couple sections of the main
page.
The main page is displayed on default and they are toggled with `h` for help
page and `esc` to return to the mage page. Visable and hidden widgets are not
implemented in termwiz yet so we have to do a bit of hacking to hide and
display widgets for our selves.
Each of the sections is stubed with place holder text for testing.
Reviewed By: xavierd
Differential Revision: D26892620
fbshipit-source-id: a7bb4d0e11f3a8968ef071e7f585d07a9c286880
Summary:
Our logs are filled with this message, making it almost impossible to read,
this doesn't need to be DBG1, let's reduce it to DBG5 similarly to what we do
for blobs.
Reviewed By: kmancini
Differential Revision: D27652526
fbshipit-source-id: c61d502a9f17d8180bad857ff6c1a9033763947d
Summary: Here we further add prefetch-metadata support to prefetching profiles
Reviewed By: genevievehelsel
Differential Revision: D27568542
fbshipit-source-id: 64507125f47cf093c0133c82fcab941ed6495f32
Summary:
On macOS, the mount syscall for NFS expects the arguments to be XDR encoded.
This set of argument roughly match with its Linux counterpart and appears to
start the mount process. It fails early on when trying to access the .hg
directory but this is probably an issue with the NFS server code, not of the
mounting code.
Reviewed By: kmancini
Differential Revision: D27306769
fbshipit-source-id: 697fadfddc4048ef56c3a23f75dd5bdbcc92af1b
Summary:
* use `std::nullopt`
* TODO about sandcastle_instance_id in opensource version
Reviewed By: chadaustin
Differential Revision: D27575732
fbshipit-source-id: bf76970a15fee5a3dc1e4e411ea70f5af7248496
Summary: To be able to quickly find all activity related to given Sandcastle job as discussed in [this workplace post](https://fb.workplace.com/groups/2120196508269853/permalink/2899040243718805).
Reviewed By: kmancini
Differential Revision: D27541803
fbshipit-source-id: a55900064bbee92da902de785ebe0c0e8738c3a2
Summary:
Currently eden on startup prints:
```
Starting edenfs (dev build), pid 190
Opening local RocksDB store...
Opened RocksDB store in 0.073 seconds.
Could not parse config.json file: couldn't read /var/twsvcscm/local/.eden/config.json: No such file or directory
Skipping remount step.
Started edenfs (pid 190)
Logs available at /var/twsvcscm/local/.eden/logs/edenfs.log
```
Would be convenient if it also printed session id, to be able to query scuba straight away.
Reviewed By: chadaustin
Differential Revision: D27522665
fbshipit-source-id: d7d4cf6c97bc551061761f2653375f208e393498
Summary: Refactoring to make the following diff smaller.
Reviewed By: chadaustin
Differential Revision: D27522581
fbshipit-source-id: 8f858714fcbfe4b8f8b1c3678bb2003623abbd94
Summary:
Still trying to enable Scuba logging in ovrsource TD to figure out what caused Buck regression when we started working with multiple watchman instances.
Did not try previous fix of pgrp (deploying stuff on Sandcastle is not trivial), although now I'm not sure that pgrp was the issue. Hard to say.
But now I'm launching CI differently and observe different symptoms.
`eden start` command finishes in 30 seconds (according to logs), but Sandcastle waits for something for 10 minutes and then somebody kills `scribe_cat` and Sandcastle continues. I don't really know what that means, but there's another issue I discovered:
`scribe_cat` inherits a copy of stderr fs created during daemon startup.
This diff fixes the issue.
Reviewed By: kmancini
Differential Revision: D27494520
fbshipit-source-id: 069f4e9ea1efb553cf7a7f18e20ae92c27da808d
Summary:
I see in Sandcastle logs it was 10 minutes between `eden start` command and the following command.
So I'm curious, is it eden took 10 minutes to start or Sandcastle did something else but did not log it?
This little message in log will help to understand that.
Reviewed By: kmancini
Differential Revision: D27491709
fbshipit-source-id: 796c8db5abe49b056bd81b03ea57585a761c3cb6
Summary:
We call `setsid` too late, [on successful startup](https://fburl.com/diffusion/p88am07t), after `scribe_cat` binary launched.
As a result, `scribe_cat` does not belong to process group of eden, it belongs to the process group of launcher process, so it is killed by `timeout` command in ovrsource CI.
To deal with it, this diff proposes additionally calling `setpgrp` call early during initialization.
Reviewed By: chadaustin
Differential Revision: D27436474
fbshipit-source-id: acb168c1ab5b78b1598c34ebece88847a9a6480d
Summary:
In the case where EdenFS crashes, any client performing IO to EdenFS would be
stuck in the D state waiting for the server to be back up and running. Since on
restart EdenFS will chose different ports, this makes it hard/impossible to
actually recover and will keep some process unkillable, often time requiring
rebooting a host.
To avoid this, we can make the mount "soft" which aborts the IO after some time
has elapsed and the server isn't back up. This should enable us to kill these
processes and recover more gracefully.
Reviewed By: chadaustin
Differential Revision: D27402817
fbshipit-source-id: ff81a4360900c4c94665174e3c0cf63402f1533e
Summary:
During an `hg update`, all the loaded inodes are unloaded and forgotten, except
the ones that had a positive refcount, these are forgotten when FUSE calls the
`forget` API. For NFS, this `forget` API doesn't exist but the protocol relies
on file handle to stay valid for some time. The duration for which a file
handle need to stay alive depends on several factor. The root one needs to
always be valid, the rest relies on the lookup cache that the client keeps.
This lookup cache is invalidated when the mtime for the parent directory
changes, but mtime changes are often not immediately detected, as attributes
are cached client side with the `acdirmin` and `acregmin` for directories and
files.
For now, this diff doesn't attempt to deal with files being out of date right
after an `hg update`, it merely accounts for the NFS client being able to pass
old file handles to EdenFS. The astude reader will have noticed that
InodeNumber are never reclaimed for now, for that, a time-based mechanism will
need to be added to forget InodeNumbers that have expired.
Reviewed By: kmancini
Differential Revision: D27405295
fbshipit-source-id: af4a4ce9e31bfcc335608da91f0247b50ab87b3f
Summary: On macOS, the mount argument are encoded with XDR, let's add them first before using them.
Reviewed By: genevievehelsel
Differential Revision: D27306770
fbshipit-source-id: 727824f05d3874119858af60c263267adfb3e61e
Summary:
The returned value was never used by any callers, let's simply not return any
value.
Reviewed By: kmancini
Differential Revision: D27418015
fbshipit-source-id: 2a6f15eee01052cdfa9ae334c34e69f2f0a74407
Summary:
When EdenFS is mounted via FUSE, the changed directory and files are flushed so
the kernel is forced to re-read them to obtain new InodeNumber. When NFS is
used, EdenFS can't directly invalidate the client view of the FS, instead we
need to rely on when and how is this invalidated. On Linux, a file handle won't
be refreshed if its parent directory hasn't changed, which is done by looking
at the mtime of the directory by performing a GETATTR call. It is thus crucial
that the mtime is updated during an `hg update`.
Reviewed By: chadaustin
Differential Revision: D27403729
fbshipit-source-id: 1f7195c6c33e2a34c3bb73145f404d652302d828
Summary:
We can move the ifdef in the function itself. This makes the functions calling
these easier to read, and it's less likely for someone to forgot to add the
proper ifdef. If we also ever want to make these do something on Windows, it
will be easier to do too.
Reviewed By: chadaustin
Differential Revision: D27403730
fbshipit-source-id: c5b78ea7c7eb70eaf8d4974e5bec14296f91576f
Summary:
Non-list optional data can be present in some XDR description, let's special
case it so the intent is clear when declaring XDR datastructures.
Reviewed By: fanzeyi
Differential Revision: D27306768
fbshipit-source-id: 9d4d18bf8deff16f859c6d28a2579341dac8ee6f
Summary:
After receiving a network packet, it's possible that more than one fragment
were received as part of it. We thus need to service all of them before
returning.
This would typically be seen when running `rg` in the repository, which would
cause hangs due to some requests not being serviced as they would stay in the
iobuf queue until a new packet was received.
Reviewed By: kmancini
Differential Revision: D27194038
fbshipit-source-id: 3d81c797b5be7d0466d4acad7208f6a82593b4ca
Summary:
Computing the length of an iobuf chain can be expensive due to having to walk
its entirety. Thanksfully, IOBufQueue can cache the total length when data is
appended to it, which makes computing the length a constant operation.
Reviewed By: kmancini
Differential Revision: D27194037
fbshipit-source-id: af659c162ada61f2796bf407f419f5f15e918c02
Summary:
By moving the work to a background threadpool, we can more quickly go back to
servicing incoming NFS requests and thus allow more work to be done
concurrently. This would allow tools like ripgrep to being able to use multiple
cores to search in the code base.
Reviewed By: genevievehelsel
Differential Revision: D27194040
fbshipit-source-id: 7f1775ddaaa7eaf8776a06d05951cb936cd3fbb5
Summary:
This used to be very useful in the early stages, as a way to manually test the
code, now that the NFS procotol is pretty well defined and tests are actually
running, this has outlived its usefulness, let's simply remove the code.
Reviewed By: kmancini
Differential Revision: D27194039
fbshipit-source-id: af86edd9f438448209a7d14ba66c9b54d90a9594
Summary:
When I wrote the NFS code, I used `std::move` a bit too much, on datastructure
where moving them is equivalent to copying them. Instead, we can simply use
references, which makes the code shorter, and thus more efficient.
Reviewed By: kmancini
Differential Revision: D27172574
fbshipit-source-id: d9f06bf3f519e3539cf5cd0a0c4e4a49ef8009a8
Summary:
This should have been added in D27243075 (5a150e125a) but I forgot to run `hg add` and it
was thus not added...
Reviewed By: fanzeyi
Differential Revision: D27279169
fbshipit-source-id: 69807cc05fef33f51b2a491b66c2e8aeb7136deb
Summary:
While clang has no issue compiling this code, gcc appears to choke on it,
failing to compile. This is unfortunate as this means we need to hardcode the
size of the serialized datastructure and validate it with a test.
Reviewed By: fanzeyi
Differential Revision: D27243075
fbshipit-source-id: 5cd59921bbd5d5be4dfb22789942eb022dac5bbe
Summary: For the prefetch-profile feature, added a matching enable option to go with the existing disable option. Enable temporarily enables the prefetch-profile feature for cases where a user has already used disable.
Reviewed By: kmancini
Differential Revision: D27191711
fbshipit-source-id: cf9cb7d4054e9fcee66bdb5f205562ec79c2757e
Summary:
When copying files from Finder to EdenFS, Finder appears to be issuing a couple
of getxattr calls to EdenFS, however having these being denied by osxfuse
causes the copy to fail, resulting in a subpar experience.
Reviewed By: fanzeyi
Differential Revision: D27137328
fbshipit-source-id: 66bcc2e316d832385da05f0768da9f82ab36a374
Summary:
This is unused, and the Thrift file mentions that it should be fine to remove
in July 2020. It's now March 2021, time to kill it.
Reviewed By: chadaustin
Differential Revision: D26852134
fbshipit-source-id: 2872185edd834f889b78802210071d16b881e14c
Summary: It's best to test for a regular file by comparing dtype_t, not mode_t directly.
Reviewed By: chadaustin
Differential Revision: D26851735
fbshipit-source-id: 62d3833a53748ea9b45794ec6390d158e888c241
Summary:
EdenFS doesn't register itself against the portmap client, and on some system
where it is not started, it appears to not work reliably, crashing EdenFS early
at startup. For now, let's only build it when services need to be registered.
Reviewed By: genevievehelsel
Differential Revision: D27162906
fbshipit-source-id: cc2a8a588a756e54253da31f9bc00fbe4e5312d9
Summary:
In a bunch of places, the code assumes that an EdenMount is associated with a
Fuse channel. With NFS, that's no longer the case, thus let's make sure to
check the return value of mount->getFuseChannel(). In the case where it will
make sense to have something for NFS, I've either added an EDEN_BUG, or a TODO,
so we can come back to it later.
Reviewed By: chadaustin
Differential Revision: D26836431
fbshipit-source-id: c061b8f20199e5af3139a5003827f184f6eac8d4
Summary:
In the case where EdenFS would fail to initialize very early, the optional
server would be empty, and thus trying to dereference it would lead to EdenFS
crashing with no good error message. Let's simply test for this.
Reviewed By: genevievehelsel
Differential Revision: D27162907
fbshipit-source-id: a078b9995a94e5a86cf2893cbebfe7f5ca6d064e
Summary:
The NFS readdir turns out to be pretty similar to the FUSE one, with a couple
of differences. For one, it only populates the directory entry name, it also
puts a limit on the total size of the serialized result, including all the
NFS/XDR overhead.
It is not specified if the . and .. entries need to be returned, but since the
NFS spec is usually pretty explicit about these and makes it clear that this is
for the most part a client burden, I didn't add these. I may have to revisit
this later when I get to manually browse a repository.
Since the READDIR RPC doesn't populate any filehandle, the client will have to
issue a LOOKUP RPC for each entries, potentially leading to some
inefficiencies. A future diff will implement the READDIRPLUS to fix these.
Reviewed By: chadaustin
Differential Revision: D26802310
fbshipit-source-id: b821b57021d0c2dca33427975b1acd665173bc5c
Summary:
This simplifies a handful of tests and will make writing the READDIR RPC a bit
less magic when computing the amount of memory needed per entry.
Reviewed By: chadaustin
Differential Revision: D26802312
fbshipit-source-id: fc66cb68f721ed34c8f9879cdda2cd8db6ed8daa
Summary: This merely adds the types for the READDIR RPC.
Reviewed By: chadaustin
Differential Revision: D26802313
fbshipit-source-id: 634ff9b3f97dc4dba56d225c1fb9eae0a94c02d5
Summary:
Looking at the spec, READDIRPLUS appears to be more complex to implement than
READDIR, for now, let's force the use of READDIR. Future changes will have to
implement READDIRPLUS as that will likely be a perf improvement.
Reviewed By: chadaustin
Differential Revision: D26802311
fbshipit-source-id: cb784d74507e6c2c2ba4dc0aebe69cfcd69db40b
Summary:
This type is very specific to Fuse, let's make it obvious. The readdir method
has also been renamed as it is also very specific to Fuse.
Reviewed By: chadaustin
Differential Revision: D26802309
fbshipit-source-id: c2acdfd1c0006935c59b685fcda729e1bef88928
Summary:
When creating the .hg directory, Mercurial issues a SYMLINK RPC, thus let's
support it.
Reviewed By: kmancini
Differential Revision: D26785005
fbshipit-source-id: a760d55e6117cc3725444c604e3e4036f4a317b2
Summary:
This is mostly just copying /usr/include/linux/fuse.h from my devserver and
updating some flags in FuseChannel to display the new flags.
Reviewed By: chadaustin
Differential Revision: D27144667
fbshipit-source-id: 4854c6edd4c793ca707db26fecd11e2a3e9d7b75
Summary:
Some of our subprocess calls are running into dylib errors. The cause looks to
be related to our environment variables. We already have environment hygenics
for buck, so lets borrow this to use elsewhere.
This is to fix prefetch profile fetching on mac, but I ran into another error
when testing `eden du --clean`.
Reviewed By: genevievehelsel
Differential Revision: D27135268
fbshipit-source-id: 3955ddefc5e9ff60e966f63f7dc65ef737186464
Summary: Free the statement cache before trying to release the database. This is causing issues in `eden rm`.
Reviewed By: chadaustin
Differential Revision: D27104645
fbshipit-source-id: 74f4c4e73b048f58b83a28c2d280e5cb1c767048
Summary:
suppress the deprecated errors.
found something similar in here: https://stackoverflow.com/questions/1902021/suppressing-is-deprecated-when-using-respondstoselector
example failure:
https://www.internalfb.com/intern/buck/build/a3b550b8-4099-4f27-8975-5bfffd6447e5/
```
eden/fs/inodes/test/OverlayTest.cpp:730:1: error:
'InstantiateTestCase_P_IsDeprecated' is deprecated: INSTANTIATE_TEST_CASE_P is deprecated, please use INSTANTIATE_TEST_SUITE_P [-Werror,-Wdeprecated-declarations]
INSTANTIATE_TEST_CASE_P(
/Users/kuki/fbsource/third-party/googletest/googletest/include/gtest/gtest-param-test.h:507:38:
note: expanded from macro 'INSTANTIATE_TEST_CASE_P'
static_assert(::testing::internal::InstantiateTestCase_P_IsDeprecated(), \
/Users/kuki/fbsource/third-party/googletest/googletest/include/gtest/internal/gtest-internal.h:1209:1: note: 'InstantiateTestCase_P_IsDeprecated' has been explicitly marked deprecated here
GTEST_INTERNAL_DEPRECATED(
/Users/kuki/fbsource/third-party/googletest/googletest/include/gtest/internal/gtest-port.h:2215:59:
note: expanded from macro 'GTEST_INTERNAL_DEPRECATED'
#define GTEST_INTERNAL_DEPRECATED(message) __attribute__((deprecated(message)))
```
Reviewed By: mzlee
Differential Revision: D27037957
fbshipit-source-id: b12cc500441c9ed4ed72825475c57047fb0c2076
Summary: This diff replaces `SqliteStatement` in `TreeOverlayStore` with the new `PersistendSqliteStatement`.
Reviewed By: chadaustin
Differential Revision: D26656314
fbshipit-source-id: dbbfeddd52d3be2a1b5d3b6121d3694c41841fd5
Summary:
This diff makes `SqliteStatement` to construct persistent statement by default.
This change should be harmless if the caching assumption is not true. If the statement is not cached by the caller, it's probably on a non-critical path.
Reviewed By: chadaustin
Differential Revision: D26639365
fbshipit-source-id: 26af6ccdba036dc0fccf774e9ffb6bb2a014c5ff
