Summary:
We want to introduce two debug commands to record perf profiles such as files read. This can later be integrated to CI so that we can have this data for troubleshooting perf issues.
* `eden debug start_recording` starts recording perf metrics such as files read/written and fetch counts/latency for a given mount.
* `eden debug end_recording` stops recording and dumps the recorded profile to a local file.
This diff adds the boilerplate `ActivityRecorder` (borrowed heavily from `HiveLogger`'s implementation). The start command would create an instance of the recorder; the end command would destroy the recorder. The recording and dumping are handled by the implementing class.
Reviewed By: genevievehelsel
Differential Revision: D29506895
fbshipit-source-id: a927a363942a041d5ae54186a265576325dfeed5
Summary: Extended eden doctor to check if the PrivHelper is accessible and report when it is not.
Reviewed By: genevievehelsel
Differential Revision: D29593250
fbshipit-source-id: 2390e75b91c9d6f713db4b6084868af91a0b6623
Summary: This creates a HiveLogger and threads it through to the ServerState.
Reviewed By: xavierd
Differential Revision: D29119501
fbshipit-source-id: d0b74733d8832d604f8f4cf0af28e767dbddeddf
Summary:
In Buck v1 the path we receive for hgPath is absolute so we can use it across
chdirs and the likes.
However, in Buck v2, it isn't. This means that using `--hgPath` doesn't work if
we go and chdir later, which we do because we run from the repo directory.
This updates the unit tests to set the path to an absolute value when they
otherwise look for their hg binary (which they do for operations in HgRepo). If
possible I would have preferred to thread this through to the HgImporter
constructor, but we have quite a few overloaded constructors already and this
turned out to be very messy.
Finally, this also updates the tests themselves to set `hgPath` to `false` so
that if we try to actually run it without configuring it, it won't work.
Reviewed By: xavierd
Differential Revision: D29060828
fbshipit-source-id: 17ad615d7ff98e9ce8278386f018167ec589c336
Summary: The new name is `LockedPtr::as_lock`.
Reviewed By: aary
Differential Revision: D28987868
fbshipit-source-id: 8abd6a69a1b9c884adf137f06c24fe0df9ddd089
Summary:
Backing stores differentiate between individual tree objects and the
root of a checkout. For example, Git and Mercurial roots are commit
hashes. Allow EdenFS to track variable-width roots to better support
arbitrary backing stores.
Reviewed By: genevievehelsel
Differential Revision: D28619584
fbshipit-source-id: d94f1ecd21a0c416c1b4933341c70deabf386496
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
A few final write calls need to be threaded as well.
Reviewed By: xavierd
Differential Revision: D28844011
fbshipit-source-id: 214853eea7fa65263fe6415e1ae8b76ca21512d6
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
this change is for rename
Reviewed By: genevievehelsel
Differential Revision: D23467437
fbshipit-source-id: e9d79c65fb5c4d686f0597550e43a0e87c4792cb
Summary:
BackingStore and LocalStore are no longer tied at the hip, so decouple
FakeBackingStore from LocalStore.
Reviewed By: kmancini
Differential Revision: D28615431
fbshipit-source-id: ee6bc807da6de4ed8fba8ab6d52ff5aeff34e8ae
Summary:
The meaning of the root ID is defined by the BackingStore, so move
parsing and rendering into the BackingStore interface.
Reviewed By: xavierd
Differential Revision: D28560426
fbshipit-source-id: 7cfed4870d48016811b604348742754f6cdbd842
Summary: The legacy format was last written in 2017, so we can remove it now.
Reviewed By: xavierd
Differential Revision: D28524663
fbshipit-source-id: 59ed437b0e5ef2e5ee3e3e9944d1049a13d64d9e
Summary:
Similarly to the NFS change, this moves all the folly::Future handling in the
FuseChannel itself instead of inside the dispatcher. This should allow the
inode code to be converted to using ImmediateFuture instead of folly::Future.
Reviewed By: genevievehelsel
Differential Revision: D28372252
fbshipit-source-id: 7aae29d4a32500513c0062dfa42b2c7a3be038db
Summary:
`getConfig` was ambiguous, so differentiate EdenConfig and
CheckoutConfig across the implementation.
Reviewed By: genevievehelsel
Differential Revision: D28398762
fbshipit-source-id: 9490e4b31c5d1b0570ee5615e5a3197400397993
Summary:
gtest includes some windows headers that will have conflicts with the
folly portability versions. This caused some issues in my in-memory tree
cache diffs (D27050310 (8a1a529fcc)).
We should probably generally be using the folly portable gtests so we can
avoid such issues in the future.
see here for more details: bd600cd4e8/folly/portability/GTest.h (L19)
I ran this with codemod yes to all
- convert all the includes with quotes:
`codemod -d eden/fs --extensions cpp,h '\#include\ "gtest/gtest\.h"' '#include <folly/portability/GTest.h>'`
- convert all the includes with brackets
`codemod -d eden/fs --extensions cpp,h '\#include\ <gtest/gtest\.h>' '#include <folly/portability/GTest.h>'`
- convert the test template
`codemod -d eden/facebook --extensions template '\#include\ <gtest/gtest\.h>' '#include <folly/portability/GTest.h>'`
then used `arc lint` to clean up all the targets files
Reviewed By: genevievehelsel, xavierd
Differential Revision: D28035146
fbshipit-source-id: c3b88df5d4e7cdf4d1e51d9689987ce039f47fde
Summary:
macOS supports NFS servers that can be reached via a unix socket as a way to
improve performance by reducing the TCP cost. To support this, let's first
allow the socket to bind to to be passed to the RpcServer, and then pass it
through to the privhelper code.
Reviewed By: kmancini
Differential Revision: D28261423
fbshipit-source-id: 78c60aac26353d1da76a67897429b964332df8b3
Summary: getTreeForManifest is no longer called, so remove it.
Reviewed By: genevievehelsel
Differential Revision: D28306796
fbshipit-source-id: e51a32fa7d75c54b2e3525e88c162247b4496560
Summary:
Chad first noted that deserializing trees from the local store can be expensive.
From the thrift side EdenFS does not have a copy of trees in memory. This
means for glob files each of the trees that have not been materialized will be
read from the local store. Since reading an deserializing trees from the local
store can be expensive lets add an in memory cache so that some of these
reads can be satisfied from here instead.
Here we actually start to use the cache!
Reviewed By: chadaustin
Differential Revision: D27050310
fbshipit-source-id: e35db193fea0af7f387b6f44c49b5bcc2a902858
Summary:
The lastCheckoutTime is mostly used to initialize the timestamps of newly
loaded inodes and since these store an EdenTimestamp, we incur a conversion for
every inode loads. Instead of doing this conversion, we can simply make it an
EdenTimestamp directly.
Similarly, the getNow function was always converted to an EdenTimestamp
(sometimes more than once), we can also make it return the EdenTimestamp
directly.
Reviewed By: chadaustin
Differential Revision: D27860652
fbshipit-source-id: 9ea8fe9a312e6c3d8667b93130bb32a46ab92961
Summary: This seems to have broken the EdenFS HgPrefetch test.
Reviewed By: xavierd
Differential Revision: D27795192
fbshipit-source-id: 80a748036961aa6a5750182bf65637fb76825341
Summary:
Previously we'd skip dynamicconfigs when there wasn't a repo available.
Now that dynamicconfig can represent the NoRepo state, let's load dynamicconfigs
in that situation.
This also supports the case where there is no user name.
Reviewed By: kulshrax
Differential Revision: D26801059
fbshipit-source-id: 377cfffb6695a7fbe31303868a88862259ebf8c4
Summary:
On macOS, the mount syscall for NFS expects the arguments to be XDR encoded.
This set of argument roughly match with its Linux counterpart and appears to
start the mount process. It fails early on when trying to access the .hg
directory but this is probably an issue with the NFS server code, not of the
mounting code.
Reviewed By: kmancini
Differential Revision: D27306769
fbshipit-source-id: 697fadfddc4048ef56c3a23f75dd5bdbcc92af1b
Summary:
Taking a fuse_setattr_in arg means that it can only be used on FUSE, and while
FUSE has been the only backend on UNIX for a while, this is changing with NFS
being added. Thus, we need to find a solution that isn't tied to FUSE.
The solution is fairly simple, let's simply have a struct with optional fields
that needs changing, FUSE and NFS will set these to what needs changing.
Reviewed By: chadaustin
Differential Revision: D26742213
fbshipit-source-id: 16e3e8cdd22d88ace43485d9e3744280de1ee8ad
Summary:
All the callers of these methods have a real context in hand, let's use it
instead of creating a null one.
Reviewed By: genevievehelsel
Differential Revision: D26699271
fbshipit-source-id: 9fb268c9b3194d7e951e30ab5e90c4f2f0388e81
Summary:
Instead of having one "Dispatcher" type that the various backend overload,
let's simply have a per-mount type dispatcher type. The previous model worked
fine when EdenFS supported only one way of mounting a repository, but with NFS
coming, unix platform will support both FUSE and NFS, making the Dispatcher
overload nonsensical.
As a behavioral change, the dispatcher lifetime and ownership is changed a bit.
It used to live for the duration of the EdenMount object, but is now tied to
the channel lifetime, as it is now owned by it.
Reviewed By: kmancini
Differential Revision: D26329477
fbshipit-source-id: 3959b90a4909e3ab0898caa308f54686f59a943c
Summary:
Since mounting EdenFS via NFS requires the same privilege as mounting EdenFS
with FUSE, let's re-use the PrivHelper infrastructure that FUSE already uses.
The macOS bit isn't yet implemented as developing on Linux is for now easier,
the mount args are also overly complicated on macOS and not well documented.
Reviewed By: fanzeyi
Differential Revision: D26255629
fbshipit-source-id: 295261dd40442fe7e0f9439c4f4c25e0d50211a3
Summary:
When NFS is enabled, we will need to have one mountd RPC program running to
be able to serve the mount requests, let's make sure that we have one by adding
it to the ServerState.
On top of the per-mount protocol setting added previously, I've also added a
global setting to make sure that we're not advertising ourself as an NFS server
before the code is more functional.
Reviewed By: genevievehelsel
Differential Revision: D26139991
fbshipit-source-id: 0d9e02618c931a0401394888ea8f3680974966b5
Summary:
The StringPiece constructor is untyped, and was only used in test. We can
afford to build the PathComponent in tests instead to avoid future headaches.
Reviewed By: genevievehelsel
Differential Revision: D25434556
fbshipit-source-id: 4b10bf2576870e81412d76c4b9755b45e26986b3
Summary:
This commit adds a new eden configuration option that
controls whether we try to load our edenfs.kext in preference to
an alternative fuse implementation on macOS.
The majority of this diff is plumbing to convey the configuration
value through to the privhelper, which is relatively restrictive
due to its root-ness.
I've also updated watchman and mercurial to be aware of the new
filesystem type that shows up in the mount table.
Reviewed By: genevievehelsel
Differential Revision: D25065462
fbshipit-source-id: 4f35b9440654298e2706a0d0613d97eb63451999
Summary:
Now that we don't ship Python 2 rpms, let's remove the build and test
targets.
Reviewed By: singhsrb, xavierd
Differential Revision: D25313325
fbshipit-source-id: 876385ccb6cb7675fef8d93978b372a3085764b0
Summary:
Mercurial is incompatible with TSAN at the moment, due to Rust/C++
compilation issues, Python multiprocess, and Tokio false positives. So
disable our HG tests when running under TSAN.
Reviewed By: fanzeyi
Differential Revision: D25109413
fbshipit-source-id: 86e51ebd287e10f92d6e3b8e7bf191e7946c709a
Summary:
This is the plumbing to allow us to skip Metadata prefetching during eden
prefetches. These can trigger a bunch of wasted network requests
when we are fetching files anyways. (These network requests are wasted since we
fetch the file contents and most of them are being throttled on sandcastle anyways.)
We won't necessarily want to skip metadata prefetching always, we will still want it
for the watchman queries, but for `eden prefetch` will probably want to skip it. This
is why we are making it an option in the GlobParams.
Reviewed By: chadaustin
Differential Revision: D24640754
fbshipit-source-id: 20db62d4c0e59fe17cb6535c86ac8f1e3877879c
Summary:
The EdenFS codebase uses folly/logging/xlog to log, but we were still relying
on glog for the various CHECK macros. Since xlog also contains equivalent CHECK
macros, let's just rely on them instead.
This is mostly codemodded + arc lint + various fixes to get it compile.
Reviewed By: chadaustin
Differential Revision: D24871174
fbshipit-source-id: 4d2a691df235d6dbd0fbd8f7c19d5a956e86b31c
Summary:
One of the issue that EdenFS on Windows is currently facing is around
invalidation during an update. In effect, EdenFS is over invalidating, which
causes update to be slower than it should be, as well as EdenFS recursively
triggering ProjectedFS callbacks during invalidation. Both of these are a
sub-par UX.
The reason this issue exist is multi-faceted. First, the update code follows
the "kPreciseInodeNumberMemory" path which enforces that a directory that is
present in the overlay needs to be invalidated, even if it isn't materialized.
The second reason is that no reclamation is done for the overlay, combine the
two and you get an update that gets both slower over time and will issue
significantly more invalidation that is needed.
Solving this is a bit involved. We could for instance start by reclaiming
inodes from the overlay, but this wouldn't be effective as we use the fact that
an inode is present in the overlay as a way to know that the file is cached in
the overlay. If we reclaim from the overlay we simply won't be invalidating
enough and some files will be out of date.
It turns out that we already have a mechanism to track what is cached by the
kernel: the fuse refcount. On Linux/macOS, everytime an inode is returned to
the kernel, this refcount incremented, and the kernel then notifies us when it
forgot about it, at which point the refcount can be decremented. On Windows,
the rules are a bit different, and a simple flag is sufficient: set when we
write a placeholder on disk (either during a directory listing, or when
ProjectedFS asks for it), and unset at invalidation time during update. There
is however a small snag in this plan. On Linux, the refcount starts at 0 when
EdenFS starts as a mount/unmount will clear all the kernel references on the
inodes. On Windows, the placeholder aren't disappearing when EdenFS dies or is
stopped, so we need a way to scan the working copy when EdenFS starts to know
which inodes should be loaded (an UnloadedInode really).
The astute reader will have noticed that this last part is effectively a
O(materialized) operation that needs to happen at startup, which would be
fairly expensive in itself. It turns out that we really don't have choice and
we need to do it regardless due to Windows not disallowing writes to the
working copy when EdenFS is stopped, and thus for EdenFS to be aware of the
actual state of the working copy, it needs to scan it at startup...
The first step in doing all of this is to simply rename the various places that
uses "fuse refcount" to "fs refcount" which is what this diff does.
Reviewed By: chadaustin
Differential Revision: D24716801
fbshipit-source-id: e9e6ccff14c454e9f2626fab23daeb3930554b1a
Summary: This merely filters out the files that don't compile on mode/win.
Reviewed By: chadaustin
Differential Revision: D24456917
fbshipit-source-id: d314ca2936e064133f2fc6a6199b38d17f2d9ef7
Summary: This can be included in all the supported platforms, thus don't #ifdef it.
Reviewed By: wez
Differential Revision: D23858664
fbshipit-source-id: e61da33a97d87cbfab5bb96d2bdaa865d2c01801
Summary:
Now that prjfs/EdenDispatcher is no longer directly tied to ProjectedFS (sort
of, this is still a bit WIP), we can move it out of the prjfs directory onto
the inodes one. This allows breaking the circular dependency cycle mentioned in
the previous diff where prjfs and inodes depend on each other.
For now, this is merely a copy/paste of the code enclosed in big #ifdef _WIN32,
we might be able to do better later, but for now this is properly good enough.
Reviewed By: wez
Differential Revision: D23857539
fbshipit-source-id: 77c620bac1656d01d7daee4dbf8b10694a589751
Summary:
Most of the non-tests callers have an ObjectFetchContext available, let's use
it instead of a static null context. This will help in understanding why some
files/trees are being fetched.
Reviewed By: chadaustin
Differential Revision: D23745752
fbshipit-source-id: b2e145d9e559bde0542adbc5b20ff56ccfc59ece
Summary: This will make it easier to build with Buck.
Reviewed By: fanzeyi
Differential Revision: D23827754
fbshipit-source-id: bf3bf4d607a08b9831f9dfea172b2e923a219561
Summary:
Now that sqliteoverlay has a TARGETS file, we can remove the manual, and use
autodeps to have the right dependencies in the TARGETS files. This will help in
getting EdenFS to build on Windows with buck.
Reviewed By: chadaustin
Differential Revision: D23820312
fbshipit-source-id: 34bfd13d2ae6d11a404a9b913562c7d45a4b3de7
Summary:
on macOS we cannot safely use `fork`.
This commit replaces the use of `fork` in the startup logger subsystem.
This was a little tricky to untangle; originally (prior to any of
the `fork` removal efforts in this diff stack), the startup flow was
to spawn a set of processes via fork:
```
edenfs (setuid)
\-----edenfs (privhelper, as root)
\------edenfs (daemonized)
```
The forked children take advantage of being able to implicitly pass state to
the child processes from the parent. That data flow needs to become explicit
when removing the fork which makes some things a little awkward.
With fork removed:
* `edenfs` unconditionally spawns `edenfs_privhelper` while it has
root privs and before most of the process has been initialized.
* That same `edenfs` process will then spawn a child `edenfs`
process which starts from scratch, but that which needs to
run as the real server instance
* The original `edenfs` instance needs to linger for a while
to remain connected to the controlling tty to pass back the
startup state to the user, before terminating.
This commit deletes the check that `edenfs` is started originally
as root; previously the logic relied on the forked startup logger
continuing past the `daemonizeIfRequested` call and simply deferring
the check until after folly::init. With these changes we can't
easily perform such a check without adding some extra gymnastics
to pass the state around; the place where that is checked is in
the spawned child of the original edenfs, which is not a privileged
process and doesn't know the original euid. I don't believe this
to be a great loss as we tuck `edenfs` away under the libexec dir.
Reviewed By: chadaustin
Differential Revision: D23696569
fbshipit-source-id: 55b95daf022601a4699274d696af419f0a11f6f2
Summary:
This commit moves a compile-time template parameter
to be a runtime boolean parameter.
There's a bit of fan-out that, while I don't think it is
super awesome, isn't super terrible either.
The case sensitivity value is read from the checkout config
added in the prior diff in this stack.
Reviewed By: xavierd
Differential Revision: D23751192
fbshipit-source-id: 46f6fe25bfa6666305096ad9c416b510cd3aac8f
Summary:
Since the Stub.h now only contains NOT_IMPLEMENTED, let's move it to its own
header outside of the win directory.
Reviewed By: genevievehelsel
Differential Revision: D23696244
fbshipit-source-id: 2dfc3204707e043ee6c89595668c484e0fa8c0d0
Summary:
Most of the RequestData code is platform generic, but bits of it are currently
strongly tied to FUSE. By splitting these 2 parts, we will be able to use the
RequestContext class in Windows too and not having to re-implement all the
logic there.
Reviewed By: chadaustin
Differential Revision: D23482072
fbshipit-source-id: 857fd9ca4264d0f308ec10cc487e9ff3eeb5ee16
Summary:
None of these were used, let's remove them.
ps: I thought we had a system to detect unused headers and lint about them?
Reviewed By: chadaustin
Differential Revision: D23465783
fbshipit-source-id: c21a34c9838db29f4fd0057d3be4e0fcb527cd6d
Summary: This file is not fuse specific, therefore, let's move it to a non-fuse folder.
Reviewed By: chadaustin
Differential Revision: D23464460
fbshipit-source-id: f70e94bb0ecc37bd74798fd230dee2058484f31b
