Summary: Fix some build warnings that show up on macOS.
Reviewed By: wez
Differential Revision: D24436572
fbshipit-source-id: bd300110df79fe9d259c8f5ce2b1085cb85a0cac
Summary:
Sometimes you only want to trace writes into the EdenFS checkout, so
add the ability to run `eden strace` with `--reads`, `--writes`, or
both in order to see only those events.
Reviewed By: wez
Differential Revision: D24468539
fbshipit-source-id: a1b3c730987cf86ce3d39952c6a5e9c5edaddfc2
Summary:
Before replacing the existing debugInodeStatus changes, make some
small refactorings to help me understand it.
Reviewed By: xavierd
Differential Revision: D24299970
fbshipit-source-id: 3b60aa8365bb9a45e12357374282ba837ccc36eb
Summary:
Now that Linux/macOS startup no longer uses fork it becomes trivial to share
the same code with Windows. This improves Windows startup in several different ways:
- `edenfsctl start` now displays the status of the start process in the console
- `edenfsctl start` no longer has an arbitrary timeout after which it reports
a timeout even though EdenFS is still starting up.
This also kills a bunch of Windows specific code that is no longer needed.
Reviewed By: fanzeyi
Differential Revision: D24393690
fbshipit-source-id: 28100aec96da81c92d5b592353edceed332e2364
Summary:
Add an `eden strace` command that subscribes to all FUSE events for
the specified mount (or current directory), and streams them to stdout
with response latency measurements and the requesting process.
If there are any pending requests at the time `eden strace` is run,
they are printed first.
Reviewed By: wez
Differential Revision: D24038978
fbshipit-source-id: 59a2112dbdb1708571d12b04bdccaf9eca725cf7
Summary:
Include opcode name and process name in eden.FuseCall Thrift structs
so we can use eden.FuseCall in a later diff in the stack.
Reviewed By: kmancini
Differential Revision: D24036420
fbshipit-source-id: fc6d8f3d174b85e07fac299a6f86b2b2d24f301d
Summary:
Watchman, the only consumer of subscribeStreamTemporary today, does
not use the published JournalPosition values. Instead, it waits a bit,
and calls either getCurrentJournalPosition or
getFilesChangedSince. This is a good thing, because write traffic to a
filesystem can easily exceed the rate at which Watchman (or even
streaming Thrift!) can consume it, spiking memory usage and degrading
performance.
The next diff in this stack will take advantage of this clarified
communication protocol to reduce edenfs memory usage under heavy write
traffic.
Reviewed By: wez
Differential Revision: D24089811
fbshipit-source-id: 49de6be24a78934888162a577a554d50e2fff355
Summary:
InodeBase::getPath is not deterministic under concurrent
renames. Instead, thread the paths down through the recursion so
they're consistent.
Reviewed By: genevievehelsel
Differential Revision: D24297724
fbshipit-source-id: 3e8664ce2bc5159e464d1d76ed37294c4eac1709
Summary: Eden mounts are identified by paths, not by strings, so fix the Thrift signature.
Reviewed By: wez
Differential Revision: D23774513
fbshipit-source-id: c0fb82c48eee5ce4e8fbffef5623f9016ef76e40
Summary:
Before I make further changes to the Journal, improve the comments and
refactor a few small things.
Reviewed By: kmancini
Differential Revision: D24089530
fbshipit-source-id: de9da2c1e6b1c87b6587781cfa55ae7cc4085eeb
Summary:
Scanning through the functionality provided in ServiceFrameworkLight,
it looks like none of it really applies to the edenfs daemon, so break
the dependency entirely. Removing this complexity would have prevented
the regression where we stopped recording Thrift call statistics.
This should also improve our build times a bit, and maybe resource
consumption.
Reviewed By: genevievehelsel
Differential Revision: D24094784
fbshipit-source-id: fefd1a648c7ecba8484296527ff8100269c176b6
Summary:
This will enable to gather a bit more debugging regarding what processes are
fetching data. The one missing bit on Windows is to collect the process name,
for now, a "NOT IMPLEMENTED" placeholder is put in place.
Reviewed By: wez
Differential Revision: D23946258
fbshipit-source-id: 9f7642c7b9207c5b48ffff0f4eb0333af00bc7d5
Summary:
Less #ifdef is always better, since normalizing a path also works on
non-Windows, there is no reason to special case Windows here.
Reviewed By: fanzeyi
Differential Revision: D24020603
fbshipit-source-id: 114dae3bd9a4743230f4c82d219ff74ffc9379c7
Summary:
Replace it by simply using std::chrono facilities to achieve the exact same
behavior.
Reviewed By: chadaustin
Differential Revision: D24027637
fbshipit-source-id: d22eefec5d408ead0b4cfaa20e716f4c10cce0b5
Summary:
There is no reason not to populate this on Windows, so let's do it to remove a
handful of #ifdef.
Reviewed By: chadaustin
Differential Revision: D24003912
fbshipit-source-id: 380b37d6d1a91f13cdc711c0055a5437f1184c20
Summary:
It looks like clang in mode/win choke on the code otherwise: P143369452. It's
definitively not clear to me why. Since fmt::format should pretty much
similarly to folly::to<std::string>, just use that instead.
Reviewed By: chadaustin, fanzeyi
Differential Revision: D23884978
fbshipit-source-id: f4a7e47169c8b78011340ec0a41871c9fa4b7181
Summary:
Now that prjfs/EdenDispatcher is no longer directly tied to ProjectedFS (sort
of, this is still a bit WIP), we can move it out of the prjfs directory onto
the inodes one. This allows breaking the circular dependency cycle mentioned in
the previous diff where prjfs and inodes depend on each other.
For now, this is merely a copy/paste of the code enclosed in big #ifdef _WIN32,
we might be able to do better later, but for now this is properly good enough.
Reviewed By: wez
Differential Revision: D23857539
fbshipit-source-id: 77c620bac1656d01d7daee4dbf8b10694a589751
Summary:
Most of the non-tests callers have an ObjectFetchContext available, let's use
it instead of a static null context. This will help in understanding why some
files/trees are being fetched.
Reviewed By: chadaustin
Differential Revision: D23745752
fbshipit-source-id: b2e145d9e559bde0542adbc5b20ff56ccfc59ece
Summary:
Now that the win directory only contains the mount directory, we can rename it
to be more faithful to its intent. Since this is about ProjectedFS, let's
rename it "prjfs".
Reviewed By: chadaustin
Differential Revision: D23828561
fbshipit-source-id: cb31fe4652fd4356dc2579028d3ae2c7935371a7
Summary: This will make it easier to build with Buck.
Reviewed By: fanzeyi
Differential Revision: D23827754
fbshipit-source-id: bf3bf4d607a08b9831f9dfea172b2e923a219561
Summary:
on macOS we cannot safely use `fork`.
This commit replaces the use of `fork` in the startup logger subsystem.
This was a little tricky to untangle; originally (prior to any of
the `fork` removal efforts in this diff stack), the startup flow was
to spawn a set of processes via fork:
```
edenfs (setuid)
\-----edenfs (privhelper, as root)
\------edenfs (daemonized)
```
The forked children take advantage of being able to implicitly pass state to
the child processes from the parent. That data flow needs to become explicit
when removing the fork which makes some things a little awkward.
With fork removed:
* `edenfs` unconditionally spawns `edenfs_privhelper` while it has
root privs and before most of the process has been initialized.
* That same `edenfs` process will then spawn a child `edenfs`
process which starts from scratch, but that which needs to
run as the real server instance
* The original `edenfs` instance needs to linger for a while
to remain connected to the controlling tty to pass back the
startup state to the user, before terminating.
This commit deletes the check that `edenfs` is started originally
as root; previously the logic relied on the forked startup logger
continuing past the `daemonizeIfRequested` call and simply deferring
the check until after folly::init. With these changes we can't
easily perform such a check without adding some extra gymnastics
to pass the state around; the place where that is checked is in
the spawned child of the original edenfs, which is not a privileged
process and doesn't know the original euid. I don't believe this
to be a great loss as we tuck `edenfs` away under the libexec dir.
Reviewed By: chadaustin
Differential Revision: D23696569
fbshipit-source-id: 55b95daf022601a4699274d696af419f0a11f6f2
Summary:
On macOS we cannot safely use `fork` to spawn processes while other threads may initialize objc classes.
This commit replaces the use of `fork` in the privhelper startup with
`SpawnedProcess` instead. We need to take care with this as we are generally
installed setuid root and we'd like to avoid being tricked into running an
arbitrary child process as root.
This commit defines a separate executable called `edenfs_privhelper` that
contains just the privhelper server code.
We need to be careful about locating this executable; to avoid invoking an
arbitrary process while we have root privileges we require that the privhelper
be a sibling to the edenfs executable and carry out some additional ownership
verification so that we can tell that the owner of edenfs also controls
edenfs_privhelper.
To facilitate this, I've added an `executablePath` function to PathFuncs; it
returns the path to the current executable image.
To make the integration test scenario simpler, I've added the edenfs_executable
binary definition alongside that of the edenfs binary in the buck and cmake
build systems. This causes the binaries to be siblings in-situ in the build
tree and avoids the need to move things into place in the test harness.
Reviewed By: chadaustin
Differential Revision: D23653343
fbshipit-source-id: 3c2539a5e0e11cee88960db49c885ce0366d314e
Summary:
Since the Stub.h now only contains NOT_IMPLEMENTED, let's move it to its own
header outside of the win directory.
Reviewed By: genevievehelsel
Differential Revision: D23696244
fbshipit-source-id: 2dfc3204707e043ee6c89595668c484e0fa8c0d0
Summary:
The ProcessNameCache code is compiled on Windows now, this definiton could
cause issues with different cpp files compiling different version of the
ProcessNameCache. To avoid this, let's remove it from Stub.h, this also removes
a bunch of #ifdef.
Reviewed By: chadaustin
Differential Revision: D23693490
fbshipit-source-id: 8f3f7b1128235b9a60f850e688b9e98910c202fc
Summary:
In some very rare scenario, we're seeing bind fail due to WSAEADDRINUSE, and
I'm suspecting that a connect race is going on, where connect creates the file
on disk while a bind is ongoing. Since at this point there is no way for
another EdenFS process to be alive due to the fs lock being held, there is no
way for the socket to be already bound to another EdenFS process. Thus, we can
remove the special case.
Reviewed By: genevievehelsel
Differential Revision: D23591009
fbshipit-source-id: 094a7926aa9ac45ea9dea5646000aa287a9b5b7a
Summary:
Removing Fuse from the enum name makes it non tied to Fuse and thus makes it
more portable. This also eliminates the last platform specific bit from
RequestData.
Reviewed By: chadaustin
Differential Revision: D23467773
fbshipit-source-id: 52515522c8ac51d0c4b56dc5e42d4b6593df6623
Summary:
This commit introduces a new process spawning class derived
from the ChildProcess class in the watchman codebase.
`SpawnedProcess` is similar to folly::Subprocess but is designed around the
idea that we will use a system provided spawning API to start a process, rather
than assuming the use of `fork`.
`fork` is to be avoided because it can be expensive for processes with large
address spaces and also because it interacts poorly with threads on macOS. In
particular, we see the objC runtime terminating our process in some scenarios
where fork and threads are mixed.
There are some important differences from `folly::Subprocess` and that means
that some assumptions and uses need to be altered slightly from their prior
workings. For example, detaching a SpawnedProcess moves the responsibility of
waiting on the child to a periodic task as there is no way to detach via
posix_spawn without also using fork.
On the plus side, this commit allows unifying spawning between posix and
windows systems, which simplifies the code!
Reviewed By: xavierd
Differential Revision: D23287763
fbshipit-source-id: b662af1d7eaaa9ed445c42f6c5765ae9af975eea
Summary:
As previous diffs in the stack show there were at least one place in the
codebase which used incorrect object context logger and that resulted in "blind
spots" in undesired file fetches logging i.e. undesired file fetches were
logged, but neither pid nor cmd-line was logged.
There are quite a few places in the codebase that use null
object fetch context, and threading the correct object fetch context to all of
them might be hard. Threading the context is a bit annoying, so it would be good to know something like "EdenDispatcher code is responsible for most of the blind spots, so let's thread the correct context there first". Or it would be equally good to know that none of the null object context are responsible for blind spots.
This diff might help us decide where we need to thread real object fetch context
first. Instead of passing null object fetch context let's pass null object
fetch context with causeDetail field. This field will be logged to scuba (see
BackingStoreLogger::logImport code), and instead of getting "Unknown" interface
we'll get e.g. "Unknown - EdenDispatcher::create", and that would highlight
where we need to thread the context.
A note about implementation - getNullContextWithCauseDetail returns a raw pointer
which is expected to be static i.e. it should work similarly to current
getNullContext implementation. It's quite a hack, but allows us to get rid of
memory allocations (we'd have one memory allocation per place in the code where
getNullContextWithCauseDetail). Let me know if you are ok with this hack.
Reviewed By: kmancini
Differential Revision: D23422526
fbshipit-source-id: e576bba9fc09e160fc42771c7589cdd1694d93c0
Summary: Use `Subprocess` in `win/utils` to call `edenfsctl redirection fixup` after mount is done.
Reviewed By: wez
Differential Revision: D22958764
fbshipit-source-id: a485994a3816169299e8514a5c355f3d37edad99
Summary:
Up to now, Windows had to have its own version of folly::{readFile, writeFile,
writeFileAtomic} as these only operate on `char *` path, which can only
represent ascii paths on Windows. Since the Windows version is slightly
different from folly, this forced the code to either ifdef _WIN32, or use the
folly version pretending that it would be OK. The Windows version was also
behaving slightly differently from folly. For instance, where folly would
return a boolean to indicate success, on Windows we would throw an exception.
To simplify our code, add type safety and unify both, we can implement our own
wrappers on top of either folly or Windows APIs.
We still have some code that uses folly::readFile but these should only be
dealing with filedescriptors. As a following step, we may want to have our own
File class that wraps a file descriptor/HANDLE so we can completely remove all
uses of folly::readFile.
Reviewed By: wez
Differential Revision: D23037325
fbshipit-source-id: 2b9a026f3ee6220ef55097abe649b23e38d9fe91
Summary: Previously we use HgImporter prefetch request in `prefetchBlobs()`, but using `getBlob()` can give us more control over the prefetch process later. So now `getBlob()` is used in `prefetchBlobs()` when `useEdenNativePrefetch` is configured as true.
Reviewed By: kmancini
Differential Revision: D22984848
fbshipit-source-id: 0bd0b1c5b50bb16da36f188915904d0223827dc3
Summary: fix the problem of printing 0% fsck progress bars when fsck is actually not happening.
Reviewed By: genevievehelsel
Differential Revision: D22927401
fbshipit-source-id: 868fa188e3c2671b0f6e18e842ec6a23281dc337
Summary: In cases where `stopWorkersOnStopListening` is disabled, `ThriftServer` continues to process inflight and even new requests after `ThriftServer::serve()` has returned. This is unintuitive and error-prone. This diff ensures that when `serve()` returns, there are no outstanding requests and new requests will not be processed.
Reviewed By: yfeldblum, genevievehelsel
Differential Revision: D22827447
fbshipit-source-id: cda35843ee6be084042e1a7c806c77fb472dd114
Summary: Since multiple mount points can share the same BackingStore object, it is better to start recording profile for each unique backing store instead of get backing stores by mounts.
Reviewed By: chadaustin
Differential Revision: D22904499
fbshipit-source-id: f95508044912503509c4ac0f84b4a061e41ca0f3
Summary:
added `record-profile` and `finish-profile` subcommands to `eden prefetch`
`eden prefetch record-profile` triggers `startRecordingFetch()` in `HgQueuedBackingStore`. `eden prefetch finish-profile` triggers `stopRecordingFetch()` in `HgQueuedBackingStore` and write the returned fetched file names to a text file. We can use the output file to predict what files are fetched for frequently used commands.
Reviewed By: chadaustin
Differential Revision: D22797896
fbshipit-source-id: ff240829b4fb3c47093279a1f2bd453009ff5f73
Summary: Since local store compaction is not a hard requirement for graceful restart, make this issue non blocking. We've seen some users fail restarts because they had compaction issues due to lack of space on their device. If we fail during the compaction stage, we should continue the restart anyway. This is also because there is a chance that the local store will clear columns that are no longer in use.
Reviewed By: chadaustin
Differential Revision: D22828433
fbshipit-source-id: 9a2aaec64e77c2d00089834fda8f8cffda472735
Summary:
TL:DR:
A codemod did something a bit unclean, so they added a lint. This will keep bugging us if we make changes here, so let's satisfy the linter.
More info:
`x.y_ref() = ...` and `*x.y_ref() = ...` are pretty much the same except `*x.y_ref() = ...` can throw for optional fields.
A codemod added a bunch of `*x.y_ref() = ...`, but after they didn't want people to copy paste this for optional fields so they added a lint that pops up on non optional fields too :(
https://fb.workplace.com/groups/thriftusers/permalink/509303206445763/
Reviewed By: chadaustin
Differential Revision: D22823686
fbshipit-source-id: b3b1b8a3b6b1f1245176be19c961476e4554a8e5
Summary:
`log()` passes fsck bars to standard output, but it will also print the same message to the log with level DBG2. (example below)
```V0713 07:05:45.971511 3510654 StartupLogger.cpp:96] [====================>] 100%: fsck on /home/ailinzhang/eden-state/clients/dev-fbsource6/local
```
Since we don't want the log file to be messed up with fsck bars, we use `logVerbose()` with level DBG7.
Reviewed By: kmancini
Differential Revision: D22727965
fbshipit-source-id: 0700503af511030df2abbca4ad2fa1540995e919
Summary:
Unfortunately, using sqlite causes `edenfsctl prefetch` to take several orders
of magnitude more time than with rocksdb, tuning sqlite to be faster (at the
expense of reliability) still doesn't come closer to rocksdb. From profiling,
this is due to sqlite only allowing a single thread to read/write onto it
serializing all the work that EdenFS is doing.
A rework on the storage will be necessary to be able to get both good
performance, and reliability but that's a long term project, for now, make
Windows use rocksdb by default.
Reviewed By: wez
Differential Revision: D22819084
fbshipit-source-id: 62f397858ed547da30ef8a6346b767461dc53493
Summary:
This adds logging for data fetches that come from the thrift globfiles call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: genevievehelsel
Differential Revision: D22489512
fbshipit-source-id: 040cf1277205d08ea864a1f30f8d3b25ee7c4508
Summary:
This adds logging for data fetches that come from the thrift getSHA1 call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: genevievehelsel
Differential Revision: D22489514
fbshipit-source-id: eb2d82c206af857cc79439d2854d682641292db8
Summary:
This adds logging for data fetches that come from the thrift getFileInformation call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: genevievehelsel
Differential Revision: D22489513
fbshipit-source-id: ab6283476d05b06b9f9e37c6b4fd81c1282046ff
Summary:
This adds logging for data fetches that come from the thrift checkout call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: chadaustin
Differential Revision: D22489504
fbshipit-source-id: 3b732a1e5627c2130f561ec0138a1df270e1925d
Summary:
We have seen that some of the unexpected data fetches do not originate from
FUSE. This adds parity to the logging for data fetches that come from the thrift
interface. Adding this logging improves the overall observability of eden, and
will help us debug the cause of unexpected data fetching.
This introduces plumbing to allow logging data fetches that originate from
thrift requests.
Reviewed By: chadaustin
Differential Revision: D22448048
fbshipit-source-id: a39dde72467c4922c07c569c14fb499341d40258
Summary:
What does this do: allocate a ThriftLogHelper and ensure it lives for the
duration of its coresponding thrift request.
Why: By ensuring that the log helper's lifetime matches that of the thrift
request, we simplify adding to the logger. We wont have to think about the
lifetime of each of the members of the logger, we simply can stick them in the
log helper and let shared_ptr do its magic.
More specific why: This refactor helps make adding Thrift data fetch logging
cleaner (see following changes). We need to ensure the ObjectFetchContext
stays around for the duration of the thrift request since we pass it around
by reference. Sticking this in ThriftLogHelper avoids adding another disjoint
piece of plumbing and makes the code easier to maintain going forward.
Reviewed By: genevievehelsel
Differential Revision: D22632546
fbshipit-source-id: 1baa79419386947e52a386d89a65f032f1988622
Summary: If the user specifies a mount, only clear for that mount, otherwise clear for all mounts.
Reviewed By: fanzeyi
Differential Revision: D22674210
fbshipit-source-id: 832c4fd37a63fdb44cb12844378b61d113731016
Summary: add a thrift call to clear `pidFetchCount_` in `ObjectStore` and call it in `eden debug gc_process_fetch`. Users might want this command to start a new recording of process fetch counts.
Reviewed By: kmancini
Differential Revision: D22583430
fbshipit-source-id: eba7d63b08da5134fd09b7512895aba06f6a7ca5
