Summary:
The EdenFS codebase uses folly/logging/xlog to log, but we were still relying
on glog for the various CHECK macros. Since xlog also contains equivalent CHECK
macros, let's just rely on them instead.
This is mostly codemodded + arc lint + various fixes to get it compile.
Reviewed By: chadaustin
Differential Revision: D24871174
fbshipit-source-id: 4d2a691df235d6dbd0fbd8f7c19d5a956e86b31c
Summary:
There were `eden top` issues on MacOS that I thought had been fixed a while ago,
but it doesn't look like we caught them all. This should catch the remaining bug
in `eden top`.
Reviewed By: genevievehelsel
Differential Revision: D23743199
fbshipit-source-id: ca66748c7a8a26062caf934c8f2c1fd13d9ae69e
Summary:
In order to allow request to timeout to display notifications to the user, the
`within` future method will need to be called on the various callback futures.
Unfortunately, once the timeout expires, the underlying future isn't cancelled
and stopped, but the unique pointer holding the context will be reclaimed.
Whenever the future actually completes, it will try to use an invalid pointer,
crashing EdenFS.
To solve this, switch to using a shared_ptr and copy it in the right places so
it will only be freed once all futures holding a reference to it will be gone.
I also took the opportunity to reduce the nesting a bit to make the code more
readable.
Reviewed By: kmancini
Differential Revision: D24809647
fbshipit-source-id: 987d6e5763106fabc6bed3ea00d28b129b5285a1
Summary: These errors are Win32 errors, we need to wrap them into a HRESULT.
Reviewed By: chadaustin
Differential Revision: D24809646
fbshipit-source-id: 9f42b9d0c43474967dc26cb2c14cbee463768b79
Summary: force-unmount-all.sh is a convenience script for edenfs, so move it into eden/fs/.
Reviewed By: fanzeyi
Differential Revision: D24745361
fbshipit-source-id: 661a6f09b73911411fbb8a00bc016757ad19eb2a
Summary: This is unecessary, remove it.
Reviewed By: chadaustin
Differential Revision: D24743519
fbshipit-source-id: 5e10eafcd3f84d9ad053be35798df86b21f97d4f
Summary:
One of the issue that EdenFS on Windows is currently facing is around
invalidation during an update. In effect, EdenFS is over invalidating, which
causes update to be slower than it should be, as well as EdenFS recursively
triggering ProjectedFS callbacks during invalidation. Both of these are a
sub-par UX.
The reason this issue exist is multi-faceted. First, the update code follows
the "kPreciseInodeNumberMemory" path which enforces that a directory that is
present in the overlay needs to be invalidated, even if it isn't materialized.
The second reason is that no reclamation is done for the overlay, combine the
two and you get an update that gets both slower over time and will issue
significantly more invalidation that is needed.
Solving this is a bit involved. We could for instance start by reclaiming
inodes from the overlay, but this wouldn't be effective as we use the fact that
an inode is present in the overlay as a way to know that the file is cached in
the overlay. If we reclaim from the overlay we simply won't be invalidating
enough and some files will be out of date.
It turns out that we already have a mechanism to track what is cached by the
kernel: the fuse refcount. On Linux/macOS, everytime an inode is returned to
the kernel, this refcount incremented, and the kernel then notifies us when it
forgot about it, at which point the refcount can be decremented. On Windows,
the rules are a bit different, and a simple flag is sufficient: set when we
write a placeholder on disk (either during a directory listing, or when
ProjectedFS asks for it), and unset at invalidation time during update. There
is however a small snag in this plan. On Linux, the refcount starts at 0 when
EdenFS starts as a mount/unmount will clear all the kernel references on the
inodes. On Windows, the placeholder aren't disappearing when EdenFS dies or is
stopped, so we need a way to scan the working copy when EdenFS starts to know
which inodes should be loaded (an UnloadedInode really).
The astute reader will have noticed that this last part is effectively a
O(materialized) operation that needs to happen at startup, which would be
fairly expensive in itself. It turns out that we really don't have choice and
we need to do it regardless due to Windows not disallowing writes to the
working copy when EdenFS is stopped, and thus for EdenFS to be aware of the
actual state of the working copy, it needs to scan it at startup...
The first step in doing all of this is to simply rename the various places that
uses "fuse refcount" to "fs refcount" which is what this diff does.
Reviewed By: chadaustin
Differential Revision: D24716801
fbshipit-source-id: e9e6ccff14c454e9f2626fab23daeb3930554b1a
Summary:
Eden can often overfetch files during checkout. There may be multiple things
going on here, but at least one of them is tied to aux data prefetching.
Responding to a FUSE request for aux data will result in the inode for that
file being loaded. During checkout if we don't short circuit the checkout
then we will create a CheckoutAction for each loaded inode. And previously
CheckoutActions will always load their inodes contents.
To fix this we have two options
1. avoid creating the CheckoutAction in the first place.
2. Avoid downloading in CheckoutAction.
1 Turns out to be more difficult. I did some thinking through this here if
you want to see: https://fb.quip.com/LeqSAb3OlpWb
2 We don't actually need to be downloading blob data here, we only really
need the eden hash and content sha1 of the blob in this code path (this is
all that is used to compare the old and new file). So we should really only
be fetching the metadata (which should be avaiable locally since the inode
was loaded unless the caches have been cleared).
This implements fix 2.
I think there is likely still over fetching having to do with trees in checkout.
And potentially we can avoid looking at metadata at all in some cases.
I will look at this next.
Reviewed By: chadaustin
Differential Revision: D23432810
fbshipit-source-id: 17c0503bf95eb360de902a370948338bf04c1887
Summary:
On Windows, we never unload any inodes until EdenFS is restarted, thus,
checkout times go up over time as more and more inodes are being loaded. While
on Windows we don't keep track of what is referenced by the kernel, the
checkout code will use the "precise inodes" code path when deciding what to
update. This means that every inode that is in the overlay will get updated
properly, and since the overlay is a superset of what is hydrated, we are
guaranteed to always invalidate what we need to.
Due to the above, this shouldn't result in any changes as we never gc the
overlay, but that will come later, at which point checkout times will stop
being more and more expensive as time goes.
Reviewed By: chadaustin
Differential Revision: D24634253
fbshipit-source-id: c7b838edc20589bbf92ff4e2b3abd079b9a4443d
Summary:
Futures are by default run inline, meaning that when the previous future is
completed, the future will run in the same context as the previous one. In the
case where the previous one is completed by another thread setting up its
promise, the future will be completed in the context of that other thread.
In most cases, this is OK, in others, this can cause a deadlock. And this is
exactly what we're seeing here. When a file is renamed concurrently to an `hg
update`, the inode the rename operates on might not be loaded, and thus both
update and the rename callback will race to load that inode. When update wins
that race, the rename callback will wait on a promise that update will then
set. When that happens, the rest of the rename callback will be run in the
update context, but that will in turn cause update to try to re-acquire the
rename lock that it already holds...
To fix this, we need to make sure that the rename callback doesn't run inline.
Reviewed By: chadaustin
Differential Revision: D24657422
fbshipit-source-id: 23b08765afae7bda4a628f0c23675bff9f486b6b
Summary:
For logging and analytics, it's convenient for the
HgQueuedBackingStore to know the manifest hash and path early in the
import process. Since every object fetch requires looking up the
HgProxyHash anyway, fetch it immediately and thread it down to the
importer.
Reviewed By: kmancini
Differential Revision: D24524319
fbshipit-source-id: 0d91d55655e5ee25a010f7664e80125b7c50cf84
Summary:
Use the empty string to indicate the moved-from state, which makes
HgProxyHash moves noexcept. I plan to look up HgProxyHash early and
move it into HgImportRequest.
Reviewed By: kmancini
Differential Revision: D24522612
fbshipit-source-id: 037b4012ad6a51ad7ebd6a96de2e391cd570771b
Summary:
Stop pretending that HgBackingStore is a standalone backing store
implementation, and instead indicate it's an implementation class used
by HgQueuedBackingStore.
Reviewed By: kmancini
Differential Revision: D24514247
fbshipit-source-id: 90c3a442d01647fa6d1337cfd814f5bf4b480137
Summary:
We had some unnamed threads that made profiling performance on macOS a
bit harder. Give them a semblance of a useful name, at least.
Reviewed By: kmancini
Differential Revision: D24640223
fbshipit-source-id: 7dd74b30a081753006df681bf97ac96147b1896c
Summary:
Until fanzeyi gets InodeMetadata moved over to SQLite, remove some
excessive logging when the InodeMetadataTable contains zeroes because
its buffers weren't flushed to disk.
Reviewed By: genevievehelsel
Differential Revision: D24377026
fbshipit-source-id: e6ffa54244730388aaf66dc53cd29a0069fba685
Summary:
Add a very simple debug command that simply prints all materialized
inodes under the given path. It is a quick way to uncover unexpected
writes (or unexpected failed dematerializations after checkou).
Reviewed By: xavierd
Differential Revision: D24378759
fbshipit-source-id: dc393d65506c74dbc0779732cdefd915cbbf9948
Summary:
Replace the old implementation of debugInodeStatus with the more
general traverseObservedInodes functionality, and add the ability to
customize its results with flags.
Reviewed By: xavierd
Differential Revision: D24300122
fbshipit-source-id: 0fbd3aa02575faa515fd7852441547d7de13426d
Summary:
Apply automated Thrift formatter. Even if the format is a bit off in a
couple places, and the bikeshed is still being painted, this avoids
unrelated formatting changes later in the stack.
Reviewed By: xavierd
Differential Revision: D24511057
fbshipit-source-id: f1b23578733a8ecf788509e407bc419fa073428d
Summary: This method is an override of its parent class, thus the need for the override.
Reviewed By: chadaustin
Differential Revision: D24657424
fbshipit-source-id: 5ce7200eeb4ef28fb51fabbe0ebb38ded51ae34e
Summary:
Dealing with non-owned path in futures is mind bending and can lead to use
after free fairly easily if not careful. It turns out that this code is
actually a bit buggy and in some cases will attempt to use a path that is
already freed. Since the callsite doesn't need to hold onto the paths, let's
just move them, which resolves the issuue.
Reviewed By: chadaustin
Differential Revision: D24657423
fbshipit-source-id: 47bbaccf18cd86e53860491e3cbfeadb4363499c
Summary: When I added `eden uptime`, I did not see that we already had `eden debug uptime`! Instead of having two, lets just use the one and remove it from debug. There seems to be a regression of the uptime I had implemented, so thats why I opted to use the implementation of the debug version.
Reviewed By: kmancini
Differential Revision: D24566844
fbshipit-source-id: d948a5303d475a543f51abbaea59f9c481dfeca2
Summary:
We don't need an ifdef here as folly::kIsWindows works just as well. The astute
reader will notice that we now call isInodeRemembered while we weren't before.
On Windows, this will pretty much always return false as this can only return
true if the fuse refcount is positive. Since we don't keep track of a fuse
refcount on Windows, this is OK to do.
Reviewed By: genevievehelsel
Differential Revision: D24636432
fbshipit-source-id: c6db7c66f7eb27894cdd276a9368149ec8056cf4
Summary:
One of the unforseen effect of D24393690 (c5d631fd09) is that EdenFS was tied to the console
that ran `edenfsctl start`. As a result, killing that console would also kill
EdenFS, which is a bit unexpected.
Console in Windows are a bit complicated, and in theory, EdenFS could simply
call `FreeConsole` once it is started, but the side effect of this is that any
process started by EdenFS (say Mercurial), would create its own console as its
parent doesn't have one, specifying CREATE_NO_WINDOW when creating these would
lead to not having any output in the edenfs logs from Mercurial (D21820195 (223846d313))...
We could solve this if we could tell Windows to allocate a hidden console
window, but that API unfortunately doesn't exist.
So, the best we can do for now is to simply start EdenFS with the
CREATE_NO_WINDOW flag, this fixes the above issue, allows for the original
console to be killed without affecting EdenFS, but it brings another issue:
`eden start` no longer shows what is going on...
For now, this is the least worst solution, so we'll go with that. In the
future, we can imagine fixing EdenFS startup code to send its output to a pipe
that edenfsctl would read and print to its standard output.
Reviewed By: chadaustin
Differential Revision: D24607540
fbshipit-source-id: c18590052a96b7dd2938b589e92e808f38b5ef59
Summary:
We want to be able to prefetch profiles on pull. That means we will need
to pick commits that will be good to prefetch for. This adds a flag --commits
that allows specifiying commits explicitly (mostly for testing), and a
flag `--predict-commits` which will pick good commits to prefetch for.
Reviewed By: genevievehelsel
Differential Revision: D23889882
fbshipit-source-id: 74e61c0c9d443ca396f326b0d547b9fc21a6364b
Summary:
We want to be able to fetch prefetch profiles on pull. That means we will need
to prefetch the contents of prefetch profiles for commits that we are not
currently on. Thus globFiles (the thrift endpoint used for prefetch profiles
fetching) needs to be able to take commit hashes to match and fetch against.
Why fetch prefetch profiles on pull? This would get the prefetch started earlier so
the files are hopefully fetched by the time the user needs them.
Reviewed By: chadaustin, genevievehelsel
Differential Revision: D23858659
fbshipit-source-id: 123e423d5117274b92405dbb5c2df690298a1c18
Summary:
Often a prefetch profile will be activated by a tool that uses that profile, so
we will by default prefetch a profile when activating it.
Reviewed By: genevievehelsel
Differential Revision: D23772446
fbshipit-source-id: 0b2e14cfa6ce079e8f04790ac2e0d196db5966ee
Summary:
Blocking checkout and pull to prefetch files could make these commands very
slow when large prefetches are triggered. To keep slow prefetches from causing
poor experiences we can background them by default.
Reviewed By: genevievehelsel
Differential Revision: D23831663
fbshipit-source-id: 4fba5ab894b927a6aa3359b516cff604ad524485
Summary: Introduces an `eden logs` command to read a large chunk (1M by default) file into a paste. This is also added to the the eden rage report to get more insight into systems in which we cannot log in to view the logs (like laptops).
Reviewed By: kmancini
Differential Revision: D24146812
fbshipit-source-id: 991f1595b974eb01f77e86559a8413b0b09a24a4
Summary:
The LookupProcessor class is built with the purpose of iterating all the inodes
in the passed in path. However, the LookupProcessor object may outlive the
lifetime of the path, and thus we need to build an iterator on the copied path,
not on the argument.
Differential Revision: D24581874
fbshipit-source-id: b66dc007920b7adad5272bf56d3034acb211fec6
Summary:
When built with buck, cmake isn't available, switch to a method of echoing that
should be available on every system: powershell.
Reviewed By: genevievehelsel
Differential Revision: D24493404
fbshipit-source-id: 5986b846a211be59f1057b1e7ecd924f3f7380a5
Summary:
We've had a handful of reports of access denied errors seen during an `hg
update`, ignore them due to:
```
// TODO(T78476916): The access denied are coming from
// PrjMarkDirectoryAsPlaceholder recursively calling into EdenFS, which
// is denied by the BAIL_ON_RECURSIVE_CALL macro.
//
// In theory this means that EdenFS is invalidating a directory that
// isn't materialized, ie: doing useless work. Despite having a negative
// performance impact, this doesn't affect correctness, so ignore for now.
//
// A long term fix will need to not issue invalidation on directories
// that aren't materialized.
```
Reviewed By: wez
Differential Revision: D24544622
fbshipit-source-id: 4f421836b6d8c7b527e860dcf55103e0e239e3ae
Summary: Fix some build warnings that show up on macOS.
Reviewed By: wez
Differential Revision: D24436572
fbshipit-source-id: bd300110df79fe9d259c8f5ce2b1085cb85a0cac
Summary:
Sometimes you only want to trace writes into the EdenFS checkout, so
add the ability to run `eden strace` with `--reads`, `--writes`, or
both in order to see only those events.
Reviewed By: wez
Differential Revision: D24468539
fbshipit-source-id: a1b3c730987cf86ce3d39952c6a5e9c5edaddfc2
Summary:
Before replacing the existing debugInodeStatus changes, make some
small refactorings to help me understand it.
Reviewed By: xavierd
Differential Revision: D24299970
fbshipit-source-id: 3b60aa8365bb9a45e12357374282ba837ccc36eb
Summary:
Add a function that makes it easy to traverse EdenFS's view of the
traversed-so-far filesystem.
Reviewed By: xavierd
Differential Revision: D24299962
fbshipit-source-id: 275c849846bf45a2e80780411d60266b961d825b
Summary:
Now that Linux/macOS startup no longer uses fork it becomes trivial to share
the same code with Windows. This improves Windows startup in several different ways:
- `edenfsctl start` now displays the status of the start process in the console
- `edenfsctl start` no longer has an arbitrary timeout after which it reports
a timeout even though EdenFS is still starting up.
This also kills a bunch of Windows specific code that is no longer needed.
Reviewed By: fanzeyi
Differential Revision: D24393690
fbshipit-source-id: 28100aec96da81c92d5b592353edceed332e2364
Summary:
Previously, when that code was ported on Windows, paths separator were
converted from '\' to '/' when a wide string was provided, all the other paths
were treated as is.
The main issue with this strategy is that not all paths can be converted, the
non-stored ones for instance are immutable, which leads to some subtle bugs
down the line. For instance, the paths: "Z:/foo/bar/baz" and "Z:\foo/bar\baz"
would not be equal as the path separator isn't the same, but both of these are
actually the same path underneath.
To solve this, this diff first introduce a Windows path separator, and then
modifies the path comparison functions to ignore the path separator and only
compare the components.
I'm definitively not a fan of the pattern I use for searching for both / and \
in paths, suggestions are welcome for how to improve that.
Reviewed By: chadaustin
Differential Revision: D24376980
fbshipit-source-id: 0702bf775c7c3937b2138abd5a63d339ac80aaed
Summary:
The path iterator functions are skipping sanity checking of paths as they
assume that the path has already been validated. Unfortunately, that isn't the
case as the only sanity checking we are doing is on the beginning and end of
the string that is passed in. Notably, `RelativePathPiece{"./foo"}` would not
fail the sanity checking even though it it's not made of valid components.
Reviewed By: chadaustin
Differential Revision: D24439804
fbshipit-source-id: 852b3a1180b185cb0bfb96bf5bcdc98b231f32c5
Summary:
As the string is created with MAX_PATH wide characters, a lot of them are going
to stay nul. A future diff adds a sanity check to the PathComponent that
validates that it doesn't contain any nul bytes, which chokes on these paths.
Reviewed By: chadaustin
Differential Revision: D24479715
fbshipit-source-id: e51c4b1c53a3f375664c0c0a1e325ebe302b0cd3
Summary: Comment out the tests that do not compile on Windows.
Reviewed By: fanzeyi
Differential Revision: D24458438
fbshipit-source-id: 14bd4803ccd68ea6e02f95f868b8a801161518ea
Summary: This merely filters out the files that don't compile on mode/win.
Reviewed By: chadaustin
Differential Revision: D24456917
fbshipit-source-id: d314ca2936e064133f2fc6a6199b38d17f2d9ef7
Summary:
FUSE_ACCESS was incorrectly categorized as an 'other' FUSE request
type. Categorize it as 'read'.
Reviewed By: wez
Differential Revision: D24437458
fbshipit-source-id: 5ea0a7d5a0fd57b9230f37674fd2568a4917bab5
Summary:
I noticed while playing with `eden strace` that, on my Linux machine,
running `ls -lA` would cause a pile of FUSE_GETXATTR requests for
`system.posix_acl_access` and `system.posix_acl_default`.
FUSE does not cache getxattr yet, but the kernel can cache ACLs if we
pretend to support them. Thus, advertise FUSE_POSIX_ACL support, but
continue returning ENOSYS for all setxattr calls.
Reviewed By: wez
Differential Revision: D24039130
fbshipit-source-id: a2e69c43b728fb51fb1d1b41ca9d31eba8efaa19
Summary:
Add an `eden strace` command that subscribes to all FUSE events for
the specified mount (or current directory), and streams them to stdout
with response latency measurements and the requesting process.
If there are any pending requests at the time `eden strace` is run,
they are printed first.
Reviewed By: wez
Differential Revision: D24038978
fbshipit-source-id: 59a2112dbdb1708571d12b04bdccaf9eca725cf7
Summary:
Buck-out can contain some files that we wont have the ability to read on mac os
(like .Spotlight-V100 and .Trashes). This means `eden du` will error out on
reading them.
We handle FileNotFound but not permission errors, lets treat them the same.
Reviewed By: genevievehelsel
Differential Revision: D24396252
fbshipit-source-id: 31e2ecfeff1783ae9c3479f59335d95a0d9b6de7
Summary:
The communication protocol between edenfs and watchman is that edenfs
advertises that "something in the mount has changed!" and it's up to
watchman to call getFilesChangedSince or getCurrentJournalPosition to
inspect what has changed.
This diff avoids redundantly notifying "something in the mount has
changed!" between the initial notification and actually reading the
journal contents. This way, a Watchman subscription cannot (*) fall
behind, and memory usage stays approximately constant during heavy
write traffic.
Hopefully, this will prevent the kernel from thinking the edenfs mount
is a slow IO device under high memory pressure.
* Technically, if there were a multitude of subscribeStreamTemporary
streams, and all but one was not calling getFilesChangedSince, it's
possible that notifications could back up on the stuck
subscriptions.
Reviewed By: wez
Differential Revision: D24090247
fbshipit-source-id: 6561c13e847b749c093adab75250df474d3210f9
Summary:
Include opcode name and process name in eden.FuseCall Thrift structs
so we can use eden.FuseCall in a later diff in the stack.
Reviewed By: kmancini
Differential Revision: D24036420
fbshipit-source-id: fc6d8f3d174b85e07fac299a6f86b2b2d24f301d
Summary:
Provide an API that allows a FuseChannel TraceBus subscriber to
request population of rendered argument strings. This is not on by
default to avoid the cost in the common case that there are no
subscribers that use the field.
Reviewed By: kmancini
Differential Revision: D24036018
fbshipit-source-id: ff12252de3777606360efc75814149931637aa9b
