Summary:
`eden du --clean` currently fails with
```
Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'
```
Full error: P149352812
It looks like this is because Buck expects to run with a different python, so
here I clear out the PYTHONHOME variable before we start buck.
This reuses out logic used elsewhere to clean up the environment before
calling buck.
Reviewed By: wez
Differential Revision: D24904105
fbshipit-source-id: 73587c52aff3ea00f68339eb44e7042329de2e44
Summary:
It feels like invalidating the entry before the directory makes slightly more
sense, so do it in that order.
Reviewed By: chadaustin
Differential Revision: D24800817
fbshipit-source-id: ed053d07bbae6954c276d1ad7a1ff247e5c055d9
Summary:
Add a TraceBus to HgQueuedBackingStore and allow tracing import events over Thrift.
This powers a new `eden trace hg` command that allows a live view of
tree and blob imports.
Reviewed By: genevievehelsel
Differential Revision: D24732059
fbshipit-source-id: 525152fe39047160a68c1706217a06a00a6dbef1
Summary: The test doesn't compile on Windows, let's just ifdef it.
Reviewed By: genevievehelsel
Differential Revision: D25033804
fbshipit-source-id: 4f312f010f9d0db42cc9ae19df3f668e8e1c4665
Summary:
Converting back and forth between folly::fs::path and AbsolutePath appears to
be problematic on Windows as NUL bytes appears in the paths, causing the tests
to fail. Instead of doing this conversion, let's simply use AbsolutePath everywhere.
Reviewed By: chadaustin
Differential Revision: D25033803
fbshipit-source-id: 6c45c2a20fc4bf18cecc838b219faacfeb8386d8
Summary:
We used to produce a confusing error message during glob evaluation
when . or .. was specified as a glob component. Instead, fail early,
with an error message that more directly explains the problem.
Reviewed By: genevievehelsel
Differential Revision: D24969096
fbshipit-source-id: fe70a8f4db1fdce8eec13890d20913b63a516518
Summary:
Be more specific about which PathComponent string failed to validate
in order to help diagnose downstream issues like glob syntax errors.
Reviewed By: genevievehelsel
Differential Revision: D24966004
fbshipit-source-id: cd3bc0aeaeb389caa13c86b91149d48c5afdb306
Summary: The test wasn't compiling due to these 3 missing headers, add them.
Reviewed By: genevievehelsel
Differential Revision: D24997597
fbshipit-source-id: 6e3be0763362e41be138c670dc88a63bd9e88024
Summary:
This is a preparatory phase to make the refcount usuable on Windows. For more
details, see D24716801 (e50725d2cb)
Reviewed By: chadaustin
Differential Revision: D24764568
fbshipit-source-id: 1e8c6ab00d4c1ec79c347fd5ae7167b2ce1dff68
Summary:
The skip_on_mode_mac argument to cpp_unittest doesn't exist on Windows, and to
be consistent with the rest of the code, we can simply ifdef the code that
either doesn't compile, or doesn't run.
Note: For some reason the accessIncrementsAccessCount test fails when running
on Windows, but only if running after accessAddsProcessToProcessNameCache...
Reviewed By: chadaustin
Differential Revision: D24496450
fbshipit-source-id: fe18fe1d791a27fbe4bd03bd3e8c811feeb23f5f
Summary:
This is what `eden debug` shows:
modified (m, a, t, e, r, i, a, l, i, z, e, d)
Enumerate all potentially-modified inode paths
Reviewed By: chadaustin
Differential Revision: D24908570
fbshipit-source-id: 62e91b6f0212c70de4bb1705539d3674e6e000d9
Summary:
On Windows, `eden debug file_stats` would crash due to the fileSize not being
filled. Since computing the file size is just a matter of calling stat(2) on
the file, let's simply do that.
Reviewed By: chadaustin
Differential Revision: D24908430
fbshipit-source-id: 07ffb97ada15a07565bea397b436fd21d09b5565
Summary: This is unused, except in tests, so let's just remove it.
Reviewed By: chadaustin
Differential Revision: D24930011
fbshipit-source-id: cb132962e1dff9d12ce12e7eb75bd34a026c58b7
Summary:
As of right now, opendir is the expensive callbacks due to fetching the sizes
for all the files in a directory. This strategy however breaks down when
timeouts are added as very large directories would trigger the timeout, not
because EdenFS is having a hard time reaching Mononoke, but because of
bandwidth limitation.
To avoid this issue, we need to have a per-file timeout and thus makes opendir
just trigger the futures, but not wait on them. The waiting bit will happen
at readdir time which will also enable having a timeout per file.
The one drawback to this is that the ObjectFetchContext that is passed in by
opendir cannot live long enough to be used in the size future. For now, we can
use a null context, but a proper context will need to be passed in, in the
future.
Reviewed By: wez
Differential Revision: D24895089
fbshipit-source-id: e10ceae2f7c49b4a006b15a34f85d06a2863ae3a
Summary:
The rest of the non debug commands use - instead of _.
Lets flip the prefretch profiles cli to be consistant.
Reviewed By: genevievehelsel
Differential Revision: D24910172
fbshipit-source-id: a5f18a9c9d5fb4ef9417f14ef9d053cdc1599d76
Summary:
The second line of the summary got cut off due to a bad multiline comment
signals_oops
Reviewed By: wez
Differential Revision: D24909886
fbshipit-source-id: 844778e7d47a2b7b413fdd0c4fa0ef71dec9dadb
Summary:
`eden prefetch` can trigger a bunch of wasted network requests to fetch metadata
when we are fetching files anyways. (These network requests are wasted since we
fetch the file contents and most of them are being throttled on sandcastle anyways.)
So lets skip metadata prefetching during eden prefetches.
Reviewed By: genevievehelsel
Differential Revision: D24658066
fbshipit-source-id: f8a32807a4e238222158f100cdd5ffa1b92fd833
Summary:
osxfuse is rebranding as macfuse in 4.x.
That has ripple effects through how the filesystem is mounted and shows up in
the system.
This commit adjusts for the new opaque and undocumented mount procedure and
speculatively updates a couple of other code locations that were sensitive to
looking for "osxfuse" when examining filesystems.
Reviewed By: genevievehelsel
Differential Revision: D24769826
fbshipit-source-id: dab81256a31702587b0683079806558e891bd1d2
Summary:
It would be nice to see if there was a fsck on startup, the duration of the fsck, and if it was able to repair all of the problems or not. This diff adds external logging for fsck runs on daemon start.
duration: how long the fsck took
success: false if was not able to repair errors, true if repaired all errors or didn't have to repair at all
attempted_repair: true if we found problems, false otherwise
Reviewed By: chadaustin
Differential Revision: D24774065
fbshipit-source-id: 2fa911652abec889299c74aaa2d53718ed3b4f92
Summary:
The XLOG_EVERY_MS doesn't use the 3rd argument as a format string, it just
prints it verbatim. To format it, we need to use fmt::format.
Reviewed By: genevievehelsel
Differential Revision: D24906819
fbshipit-source-id: 7d45787301086fb87dd8f5d478af8007df82c0b6
Summary:
The move constructor needs to be noexcept and should also initialize the
members in the right order.
Reviewed By: genevievehelsel
Differential Revision: D24874304
fbshipit-source-id: a3db5dcdab1397b872b8f13ec5c7fd45baad5e6f
Summary:
The components iterator return pieces of the original path, using a reference
makes little sense and the compiler complains.
Reviewed By: genevievehelsel
Differential Revision: D24873851
fbshipit-source-id: 40d414dcb4a0539167ab4760dfc0095af8245b3a
Summary:
The documentation for PrjFillDirEntryBuffer states that if no entries could be
added, then the ERROR_INSUFFICIENT_BUFFER errors need to be returned as is, the
code didn't do that.
Reviewed By: chadaustin
Differential Revision: D24764566
fbshipit-source-id: d6411822eac71b2f9aa7cf07858d09115767cc59
Summary:
This is the plumbing to allow us to skip Metadata prefetching during eden
prefetches. These can trigger a bunch of wasted network requests
when we are fetching files anyways. (These network requests are wasted since we
fetch the file contents and most of them are being throttled on sandcastle anyways.)
We won't necessarily want to skip metadata prefetching always, we will still want it
for the watchman queries, but for `eden prefetch` will probably want to skip it. This
is why we are making it an option in the GlobParams.
Reviewed By: chadaustin
Differential Revision: D24640754
fbshipit-source-id: 20db62d4c0e59fe17cb6535c86ac8f1e3877879c
Summary:
We will start opting-in and rolling prefetch profiles mvp out to users soon.
This is a switch to allow users to opt-in, us to gradually rollout, and to
quickly turn prefetch profiles off if this causes issues for users.
Reviewed By: genevievehelsel
Differential Revision: D24803728
fbshipit-source-id: 0456f2a733958b495e5d84f7177c99f3ef481f57
Summary:
On Windows, /bin/sh doesn't exist. To spawn a command in a shell, we need to
use Powershell.
Reviewed By: genevievehelsel
Differential Revision: D24864355
fbshipit-source-id: 3bcf630a90e644a31ff9db8fea9891476cad641d
Summary:
While doing notifications, I struggled a bit to get them working and thought
the special quoting on Windows didn't work as expected. It turns out the error
was cmd related and using a modern shell (PowerShell) fixed it.
Having a test for the quoting is a good idea nonetheless, so let's have one.
Reviewed By: genevievehelsel
Differential Revision: D24864357
fbshipit-source-id: 6b1ac50f3b7b1ef469378d5de21f56c24c0945f9
Summary:
The EdenFS codebase uses folly/logging/xlog to log, but we were still relying
on glog for the various CHECK macros. Since xlog also contains equivalent CHECK
macros, let's just rely on them instead.
This is mostly codemodded + arc lint + various fixes to get it compile.
Reviewed By: chadaustin
Differential Revision: D24871174
fbshipit-source-id: 4d2a691df235d6dbd0fbd8f7c19d5a956e86b31c
Summary:
There were `eden top` issues on MacOS that I thought had been fixed a while ago,
but it doesn't look like we caught them all. This should catch the remaining bug
in `eden top`.
Reviewed By: genevievehelsel
Differential Revision: D23743199
fbshipit-source-id: ca66748c7a8a26062caf934c8f2c1fd13d9ae69e
Summary:
In order to allow request to timeout to display notifications to the user, the
`within` future method will need to be called on the various callback futures.
Unfortunately, once the timeout expires, the underlying future isn't cancelled
and stopped, but the unique pointer holding the context will be reclaimed.
Whenever the future actually completes, it will try to use an invalid pointer,
crashing EdenFS.
To solve this, switch to using a shared_ptr and copy it in the right places so
it will only be freed once all futures holding a reference to it will be gone.
I also took the opportunity to reduce the nesting a bit to make the code more
readable.
Reviewed By: kmancini
Differential Revision: D24809647
fbshipit-source-id: 987d6e5763106fabc6bed3ea00d28b129b5285a1
Summary: These errors are Win32 errors, we need to wrap them into a HRESULT.
Reviewed By: chadaustin
Differential Revision: D24809646
fbshipit-source-id: 9f42b9d0c43474967dc26cb2c14cbee463768b79
Summary: force-unmount-all.sh is a convenience script for edenfs, so move it into eden/fs/.
Reviewed By: fanzeyi
Differential Revision: D24745361
fbshipit-source-id: 661a6f09b73911411fbb8a00bc016757ad19eb2a
Summary: This is unecessary, remove it.
Reviewed By: chadaustin
Differential Revision: D24743519
fbshipit-source-id: 5e10eafcd3f84d9ad053be35798df86b21f97d4f
Summary:
One of the issue that EdenFS on Windows is currently facing is around
invalidation during an update. In effect, EdenFS is over invalidating, which
causes update to be slower than it should be, as well as EdenFS recursively
triggering ProjectedFS callbacks during invalidation. Both of these are a
sub-par UX.
The reason this issue exist is multi-faceted. First, the update code follows
the "kPreciseInodeNumberMemory" path which enforces that a directory that is
present in the overlay needs to be invalidated, even if it isn't materialized.
The second reason is that no reclamation is done for the overlay, combine the
two and you get an update that gets both slower over time and will issue
significantly more invalidation that is needed.
Solving this is a bit involved. We could for instance start by reclaiming
inodes from the overlay, but this wouldn't be effective as we use the fact that
an inode is present in the overlay as a way to know that the file is cached in
the overlay. If we reclaim from the overlay we simply won't be invalidating
enough and some files will be out of date.
It turns out that we already have a mechanism to track what is cached by the
kernel: the fuse refcount. On Linux/macOS, everytime an inode is returned to
the kernel, this refcount incremented, and the kernel then notifies us when it
forgot about it, at which point the refcount can be decremented. On Windows,
the rules are a bit different, and a simple flag is sufficient: set when we
write a placeholder on disk (either during a directory listing, or when
ProjectedFS asks for it), and unset at invalidation time during update. There
is however a small snag in this plan. On Linux, the refcount starts at 0 when
EdenFS starts as a mount/unmount will clear all the kernel references on the
inodes. On Windows, the placeholder aren't disappearing when EdenFS dies or is
stopped, so we need a way to scan the working copy when EdenFS starts to know
which inodes should be loaded (an UnloadedInode really).
The astute reader will have noticed that this last part is effectively a
O(materialized) operation that needs to happen at startup, which would be
fairly expensive in itself. It turns out that we really don't have choice and
we need to do it regardless due to Windows not disallowing writes to the
working copy when EdenFS is stopped, and thus for EdenFS to be aware of the
actual state of the working copy, it needs to scan it at startup...
The first step in doing all of this is to simply rename the various places that
uses "fuse refcount" to "fs refcount" which is what this diff does.
Reviewed By: chadaustin
Differential Revision: D24716801
fbshipit-source-id: e9e6ccff14c454e9f2626fab23daeb3930554b1a
Summary:
Eden can often overfetch files during checkout. There may be multiple things
going on here, but at least one of them is tied to aux data prefetching.
Responding to a FUSE request for aux data will result in the inode for that
file being loaded. During checkout if we don't short circuit the checkout
then we will create a CheckoutAction for each loaded inode. And previously
CheckoutActions will always load their inodes contents.
To fix this we have two options
1. avoid creating the CheckoutAction in the first place.
2. Avoid downloading in CheckoutAction.
1 Turns out to be more difficult. I did some thinking through this here if
you want to see: https://fb.quip.com/LeqSAb3OlpWb
2 We don't actually need to be downloading blob data here, we only really
need the eden hash and content sha1 of the blob in this code path (this is
all that is used to compare the old and new file). So we should really only
be fetching the metadata (which should be avaiable locally since the inode
was loaded unless the caches have been cleared).
This implements fix 2.
I think there is likely still over fetching having to do with trees in checkout.
And potentially we can avoid looking at metadata at all in some cases.
I will look at this next.
Reviewed By: chadaustin
Differential Revision: D23432810
fbshipit-source-id: 17c0503bf95eb360de902a370948338bf04c1887
Summary:
On Windows, we never unload any inodes until EdenFS is restarted, thus,
checkout times go up over time as more and more inodes are being loaded. While
on Windows we don't keep track of what is referenced by the kernel, the
checkout code will use the "precise inodes" code path when deciding what to
update. This means that every inode that is in the overlay will get updated
properly, and since the overlay is a superset of what is hydrated, we are
guaranteed to always invalidate what we need to.
Due to the above, this shouldn't result in any changes as we never gc the
overlay, but that will come later, at which point checkout times will stop
being more and more expensive as time goes.
Reviewed By: chadaustin
Differential Revision: D24634253
fbshipit-source-id: c7b838edc20589bbf92ff4e2b3abd079b9a4443d
Summary:
Futures are by default run inline, meaning that when the previous future is
completed, the future will run in the same context as the previous one. In the
case where the previous one is completed by another thread setting up its
promise, the future will be completed in the context of that other thread.
In most cases, this is OK, in others, this can cause a deadlock. And this is
exactly what we're seeing here. When a file is renamed concurrently to an `hg
update`, the inode the rename operates on might not be loaded, and thus both
update and the rename callback will race to load that inode. When update wins
that race, the rename callback will wait on a promise that update will then
set. When that happens, the rest of the rename callback will be run in the
update context, but that will in turn cause update to try to re-acquire the
rename lock that it already holds...
To fix this, we need to make sure that the rename callback doesn't run inline.
Reviewed By: chadaustin
Differential Revision: D24657422
fbshipit-source-id: 23b08765afae7bda4a628f0c23675bff9f486b6b
Summary:
For logging and analytics, it's convenient for the
HgQueuedBackingStore to know the manifest hash and path early in the
import process. Since every object fetch requires looking up the
HgProxyHash anyway, fetch it immediately and thread it down to the
importer.
Reviewed By: kmancini
Differential Revision: D24524319
fbshipit-source-id: 0d91d55655e5ee25a010f7664e80125b7c50cf84
Summary:
Use the empty string to indicate the moved-from state, which makes
HgProxyHash moves noexcept. I plan to look up HgProxyHash early and
move it into HgImportRequest.
Reviewed By: kmancini
Differential Revision: D24522612
fbshipit-source-id: 037b4012ad6a51ad7ebd6a96de2e391cd570771b
Summary:
Stop pretending that HgBackingStore is a standalone backing store
implementation, and instead indicate it's an implementation class used
by HgQueuedBackingStore.
Reviewed By: kmancini
Differential Revision: D24514247
fbshipit-source-id: 90c3a442d01647fa6d1337cfd814f5bf4b480137
Summary:
We had some unnamed threads that made profiling performance on macOS a
bit harder. Give them a semblance of a useful name, at least.
Reviewed By: kmancini
Differential Revision: D24640223
fbshipit-source-id: 7dd74b30a081753006df681bf97ac96147b1896c
Summary:
Until fanzeyi gets InodeMetadata moved over to SQLite, remove some
excessive logging when the InodeMetadataTable contains zeroes because
its buffers weren't flushed to disk.
Reviewed By: genevievehelsel
Differential Revision: D24377026
fbshipit-source-id: e6ffa54244730388aaf66dc53cd29a0069fba685
Summary:
Add a very simple debug command that simply prints all materialized
inodes under the given path. It is a quick way to uncover unexpected
writes (or unexpected failed dematerializations after checkou).
Reviewed By: xavierd
Differential Revision: D24378759
fbshipit-source-id: dc393d65506c74dbc0779732cdefd915cbbf9948
Summary:
Replace the old implementation of debugInodeStatus with the more
general traverseObservedInodes functionality, and add the ability to
customize its results with flags.
Reviewed By: xavierd
Differential Revision: D24300122
fbshipit-source-id: 0fbd3aa02575faa515fd7852441547d7de13426d
Summary:
Apply automated Thrift formatter. Even if the format is a bit off in a
couple places, and the bikeshed is still being painted, this avoids
unrelated formatting changes later in the stack.
Reviewed By: xavierd
Differential Revision: D24511057
fbshipit-source-id: f1b23578733a8ecf788509e407bc419fa073428d
Summary: This method is an override of its parent class, thus the need for the override.
Reviewed By: chadaustin
Differential Revision: D24657424
fbshipit-source-id: 5ce7200eeb4ef28fb51fabbe0ebb38ded51ae34e
Summary:
Dealing with non-owned path in futures is mind bending and can lead to use
after free fairly easily if not careful. It turns out that this code is
actually a bit buggy and in some cases will attempt to use a path that is
already freed. Since the callsite doesn't need to hold onto the paths, let's
just move them, which resolves the issuue.
Reviewed By: chadaustin
Differential Revision: D24657423
fbshipit-source-id: 47bbaccf18cd86e53860491e3cbfeadb4363499c
