Summary:
When matching against the ** pattern, EdenFS would match against the full path
instead of against just the current entry. This means that what was matched by
the glob prior to ** will be rechecked against by the pattern that follows **,
which isn't what is expected.
As a bonus, the `candidateName` variable will be constructed less than before
leading to less memory allocations.
Reviewed By: chadaustin
Differential Revision: D29079762
fbshipit-source-id: af15ecf229ce7119100dd375df23269bb7cdb1c0
Summary:
When using `eden glob` at the root of the repository, the `searchRoot` is
initialized with ".", which is an invalid PathComponent. Let's special case
this in edenfs itself as python's Path appear to always want to be "." and thus
it's easier to fix in EdenFS.
Reviewed By: chadaustin
Differential Revision: D29072478
fbshipit-source-id: 0108294e407784832cd35b3dd6aaceacf4bd7dd9
Summary:
These tests rely on the wall clock and fail somewhat regularly on CI
and on my local machine, so remove them.
Reviewed By: fanzeyi
Differential Revision: D29047847
fbshipit-source-id: 4008d079fac55f52827be534ab1b18b93c1d2f2f
Summary:
Whenever a file is modified in an EdenFS mount and a watchman subscription is
active, watchman will be nodified and will issue a getFilesChangedSince Thrift
call. In order to do that, Watchman ends up always re-creating a new connection
to EdenFS, causing the .eden/socket or .eden/config to be re-read in order to
find EdenFS's socket.
For workloads with heavy write traffic to EdenFS, this readlink/read can add
up. On Windows, writing ~2.5GB worth of data lead Watchman to read over 650MB
worth of data from the .eden/config!
Reviewed By: chadaustin
Differential Revision: D28912258
fbshipit-source-id: 549d57592672ef9cbdcccc213d0612d019677923
Summary:
## This diff
* Implement `getCauseDetail` for Fuse fetch context so that we log which Fuse call triggers the fetch.
Reviewed By: chadaustin, kmancini
Differential Revision: D29092690
fbshipit-source-id: 230ef193332a90a01ca9b89f5b8d8f20cdf4f321
Summary: The new name is `LockedPtr::as_lock`.
Reviewed By: aary
Differential Revision: D28987868
fbshipit-source-id: 8abd6a69a1b9c884adf137f06c24fe0df9ddd089
Summary: Buck has not relied on the .buckversion file for a while now. Trying to clean up the number of configs at the root of the cell for buck. This diff attempts to remove .buckversion code referecnes. Instead of calling `cat .buckversion` to get the buckversion hash, you can call `buck --fast-version` which parses the `buck-java11` file without downloading buck. Alternatively, you can also do something like ` cat .buck-java11 | grep -o -E -e "[0-9a-f]{40}" | head -1` to get the buckversion hash.
Reviewed By: DrMarcII
Differential Revision: D28553844
fbshipit-source-id: ab90fa6a5e7467f6d5c11ffa9d0e10a03433975f
Summary:
Using a PathMap to hold the list of mounts gives us case sensitivity for free
without needing to have #ifdef _WIN32, we can also avoid copying the paths just
to normalize them.
Reviewed By: fanzeyi
Differential Revision: D28969795
fbshipit-source-id: 859cdc3881e36db120913099ce3f451940fa48a4
Summary:
We're planning on rolling out NFS to our users soon, let's make sure NFS is
enabled.
Reviewed By: singhsrb
Differential Revision: D29002395
fbshipit-source-id: 8351fa45c9bd5e68e162baa80e20bd34564b5ece
Summary:
The InodeMap is fully functional on Windows, there is no need to ifdef out its
counters.
Reviewed By: genevievehelsel
Differential Revision: D28970898
fbshipit-source-id: adeb2db19c70d5ff4a3a162fe34c32195fc2a1e0
Summary:
On Windows, the working copy doesn't go away on unmount, instead placeholders
and full files[0] are still present on disk. For this reason, EdenFS needs to
either overly invalidate files and directories at update times, or need to
remember what is present on disk so the state can be recovered.
For now, this diff simply focus on reloading all the inodes that are present on
disk in the InodeMap.
[0]: https://docs.microsoft.com/en-us/windows/win32/projfs/cache-state
Reviewed By: chadaustin
Differential Revision: D28889082
fbshipit-source-id: 90170c1291da563bea455c8032dc8282a093c9b3
Summary: This will implement getTree API of RE-CAS
Reviewed By: chadaustin
Differential Revision: D28157535
fbshipit-source-id: 6b09338f0635aa948497d934a8351a9ba362f1ab
Summary: As we need to debugging a IO related performance issue. We want to see if writes in overlay has caused any issues. This diff adds an option to allow us to create in-memory overlays.
Reviewed By: chadaustin
Differential Revision: D28951905
fbshipit-source-id: 9ef1da183771ee69b855b49bbabc2ef02a7ec9ac
Summary:
Like it says in the title. Right now this dependency is invisible to Buck so
we rely on reading it from the checkout. However, while this works on Buck 1
(I guess we run the tests from the repo root), it doesn't work Buck 2, and it
won't work if we try to run those tests remotely.
Let's make the dependency explicit!
Reviewed By: xavierd
Differential Revision: D28936482
fbshipit-source-id: 71c4a492c33f586abe6b721b90898f5424c06e94
Summary:
Backing stores differentiate between individual tree objects and the
root of a checkout. For example, Git and Mercurial roots are commit
hashes. Allow EdenFS to track variable-width roots to better support
arbitrary backing stores.
Reviewed By: genevievehelsel
Differential Revision: D28619584
fbshipit-source-id: d94f1ecd21a0c416c1b4933341c70deabf386496
Summary:
Making it the default should make cloning using EdenFS easier and will remove
the need to manually pass the `--nfs` command line to `eden clone`.
Reviewed By: kmancini
Differential Revision: D28913818
fbshipit-source-id: 742c35c950cb5edf34fd9769fb78bf26095af0d5
Summary:
WAL is known to be significantly faster than the default DELETE journaling
mode, let's enable it.
Reviewed By: fanzeyi
Differential Revision: D28915718
fbshipit-source-id: 3ee10d10dbaf07a01a23101a6266874a41784cfc
Summary:
These appear to be unecessary, thus there is no need to keep them in place as
it makes the code harder to follow.
Reviewed By: chadaustin
Differential Revision: D28850241
fbshipit-source-id: 129b853e88f0291bb637c9d24de4872a222a5af4
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
A few final write calls need to be threaded as well.
Reviewed By: xavierd
Differential Revision: D28844011
fbshipit-source-id: 214853eea7fa65263fe6415e1ae8b76ca21512d6
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
Some thrift methods used by watchman to get metadata implicitly are not threaded.
These can cause fetches, so let's thread the fetch context here too.
Reviewed By: genevievehelsel
Differential Revision: D28842300
fbshipit-source-id: b1e4b3aea879d6ed7b92afa26184616dedad5935
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
This change is for symlink
Reviewed By: genevievehelsel
Differential Revision: D28841453
fbshipit-source-id: 080eb62f0b562f8e0995c34e9a8302238fc59ed8
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
Most of rmdir is already threaded, not sure if this case can actuall cause
fetches in production, but might as well thread.
Reviewed By: genevievehelsel
Differential Revision: D28840211
fbshipit-source-id: 8dea08e775be470dd1730e2d32750a6912650ee0
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
this change is for rename
Reviewed By: genevievehelsel
Differential Revision: D23467437
fbshipit-source-id: e9d79c65fb5c4d686f0597550e43a0e87c4792cb
Summary:
we now want macFUSE to be installed instead of osxfuse. eden doctor will spew
errors if osxfuse is not installed or loaded, so it perpetually complains.
Update these checks for macFUSE now.
Reviewed By: xavierd
Differential Revision: D28751766
fbshipit-source-id: 4bc61349e33492aebe888a4e869ef7620c74768e
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
This change is for mkdir
Reviewed By: genevievehelsel
Differential Revision: D23458622
fbshipit-source-id: f3914a4f692490434882143664a5d5f1701e93ba
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
This change is for create
Reviewed By: genevievehelsel
Differential Revision: D23457862
fbshipit-source-id: d4c9cc658c26b3119b2b2a1da061e299eaf510c9
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
Most of lookup is already threaded. This finishes the threading for lookup.
Reviewed By: xavierd
Differential Revision: D23456910
fbshipit-source-id: fab7397caeee19f921d8fba1fb6528baa5cf2960
Summary:
BackingStore and LocalStore are no longer tied at the hip, so decouple
FakeBackingStore from LocalStore.
Reviewed By: kmancini
Differential Revision: D28615431
fbshipit-source-id: ee6bc807da6de4ed8fba8ab6d52ff5aeff34e8ae
Summary:
The meaning of the root ID is defined by the BackingStore, so move
parsing and rendering into the BackingStore interface.
Reviewed By: xavierd
Differential Revision: D28560426
fbshipit-source-id: 7cfed4870d48016811b604348742754f6cdbd842
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
this change is for mknod
Reviewed By: chadaustin
Differential Revision: D23452153
fbshipit-source-id: 7b9bc6b624fbe81b91770bc65a0d27bc9d397032
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
this change is for getxattr
Reviewed By: chadaustin
Differential Revision: D23451954
fbshipit-source-id: bae73878754d59661cddf7c0b001e506bbc88d13
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
this change is for readlink
Reviewed By: chadaustin
Differential Revision: D23451821
fbshipit-source-id: 1f8ee369a992ab3489a9366f9a972f67461970de
Summary:
Tree metadata prefetching is a second step after a normal fetch. Let's
add it to the batch tree fetching path. Ideally we'd unify this with the
non-batch tree fetching path a bit. Or just get rid of the non-batch path
entirely.
Reviewed By: chadaustin
Differential Revision: D28427267
fbshipit-source-id: e23fc667f609925e8671a71876165ad2cea161b9
Summary:
Let's move the tree metadata fetching logic to a function so it can be
reused for tree batch fetching.
Reviewed By: chadaustin
Differential Revision: D28719443
fbshipit-source-id: 08197c63918334283e29c8327e244114d33ddb4c
Summary:
Now that the backing store has a batch tree API, let's make Eden take
advantage of it.
This is largely just a copy of how blob batch fetching works. The next diff will
add tree metadata prefetching to this fetch path as well.
Reviewed By: chadaustin
Differential Revision: D28426788
fbshipit-source-id: 0dbd2d37950af88a470ef9d8996e9b912988d363
Summary:
In the eden benchmark, EdenFS didn't return the file list, while Watchman would
always do it, thus let's always return the file list.
I've also removed the calls to SetItemsProcessed as the output is completely
misleading. The "Time" column is what needs to be looked at and not the
"items_per_second" that SetItemsProcessed shows.
Reviewed By: kmancini
Differential Revision: D28664475
fbshipit-source-id: e920b311464667a720d9ab00286e094829e292f7
Summary:
Previously if you `sudo umount -f fbsource-nfs` then try to
`eden mount fbsource-nfs`, the mount will fail because the EdenMount already
exists and is still running.
Let's properly unmount our selfs on a force unmount like we do for fuse.
There are two potential ways to detect a fource unmount: the UMNT call to the
mount deamon or the socket to the nfsd closing. The UMNT call is unreliable
(on Linux we do not get the UNMT call on `umount -l`), so this diff pursues the
socket closing option.
When the nfsd socket is closed we trigger the EdenMount unmounting process if
this has not already started.
Reviewed By: xavierd
Differential Revision: D28329482
fbshipit-source-id: 5df8f3eb818a92536095195f1b3a9e412394fbf6
Summary:
The evaluateRecursiveComponentImpl function calls itself recursively, but would
always check for every invocation whether the recursiveChildren_ was empty. We
know that if evaluateRecursiveComponentImpl was called that recursiveChildren_
wasn't empty, so we can remove the check and move it to its caller.
Reviewed By: chadaustin
Differential Revision: D28724574
fbshipit-source-id: b1e07393d2f1cdb605550eca5f0b0b01591a785d
Summary:
Running the glob benchmark, creating the future causes EdenFS's CPU usage to
explode to 1000% for a single threaded glob query. Removing this significantly
reduce the CPU usage to 100% while not affecting performance, despite what the
comment claimed.
Reviewed By: kmancini
Differential Revision: D28617400
fbshipit-source-id: 42ff3401ff1da9d7a7f63bd1c91c8b5b30b182fd
Summary:
This is the last piece that allows both FUSE and NFS to not allocate futures
when replying to getattr calls.
Reviewed By: chadaustin
Differential Revision: D28396819
fbshipit-source-id: 9a4f8d5d03991dbacc4c56116bd56a38773dcd63
Summary:
Since Buck issues glob requests via Watchman, I was curious to understand the
overhead that Watchman adds to see if that is worth optimizing or bypassing.
From the results shown below, it looks like talking to EdenFS directly would
yield a ~4x globbing improvement.
Reviewed By: chadaustin
Differential Revision: D28644517
fbshipit-source-id: 1aa4c43d45248626adde31491fe40836ad6c436a
Summary:
For non-UTF8 names, the PathComponent constructor would raise an exception, and
since that exception wasn't caught by the handler itself, it would bubble up to
the RPC server and a generic "server IO error" would be sent back to the
client. Since non-UTF8 names aren't a server error, but an invalid argument, we
should instead return a different error.
Unfortunately, EILSEQ isn't an error that an NFS server can return, instead
let's use EINVAL as the argument is clearly invalid.
Reviewed By: chadaustin
Differential Revision: D28482032
fbshipit-source-id: b59044f1a76f7eac79e2df07356a0aeafa22e3c5
Summary:
The modeToNfsMode simply didn't consider all the mode bits to be translated to
the proper NFS mode bits. It now does.
Reviewed By: chadaustin
Differential Revision: D28459428
fbshipit-source-id: d879fb1be2085e44110ba552bc47d2770637fc86
Summary:
An NFS client caches the attributes of files to avoid having to request these
very frequently. What this means is that a file changed by another client (or
by the server itself) may take some time to be reflected on the client, that
time depends on the attribute caching configuration of the mount point.
For EdenFS, files can changed in 2 ways:
- Either it is changed by the user via the mount point,
- Or the user runs an `hg update`
For the first one, the client will simply update its attributes appropriately,
but for the second one, the cached attributes will only be updated when the
user does opens the file, any calls to stat prior will return the old
attributes. Since EdenFS runs on the same host, we can force the attributes
caches to be discarded by simply issuing an open call on the file that changed.
Reviewed By: chadaustin
Differential Revision: D28456482
fbshipit-source-id: 91022d35a33e436c47d94403d0c139992f880cf9
Summary: There are some unused warning on Windows, should be an easy fix.
Reviewed By: fanzeyi
Differential Revision: D28663227
fbshipit-source-id: cee18ec28283c0aef18151465a3c13e9be604d7d
