Summary:
Logging all these throttling notifications is not necessary. There can
sometimes be big batches of fetches (like 100s of K). Lets reduce this by a
factor of 1000.
Note we also would like to add logging of what process triggered these fetches
what endpoint they use etc. This will help us identify the workflows causing it,
so we could address them or skip aux data fetching in these code paths.
But this requires some fiddling with ObjectFetchContext and the logging
code, so its gonna take a bit longer :(
Reviewed By: genevievehelsel
Differential Revision: D25505654
fbshipit-source-id: e7c40164db86fadf4baf0afd0c52879e0cb2568b
Summary: Update name to match usage of Try as tri-state, since this method also throws if the Try is empty
Reviewed By: yfeldblum
Differential Revision: D25737810
fbshipit-source-id: a4166153362f07353d212216fbaf7105867eef2a
Summary:
On Windows, the GUID of the mount point identifies the virtualization instance,
that GUID is then propagated automatically to the created placeholders when
these are created as a response to a getPlaceholderInfo callback.
When the placeholders are created by EdenFS when invalidating directories we
have to pass GUID. The documentation isn't clear about whether that GUID needs
to be identical to the mount point GUID, but for a very long time these have
been mismatching due to the mount point GUID being generated at startup time
and not re-used.
One of the most common issue that users have reported is that sometimes
operations on the repository start failing with the error "The provider that
supports file system virtualization is temporarily unavailable". Looking at the
output of `fsutil reparsepoint query` for all the directories from the file
that triggers the error to the root of the repositories, shows that one of the
folder and its descendant don't share the same GUID, removing it solves the
issue.
It's not clear to me why this issue doesn't always reproduce when restarting
EdenFS, but a simple step that we can take to solve this is to always re-use
the GUID, and that hopefully will lead to the GUID always being the same and
the error to go away.
Reviewed By: fanzeyi
Differential Revision: D25513122
fbshipit-source-id: 0058dedbd7fd8ccae1c9527612ac220bc6775c69
Summary:
This commit adds a new eden configuration option that
controls whether we try to load our edenfs.kext in preference to
an alternative fuse implementation on macOS.
The majority of this diff is plumbing to convey the configuration
value through to the privhelper, which is relatively restrictive
due to its root-ness.
I've also updated watchman and mercurial to be aware of the new
filesystem type that shows up in the mount table.
Reviewed By: genevievehelsel
Differential Revision: D25065462
fbshipit-source-id: 4f35b9440654298e2706a0d0613d97eb63451999
Summary:
One of the main sub-par user experience on Windows is the lack of notification
of any kind when EdenFS can't reach the Mercurial servers. Prior to this diff,
the callbacks would never return, causing commands to simply hangs for the
user.
As a first step, let's add a timeout, a later step will hook the notification
mechanism used on macOS/Linux to display a notification when timeouts occurs.
The only callback that doesn't have a proper timeout is the notification one,
as timing out on these would mean that EdenFS won't have registered that some
files/directories have been materialized which will lead to inconsistencies
later.
Reviewed By: kmancini
Differential Revision: D24809645
fbshipit-source-id: 0ddd9d443a17db405a3edbaa8edecf3764c31d37
Summary:
Converting back and forth between folly::fs::path and AbsolutePath appears to
be problematic on Windows as NUL bytes appears in the paths, causing the tests
to fail. Instead of doing this conversion, let's simply use AbsolutePath everywhere.
Reviewed By: chadaustin
Differential Revision: D25033803
fbshipit-source-id: 6c45c2a20fc4bf18cecc838b219faacfeb8386d8
Summary:
The EdenFS codebase uses folly/logging/xlog to log, but we were still relying
on glog for the various CHECK macros. Since xlog also contains equivalent CHECK
macros, let's just rely on them instead.
This is mostly codemodded + arc lint + various fixes to get it compile.
Reviewed By: chadaustin
Differential Revision: D24871174
fbshipit-source-id: 4d2a691df235d6dbd0fbd8f7c19d5a956e86b31c
Summary:
The path iterator functions are skipping sanity checking of paths as they
assume that the path has already been validated. Unfortunately, that isn't the
case as the only sanity checking we are doing is on the beginning and end of
the string that is passed in. Notably, `RelativePathPiece{"./foo"}` would not
fail the sanity checking even though it it's not made of valid components.
Reviewed By: chadaustin
Differential Revision: D24439804
fbshipit-source-id: 852b3a1180b185cb0bfb96bf5bcdc98b231f32c5
Summary:
This stack updates eden to be able to check all of the locations that able
users certificate may reside.
There can be multiple places where a cert may reside (we cant always
definitively choose one place to look based on the platform). Thus we
need to be able to configure multiple locations for certs in our eden
config.
This makes the switch over in eden from using one place for the client
cert to use and using the first available client cert from a list.
NOTE: most of this diff is fixing unit tests take a look at `EdenConfig.h` and `EdenConfig.cpp` first
Reviewed By: wez
Differential Revision: D23359939
fbshipit-source-id: 44beecce3ef098a734dbd7c5eb3fa5f0aad6b50b
Summary:
These shows up on Windows when building with mode/win, silencing them is easy,
so let's do it.
Reviewed By: wez
Differential Revision: D23871727
fbshipit-source-id: 7d7ea9504c397b72903e98967188a5295f2f1040
Summary: This will make it easier to build with Buck.
Reviewed By: fanzeyi
Differential Revision: D23827754
fbshipit-source-id: bf3bf4d607a08b9831f9dfea172b2e923a219561
Summary:
This diff teaches the CheckoutConfig how to determine
whether a given checkout should be case-sensitive (the default)
or case-insensitive-case-preserving.
This option is passed through to the fuse channel initialization,
so that the kernel will respect it, however, our DirEntry layer
doesn't yet know that it should respect this.
There's currently no UI to set this option. My game plan
is to suggest the following steps to folks that want to try
this out:
```
$ eden stop
$ vim ~/local/.eden/clients/ovrsource/config.toml
```
and then add this line to the `[repository]` section:
```
case-sensitive = false
```
and finally:
```
$ eden start
```
Reviewed By: xavierd
Differential Revision: D23751184
fbshipit-source-id: 6facb23c460cfff6e37d0091b51b97ab06f62c91
Summary:
Since the Stub.h now only contains NOT_IMPLEMENTED, let's move it to its own
header outside of the win directory.
Reviewed By: genevievehelsel
Differential Revision: D23696244
fbshipit-source-id: 2dfc3204707e043ee6c89595668c484e0fa8c0d0
Summary:
This stack updates eden to be able to check all of the locations that able
users certificate may reside.
THRIFT_TLS_CL_CERT_PATH is usally set with the location for the users x509
certs. So it seems best to check this location. In order to be able to check
this location, we need to be able to resolve the enviroment variable in our
parsing.
Reviewed By: wez, genevievehelsel
Differential Revision: D23359815
fbshipit-source-id: 2008cc52ab64d23dbcfda41292a60a4bf77a80df
Summary:
This stack updates eden to be able to check all of the locations that able
users certificate may reside.
There can be multiple places where a cert may reside (we cant always
definitively choose one place to look based on the platform). Thus we
need to be able to configure multiple locations for certs in our eden
config.
Thus we need to be able to parse a list of options for a key in our config
parsing.
**Disclaimer this is really icky**
Our `FieldConverter` interface takes a string to parse. So this means
that after parsing the config file for each value we have to re-serialize it
into a string to pass it in here. Previously we only supported string and
bool values so this re-serialization was not too terrible. Now that we want
to support arrays this re-serialization is extra gross. To minimize the grossness,
I am reusing cpptoml for serializing / deserializing around the `FieldConverter`
interface.
Long term it would be better if FieldConverter took a cpptoml::base or
something more generic instead of a string so we dont have to do this.
But that will be a big refactor, and I don't currently have bandwidth for it :(
Reviewed By: wez
Differential Revision: D23359928
fbshipit-source-id: 7c89de485706dd13a05adf19df28425d2c1756a8
Summary:
getConfigStat had a bug where it, instead of clearing the bits of
*configStat, cleared the bits of the pointer itself. This caused the
stat struct for missing files to be uninitialized memory, causing
configs to reload. Write a test and fix the bug.
Reviewed By: xavierd
Differential Revision: D23645087
fbshipit-source-id: ad42f7ec1b313f668604e3a7f6c8200f6b94b23d
Summary:
While hacking on some code, I ran into a situation where some
zero-initialized stat structs weren't actually being zeroed. This was
either a compiler bug or a situation where the build system was not
correctly rebuilding everything after my changes, and I did not have
enough disassembly available to investigate.
Either way, since this code assumes zero bits in some nonobvious ways,
explicitly assert they are.
Reviewed By: xavierd
Differential Revision: D23644819
fbshipit-source-id: eb6bff9ff997379113db1e1bf9d6a0a538f10f0b
Summary:
We noticed spurious config file reloads, so add some logging to help
track that down.
Reviewed By: xavierd
Differential Revision: D23644447
fbshipit-source-id: 9953a17de402660c7f6491fb9abd8d702fa290e8
Summary:
Currently we use a single path prefix to configure data fetch logging in eden
(i.e if the path of a file which we fetch is an extension of our configured
path, then we log that data fetch. )
There is some interest in extending this to multiple path prefixes, so that we
can log separate parts repo.
Reviewed By: StanislavGlebik
Differential Revision: D22877942
fbshipit-source-id: f6eb3dcb4fa460b4acab09677e972caf9421ddff
Summary:
Up to now, Windows had to have its own version of folly::{readFile, writeFile,
writeFileAtomic} as these only operate on `char *` path, which can only
represent ascii paths on Windows. Since the Windows version is slightly
different from folly, this forced the code to either ifdef _WIN32, or use the
folly version pretending that it would be OK. The Windows version was also
behaving slightly differently from folly. For instance, where folly would
return a boolean to indicate success, on Windows we would throw an exception.
To simplify our code, add type safety and unify both, we can implement our own
wrappers on top of either folly or Windows APIs.
We still have some code that uses folly::readFile but these should only be
dealing with filedescriptors. As a following step, we may want to have our own
File class that wraps a file descriptor/HANDLE so we can completely remove all
uses of folly::readFile.
Reviewed By: wez
Differential Revision: D23037325
fbshipit-source-id: 2b9a026f3ee6220ef55097abe649b23e38d9fe91
Summary:
From a quick experiment, this greatly cuts down on the amount requests to
nonexistent files. For instance, the .hg directory in folders is now only
looked up once and no longer afterwards.
Reviewed By: wez
Differential Revision: D23112343
fbshipit-source-id: 223134ca591054ae9ac2e839033bbd1b714443da
Summary: Previously we use HgImporter prefetch request in `prefetchBlobs()`, but using `getBlob()` can give us more control over the prefetch process later. So now `getBlob()` is used in `prefetchBlobs()` when `useEdenNativePrefetch` is configured as true.
Reviewed By: kmancini
Differential Revision: D22984848
fbshipit-source-id: 0bd0b1c5b50bb16da36f188915904d0223827dc3
Summary:
Previously we fetched metadata by commit hash and path. We knew this would be a
little extra expensive, but turns out this is a lot extra expensive.
Wait why is it expensive?
In short: lots of extra lookups that are not satisfied by cache :(
In long:
1. Each piece of the path would require a read to fetch the fsnode for that tree.
So this means asking for the metadata of a/b/c/d/e means 5 reads.
2. Normally these reads could be cached, but often we would make these requests
with a commit hash for a draft commit. On the server side this info is not
cached for a draft commit, this means a lot of database reads and recalculating.
(Most of the real uses of metadata prefetching is when an engineer is working
on a local commit. We just use the commit hash of the commit the user was on
when fetching metadata for a tree, even if that tree hasn't changed since a public
commit. so this means lots of requests with draft commit hashes).
Fetching by manifest id we are able to bypass this sequential path look up.
(and even if we are on a draft commit, if the tree has not locally changed
since a public commit, the manifest id will be the same as the public commit
avoiding this whole draft commit issue).
This allows us to query scs with a manifest id for a tree.
Reviewed By: wez
Differential Revision: D22990687
fbshipit-source-id: aa81d67de1f1d04a14d174774ee216f5ac6be5ba
Summary:
We are unifying C++ APIs for accessing optional and unqualified fields:
https://fb.workplace.com/groups/1730279463893632/permalink/2541675446087359/.
This diff migrates code from accessing data members generated from unqualified
Thrift fields directly to the `field_ref` API, i.e. replacing
```
thrift_obj.field
```
with
```
*thrift_obj.field_ref()
```
The `_ref` suffixes will be removed in the future once data members are private
and names can be reclaimed.
The output of this codemod has been reviewed in D20039637.
The new API is documented in
https://our.intern.facebook.com/intern/wiki/Thrift/FieldAccess/.
drop-conflicts
Reviewed By: yfeldblum
Differential Revision: D22631599
fbshipit-source-id: 9bfcaeb636f34a32fd871c7cd6a2db4a7ace30bf
Summary:
Prefetching metadata for the entries in a tree when we fetch it saves us
an extra round trip to the server to fetch a blob when only the metadata
for that blob is fetched. (This can happen often while parsing targets in
builds)
This implements a custom metadata fetcher to fetch this data when we
fetch a tree from the server.
Reviewed By: chadaustin
Differential Revision: D22086639
fbshipit-source-id: 5fe31d375bf6f7376eb67496d553d6b4540fc0c9
Summary:
This introduces a class to manipulate the metadata for all the entries in a
tree. This adds serialization and deserialization to this class so that it can
be written to the local store.
Why do we need this? We need some way to easily check when we have already
fetched metadata for a tree and do not need to refetch this from the server to
avoid expensive network requests. Later diffs add functionally to store the metadata
for tree entries in the local store under the tree hash using this class.
Reviewed By: chadaustin
Differential Revision: D21959015
fbshipit-source-id: 0c0e8750737f3076c1f9604d0319cab7f2658656
Summary: This diff made fetch threshold configurable, so we can change it later as repository size grows.
Reviewed By: fanzeyi
Differential Revision: D22337850
fbshipit-source-id: 4b46420cb4e7164a3f1080279d67fa5f90549cd8
Summary:
Eden can sometimes unexpectedly fetch files from the server, and we want
to know why this is happening. This adds logging for the source of
data fetching in edens backing store to help obviate why these fetches
are happening.
This temporarily adds the logging in the HgQueuedBacking store to get a naive
version of logging rolled out sooner. Follow up changes will move this logging
closer to the data fetching itself if possible (in HgDatapackStore and HgImporter).
Reviewed By: chadaustin
Differential Revision: D22012572
fbshipit-source-id: b1b012ce4ee133fbacecd586b7365c3c5a5386df
Summary:
The old `py` Thrift language support doesn't correctly handle string
vs. bytes, which causes an exception to be thrown when deserializing
paths or blobs that aren't UTF-8.
We will eventually want to migrate to the py3 language implementation,
which supports streaming.
Reviewed By: genevievehelsel
Differential Revision: D21693082
fbshipit-source-id: 0ea10fd3960f5acba353bccb83b5cf539e7eeffb
Summary: Reading from hgcache directly has been rolled out for quite a while now. Let's remove the configuration and make it default behavior.
Reviewed By: chadaustin
Differential Revision: D21585482
fbshipit-source-id: 7f30e262642adf58388a0580aa5a63595fa89155
Summary:
The disabled tests are due to the st_mtime stored in the struct stat only
having a second granularity (as expected), but the tests are changing these
files faster than that, causing them to fail. I'll attempt to fix the file
change detection mechanism on a later diff.
Reviewed By: wez
Differential Revision: D21319025
fbshipit-source-id: a3f2d62a3ce56fea37a62bfad73306ac1556f772
Summary:
Same as the previous one, paths are different on Windows, and thus we need to
test against a different path.
Reviewed By: wez
Differential Revision: D21319021
fbshipit-source-id: 849c86574e69c3f638ea180c3b594aaeae2970ad
Summary:
The use of realpath is needed to resolve a unix-style path to a Windows style
one. This helps keep the test generic with no `#ifdef _WIN32`
Reviewed By: wez
Differential Revision: D21319019
fbshipit-source-id: b8cdd81f0afdd135849a5b850d854399cef8cef8
Summary:
Add new fields to EdenConfig to control log rotation settings, and update
edenfs_monitor to set up the log rotation strategy using these settings.
Reviewed By: chadaustin
Differential Revision: D20427271
fbshipit-source-id: 9960bdb6f4d077a4e21fb4a6209aa02ab21ad653
Summary:
The disabled tests are due to the precision of st_mtime on Windows only being
a second, but the tests are changing the config faster than that.
Reviewed By: wez
Differential Revision: D21319023
fbshipit-source-id: bcdce24f70ce99984cabb290338ac94a2459e9de
Summary:
On Windows, the following pseudo code:
int fd = open("file");
struct stat st;
fstat(fd, &st);
Will have a different st_dev than the following code:
struct stat st;
stat("file", &st);
Since the FileChangeMonitor uses st_dev as a way to compare if a file
changed, the config is always reloaded.
For our use case, the filesize and its mtime should be enough to know if the
configuration changed, so let's only use these 2 on Windows.
Reviewed By: wez
Differential Revision: D21312679
fbshipit-source-id: f08b3eb7d6037f5d88ece82efe3a5437b1954ba2
Summary:
This brings it closer to folly::writeFile which should help in avoiding ifdef
whenever we want to use it.
Reviewed By: wez
Differential Revision: D21319020
fbshipit-source-id: 80fbf7fba671b18b5ef68375910e1a2a8869f590
Summary:
The bind-mounts configuration has been ignored by EdenFS since D17236366.
This removes all CLI code for dealing with this config section.
Reviewed By: wez
Differential Revision: D20876460
fbshipit-source-id: 6b3f3552de25ee28fc0418a6aaec14446520203c
Summary:
In rare situations users end up manually deleting or removing their `.eden`
state directory without ever killing their running `edenfs` process. This can
leave this old process running indefinitely despite the fact that it's state
directory is no longer present (or has perhaps even been replaced with new
data).
This updates edenfs to periodically check if its lock file is still valid, and
quit if it isn't. This will help prevent old `edenfs` processes from running
indefinitely after their state directory is no longer valid.
Reviewed By: wez
Differential Revision: D20613841
fbshipit-source-id: d9a3a1e7e9b05806e086e794ebbc36e1cc71831a
Summary: It seems to be stable and not causing issues. Let's make it default everywhere.
Reviewed By: wez
Differential Revision: D19896738
fbshipit-source-id: cf6abe8f536e570017742b3a0674213a932a6a4d
Summary:
This commit causes a desktop notification to be shown if we generate
EIO or ETIMEDOUT responses via fuse; the prompt is intended to make it obvious
to the user that they need to connect to the VPN.
The commit by itself doesn't show a notification, it allows configuring a
command that can be run to do something to show a notification.
The test plan includes one such configuration for our corp environment.
* It doesn't trigger for thrift-originated downloads (eg: prefetch), only for
VFS operations through FUSE.
* Ideally we'd know exactly when we have a network related error in the store
code and use that to trigger the notification. However, we have a rather
convoluted set of importers and fallbacks today, one of which is interpreting
a generic response returned from a pipe, so it is not especially clear
exactly where we should locate the logic
Reviewed By: chadaustin
Differential Revision: D17513364
fbshipit-source-id: 45134f3672679cb5580cb0c1bc12a0d6e38525ca
Summary:
cpptoml has traversal functionality for table reads, but not for
writes. Add a helper function for reading a config value and updating
the TOML table if it's unset.
Reviewed By: fanzeyi
Differential Revision: D19671264
fbshipit-source-id: e2b78d338af35d51fddaa258b7f45f8966d00a26
Summary: Easier to zero initialize structs with braces, which defines that even padding is zeroed.
Reviewed By: wez
Differential Revision: D19655675
fbshipit-source-id: 2fd12383324029646707e93008cf9ad34e9f1dce
Summary: This code is no longer used now that `eden redirect` exists.
Reviewed By: wez
Differential Revision: D19565010
fbshipit-source-id: 4703bc39a024c4c491a83ef5a6e85711a8f7e4d8
