Summary:
This will enable to gather a bit more debugging regarding what processes are
fetching data. The one missing bit on Windows is to collect the process name,
for now, a "NOT IMPLEMENTED" placeholder is put in place.
Reviewed By: wez
Differential Revision: D23946258
fbshipit-source-id: 9f7642c7b9207c5b48ffff0f4eb0333af00bc7d5
Summary: This is not needed, we can use the PrjfsChannel class directly where needed.
Reviewed By: chadaustin
Differential Revision: D23946259
fbshipit-source-id: eafcd38c0927fa282d62ada0986a7ef8b612174b
Summary:
These shows up on Windows when building with mode/win, silencing them is easy,
so let's do it.
Reviewed By: wez
Differential Revision: D23871727
fbshipit-source-id: 7d7ea9504c397b72903e98967188a5295f2f1040
Summary:
These don't compile on Windows, and in order to get mode/win to compile, we
need to avoid compiling their contents. Ideally, we could do that in the
TARGETS files with the select statement, but that's not available in fbcode.
Thus, we do the next best thing: ifdef the file entirely.
Reviewed By: wez
Differential Revision: D23871728
fbshipit-source-id: b4d9df6503eaa008e649afd7bdc665cd37a9585d
Summary: This can be included in all the supported platforms, thus don't #ifdef it.
Reviewed By: wez
Differential Revision: D23858664
fbshipit-source-id: e61da33a97d87cbfab5bb96d2bdaa865d2c01801
Summary:
This enables autodeps, and brings us one step closer to building EdenFS with
Buck on Windows.
Reviewed By: fanzeyi
Differential Revision: D23857794
fbshipit-source-id: c8587a6f7b9e4d9575a62f592c1d0737dff2a8f0
Summary:
Now that prjfs/EdenDispatcher is no longer directly tied to ProjectedFS (sort
of, this is still a bit WIP), we can move it out of the prjfs directory onto
the inodes one. This allows breaking the circular dependency cycle mentioned in
the previous diff where prjfs and inodes depend on each other.
For now, this is merely a copy/paste of the code enclosed in big #ifdef _WIN32,
we might be able to do better later, but for now this is properly good enough.
Reviewed By: wez
Differential Revision: D23857539
fbshipit-source-id: 77c620bac1656d01d7daee4dbf8b10694a589751
Summary:
Most of the non-tests callers have an ObjectFetchContext available, let's use
it instead of a static null context. This will help in understanding why some
files/trees are being fetched.
Reviewed By: chadaustin
Differential Revision: D23745752
fbshipit-source-id: b2e145d9e559bde0542adbc5b20ff56ccfc59ece
Summary:
This allows removal of a NullContext which will provide more details as to why
fetches are triggered.
Reviewed By: chadaustin
Differential Revision: D23745755
fbshipit-source-id: c26f648b8695b86caf8fe15c4bc6c4128d345aa1
Summary:
Now that the win directory only contains the mount directory, we can rename it
to be more faithful to its intent. Since this is about ProjectedFS, let's
rename it "prjfs".
Reviewed By: chadaustin
Differential Revision: D23828561
fbshipit-source-id: cb31fe4652fd4356dc2579028d3ae2c7935371a7
Summary: This will make it easier to build with Buck.
Reviewed By: fanzeyi
Differential Revision: D23827754
fbshipit-source-id: bf3bf4d607a08b9831f9dfea172b2e923a219561
Summary:
Now that sqliteoverlay has a TARGETS file, we can remove the manual, and use
autodeps to have the right dependencies in the TARGETS files. This will help in
getting EdenFS to build on Windows with buck.
Reviewed By: chadaustin
Differential Revision: D23820312
fbshipit-source-id: 34bfd13d2ae6d11a404a9b913562c7d45a4b3de7
Summary:
This commit moves a compile-time template parameter
to be a runtime boolean parameter.
There's a bit of fan-out that, while I don't think it is
super awesome, isn't super terrible either.
The case sensitivity value is read from the checkout config
added in the prior diff in this stack.
Reviewed By: xavierd
Differential Revision: D23751192
fbshipit-source-id: 46f6fe25bfa6666305096ad9c416b510cd3aac8f
Summary:
This diff teaches the CheckoutConfig how to determine
whether a given checkout should be case-sensitive (the default)
or case-insensitive-case-preserving.
This option is passed through to the fuse channel initialization,
so that the kernel will respect it, however, our DirEntry layer
doesn't yet know that it should respect this.
There's currently no UI to set this option. My game plan
is to suggest the following steps to folks that want to try
this out:
```
$ eden stop
$ vim ~/local/.eden/clients/ovrsource/config.toml
```
and then add this line to the `[repository]` section:
```
case-sensitive = false
```
and finally:
```
$ eden start
```
Reviewed By: xavierd
Differential Revision: D23751184
fbshipit-source-id: 6facb23c460cfff6e37d0091b51b97ab06f62c91
Summary:
Now that all the pieces are in place, we can plumb the request context in. For
now, this adds it to only one callback as I figure out more about it and tweak
it until I have something satisfactory. There are some rough edges with it that
I'm not entirely happy about, but as I change the notification callback to be
more async, I'm hoping to make more convenient to use and less clanky.
Reviewed By: fanzeyi
Differential Revision: D23505508
fbshipit-source-id: d5f12e22a8f67dfa061b8ad82ea718582c323b45
Summary:
Since the Stub.h now only contains NOT_IMPLEMENTED, let's move it to its own
header outside of the win directory.
Reviewed By: genevievehelsel
Differential Revision: D23696244
fbshipit-source-id: 2dfc3204707e043ee6c89595668c484e0fa8c0d0
Summary:
With this gone, we will be able to rename and move Stub.h outside of the win
directory.
Reviewed By: genevievehelsel
Differential Revision: D23696243
fbshipit-source-id: ea05b10951fa38a77ce38cd6a09a293364dbeec9
Summary:
While the code isn't compiled, this makes the thrift definition available to
the rest of the code, eliminating the need for having a stub for
SerializedInodeMap on Windows.
Reviewed By: genevievehelsel
Differential Revision: D23696242
fbshipit-source-id: 8a42dd2ed16887f3b7d161511e07aaa35fd1b968
Summary:
This was only used in EdenMount.h, to declare a method that was not compiled on
Windows, let's ifdef that method instead.
Reviewed By: chadaustin
Differential Revision: D23693494
fbshipit-source-id: 1eda62f2ae3a38a30aa0b517911635ef3d3896c2
Summary: A following diff will make use of the channel.
Reviewed By: wez
Differential Revision: D23505510
fbshipit-source-id: c044fff51c8771b1ead86333317e5c617184075c
Summary:
Readdir tries to be smart and prefetch the metadata for each of the children.
But this uses the old path to read metadata for files which can cause eden to
download the blob. When metadata prefetching is turned on in the backing store
it is better to leave this to metadata prefetching to the backing store.
Reviewed By: wez
Differential Revision: D23476876
fbshipit-source-id: 41cc5e6f423f19adb18581564c069c12621b6c1b
Summary:
Most of the RequestData code is platform generic, but bits of it are currently
strongly tied to FUSE. By splitting these 2 parts, we will be able to use the
RequestContext class in Windows too and not having to re-implement all the
logic there.
Reviewed By: chadaustin
Differential Revision: D23482072
fbshipit-source-id: 857fd9ca4264d0f308ec10cc487e9ff3eeb5ee16
Summary:
We are unifying C++ APIs for accessing optional and unqualified fields:
https://fb.workplace.com/groups/1730279463893632/permalink/2541675446087359/.
This diff migrates code from accessing data members generated from unqualified
Thrift fields directly to the `field_ref` API, i.e. replacing
```
thrift_obj.field
```
with
```
*thrift_obj.field_ref()
```
The `_ref` suffixes will be removed in the future once data members are private
and names can be reclaimed.
The output of this codemod has been reviewed in D20039637.
The new API is documented in
https://our.intern.facebook.com/intern/wiki/Thrift/FieldAccess/.
drop-conflicts
Reviewed By: simpkins
Differential Revision: D23465292
fbshipit-source-id: bb9df3ad183685fae28173da7275e6ecd34df048
Summary:
None of these were used, let's remove them.
ps: I thought we had a system to detect unused headers and lint about them?
Reviewed By: chadaustin
Differential Revision: D23465783
fbshipit-source-id: c21a34c9838db29f4fd0057d3be4e0fcb527cd6d
Summary:
This is not per-se fuse related, thus move it to a common location and remove
the duplicated define in FileInode.h
Reviewed By: chadaustin
Differential Revision: D23465192
fbshipit-source-id: 5fa7709f127c2d3372ee5ea3aeb89e793ea5b9f7
Summary: This file is not fuse specific, therefore, let's move it to a non-fuse folder.
Reviewed By: chadaustin
Differential Revision: D23464460
fbshipit-source-id: f70e94bb0ecc37bd74798fd230dee2058484f31b
Summary:
This commit introduces a new process spawning class derived
from the ChildProcess class in the watchman codebase.
`SpawnedProcess` is similar to folly::Subprocess but is designed around the
idea that we will use a system provided spawning API to start a process, rather
than assuming the use of `fork`.
`fork` is to be avoided because it can be expensive for processes with large
address spaces and also because it interacts poorly with threads on macOS. In
particular, we see the objC runtime terminating our process in some scenarios
where fork and threads are mixed.
There are some important differences from `folly::Subprocess` and that means
that some assumptions and uses need to be altered slightly from their prior
workings. For example, detaching a SpawnedProcess moves the responsibility of
waiting on the child to a periodic task as there is no way to detach via
posix_spawn without also using fork.
On the plus side, this commit allows unifying spawning between posix and
windows systems, which simplifies the code!
Reviewed By: xavierd
Differential Revision: D23287763
fbshipit-source-id: b662af1d7eaaa9ed445c42f6c5765ae9af975eea
Summary:
As previous diffs in the stack show there were at least one place in the
codebase which used incorrect object context logger and that resulted in "blind
spots" in undesired file fetches logging i.e. undesired file fetches were
logged, but neither pid nor cmd-line was logged.
There are quite a few places in the codebase that use null
object fetch context, and threading the correct object fetch context to all of
them might be hard. Threading the context is a bit annoying, so it would be good to know something like "EdenDispatcher code is responsible for most of the blind spots, so let's thread the correct context there first". Or it would be equally good to know that none of the null object context are responsible for blind spots.
This diff might help us decide where we need to thread real object fetch context
first. Instead of passing null object fetch context let's pass null object
fetch context with causeDetail field. This field will be logged to scuba (see
BackingStoreLogger::logImport code), and instead of getting "Unknown" interface
we'll get e.g. "Unknown - EdenDispatcher::create", and that would highlight
where we need to thread the context.
A note about implementation - getNullContextWithCauseDetail returns a raw pointer
which is expected to be static i.e. it should work similarly to current
getNullContext implementation. It's quite a hack, but allows us to get rid of
memory allocations (we'd have one memory allocation per place in the code where
getNullContextWithCauseDetail). Let me know if you are ok with this hack.
Reviewed By: kmancini
Differential Revision: D23422526
fbshipit-source-id: e576bba9fc09e160fc42771c7589cdd1694d93c0
Summary:
Scuba logging that tracks undesired file fetches has some blind spots i.e. a
lot of fetches have null pid and null cmd line. This diff tries to fix another
part of the problem.
TreeInode::getOrLoadChild() has TODO `pass a fetch context down through
getOrLoadChild to track this load`. This diff fixes this TODO, and also starts
to pass context from EdenDispatcher:lookup method.
Note that it adds quite a lot of new `ObjectFetchContext::getNullContext()`
calls, and potentially those might be responsible for blind spots in logging.
I'll try to address this problem in the next diffs.
Reviewed By: kmancini
Differential Revision: D23418218
fbshipit-source-id: 319d7436494d8dce3580289aae9963aa13bfc191
Summary:
Scuba logging that tracks undesired file fetches has some blind spots i.e. a
lot of fetches have null pid and null cmd line. This diff fixes at least part
of the problem.
TreePrefetchContext which is used from TreePrefetchLease didn't logged client
pid at all (in fact, it logged almost nothing). This diff fixes at least one blind spot, however it doesn't look like this is the only one.
Reviewed By: kmancini
Differential Revision: D23417451
fbshipit-source-id: 107884e94c6b40de999328ec2ef78fe22174c1ca
Summary:
Setting up, tearing down, and querying RequestContext has some
overhead that I would like to avoid in the inner FUSE loop, so replace
RequestData with a single class that's heap-allocated at the start of
a request and is guaranteed to survive until the request ends, and is
otherwise explicitly passed where it's needed.
Reviewed By: kmancini
Differential Revision: D22712310
fbshipit-source-id: fc30d0b0f7e22b39306b857194ea07a913110b0f
Summary: Use `Subprocess` in `win/utils` to call `edenfsctl redirection fixup` after mount is done.
Reviewed By: wez
Differential Revision: D22958764
fbshipit-source-id: a485994a3816169299e8514a5c355f3d37edad99
Summary:
Up to now, Windows had to have its own version of folly::{readFile, writeFile,
writeFileAtomic} as these only operate on `char *` path, which can only
represent ascii paths on Windows. Since the Windows version is slightly
different from folly, this forced the code to either ifdef _WIN32, or use the
folly version pretending that it would be OK. The Windows version was also
behaving slightly differently from folly. For instance, where folly would
return a boolean to indicate success, on Windows we would throw an exception.
To simplify our code, add type safety and unify both, we can implement our own
wrappers on top of either folly or Windows APIs.
We still have some code that uses folly::readFile but these should only be
dealing with filedescriptors. As a following step, we may want to have our own
File class that wraps a file descriptor/HANDLE so we can completely remove all
uses of folly::readFile.
Reviewed By: wez
Differential Revision: D23037325
fbshipit-source-id: 2b9a026f3ee6220ef55097abe649b23e38d9fe91
Summary:
After enabling strace profiling for the dispatcher, it became very clear that
once files are populated on disk, most of the notification in edenfs come from
listing directories. While looking at the code to find any improvements, I
picked up a couple of low hanging fruits and cleanup that should help (not
measured).
Reviewed By: chadaustin
Differential Revision: D23121434
fbshipit-source-id: 8a8b369f3be6ffb6097e6e12ab0f07af0a6d56b7
Summary:
From a quick experiment, this greatly cuts down on the amount requests to
nonexistent files. For instance, the .hg directory in folders is now only
looked up once and no longer afterwards.
Reviewed By: wez
Differential Revision: D23112343
fbshipit-source-id: 223134ca591054ae9ac2e839033bbd1b714443da
Summary:
This is purely moving code around to be more similar between Unices/Windows.
At some point in the future, I'd like to see just one EdenDispatcher class in
the shared code with as few platform specific code (ie: ifdef) as possible,
this is small step in this direction.
Reviewed By: wez
Differential Revision: D23112345
fbshipit-source-id: feb276040aea106d5951af30c5d445b4cb654e01
Summary:
On Linux, all the notifications are sent to the strace logger, let's do the
same on Windows. Whenever possible, I've tried to use the name of the Linux
syscalls that correspond to the notification.
Reviewed By: chadaustin
Differential Revision: D23105536
fbshipit-source-id: 31c6e545cdec93b0f751682e34c34b894b2890c4
Summary:
The StringConv.h header contains many functions to convert from Windows paths
to Eden's path (and vice versa) to workaround the fact that Eden's path don't
support wide strings that Windows uses. Let's simply add support for these wide
strings in PathFuncs so we can greatly simplify all the call sites. Instead of
calling "edenToWinName(winstr)", "PathComponent(winstr)" is both more
descriptive and more idiomatic.
To be fair, I'm not entirely a fan of the approach taken in this diff, as this
adds Windows specific code to PathFuncs.h, but I feel that the benefit is too
big to not do that.
Reviewed By: chadaustin
Differential Revision: D23004523
fbshipit-source-id: 3a1507e398a66909773251907db01e06603b91dd
Summary:
Reading the entire file to compute its sha1 can cause EdenFS to OOM on large
files. Instead, let's simpy read 8k at a time, like what we do on Linux.
Reviewed By: chadaustin
Differential Revision: D22924710
fbshipit-source-id: cd472f396a4385049994a1976c7d046cb901337a
Summary:
The initialized_ field was never set, which meant that Overlay::close would
never close the sqlite overlay. This had the negative effect of allowing the
unmount thrift call to return before the sqliteoverlay is properly closed which
on Windows meant that the overlay couldn't be deleted.
Reviewed By: chadaustin
Differential Revision: D22915353
fbshipit-source-id: e501b2d0268449d109cc37dfdc62faebf1953e09
Summary:
Cache invalidation is hard, and on Windows we avoided doing a lot of them. It
turns out, this was the wrong decision as it's fairly easy to find cases where
the filesystem view is different from the manifest state.
Since the Linux code is most likely correct in where the invalidation is done,
let's also do the same on Windows, removing a whole lot of #ifdef. It is very
likely that as a result of this diff we end up invalidating more than needed,
thus slowing down EdenFS, but at this point I'd prefer to err on the side of
correctness, performance will come later.
While invalidating files should use PrjDeleteFile, for directories, we simply
need to mark them as placeholder, as directories created by a user won't have a
placeholder, thus ProjectedFS would bypass EdenFS when listing in.
Reviewed By: chadaustin
Differential Revision: D22833202
fbshipit-source-id: d807557f5e44279c49ab701b7a797253ef1f0717
Summary:
Avoid some overhead and complexity by storing BufVec as a
unique_ptr<IOBuf>. The complexity can be reintroduced if we ever find
FUSE splice support to be a performance win for us.
Reviewed By: kmancini
Differential Revision: D22710795
fbshipit-source-id: e58eedc0fb5cea9e9743ccd20d3e4e2b7cc5d198
Summary:
Avoid the cost of dynamically querying whether we are in a FUSE
request handler or not by passing a flag.
Reviewed By: kmancini
Differential Revision: D22710480
fbshipit-source-id: 010bb8efee8074441aa20aab0eb12277452c5252
Summary:
Avoid the cost of dynamically querying whether we are in a FUSE
request handler or not by passing a flag.
Reviewed By: kmancini
Differential Revision: D22710452
fbshipit-source-id: 818035b72b793fa895147d9df3bb668d5b9c55f3
Summary:
Avoid the cost of dynamically querying whether we are in a FUSE
request handler or not by passing a flag.
Reviewed By: kmancini
Differential Revision: D22710422
fbshipit-source-id: 65b0737ad5f8ca74d12f2c657691d3751df4aa54
Summary:
Avoid the cost of dynamically querying whether we are in a FUSE
request handler or not by passing a flag.
Reviewed By: genevievehelsel
Differential Revision: D22710397
fbshipit-source-id: 7c62f45dfc227416c91070842a349b9d0c626cba
Summary: This is unused, and won't ever be used.
Reviewed By: genevievehelsel
Differential Revision: D22744352
fbshipit-source-id: 9d20db608f972288eaf33e3ea0a79ffe5e13e03e
Summary:
Both unices and Windows needs to invalidate the cache in the same place, let's
avoid ifdef and consolidate the function name to clean things up a bit.
Reviewed By: chadaustin
Differential Revision: D22741709
fbshipit-source-id: 04060c0080eff9840abd22747ea48404fa50fd86
Summary:
We're still reading the entire file at a time, but this paves the way to not do
that.
While the change looks big, a lot of it is just moving code around. The main
gist of it is removing EdenMount::readFile and writing the proper future
combinator in EdenDispatcher::getFileData.
Reviewed By: wez
Differential Revision: D22361748
fbshipit-source-id: 6391a29d25a4c9e61b91952c40c21ad52e728c8b
Summary:
This would enable the enumeration callback to use ProjectedFS asynchronous
completion.
Reviewed By: chadaustin
Differential Revision: D22361747
fbshipit-source-id: b2d31533ee5128e9dd3da7f91d5225331cf8e926
