Summary:
Now that `mount(2)` complete, we can start plumbing the rest of the mount code
to accomodate for NFS. Trying to find a common ground between Prjfs, FUSE and
Prjfs is harder than I wish it would be and thus I wasn't able to find a
satisfatory solution. For now, I went with a std::variant that stores either a
FuseChannel, or the Nfsd3. In the future once Nfs is stabilized and we have a
good understanding of how it differs (and where it doesn't), EdenMount would
probably need a good refactoring.
Reviewed By: kmancini
Differential Revision: D26500111
fbshipit-source-id: f02a2eaf8890261f150d7cdd2cdfd134aa62c2aa
Summary:
By moving the createChannel outside of the EdenMount class, we no longer need
the channelType alias, so let's do it.
Reviewed By: kmancini
Differential Revision: D26500113
fbshipit-source-id: f992ed2aaac5d692d316d06340bf9b219a2d7006
Summary:
Instead of having one "Dispatcher" type that the various backend overload,
let's simply have a per-mount type dispatcher type. The previous model worked
fine when EdenFS supported only one way of mounting a repository, but with NFS
coming, unix platform will support both FUSE and NFS, making the Dispatcher
overload nonsensical.
As a behavioral change, the dispatcher lifetime and ownership is changed a bit.
It used to live for the duration of the EdenMount object, but is now tied to
the channel lifetime, as it is now owned by it.
Reviewed By: kmancini
Differential Revision: D26329477
fbshipit-source-id: 3959b90a4909e3ab0898caa308f54686f59a943c
Summary:
This will enable to gather a bit more debugging regarding what processes are
fetching data. The one missing bit on Windows is to collect the process name,
for now, a "NOT IMPLEMENTED" placeholder is put in place.
Reviewed By: wez
Differential Revision: D23946258
fbshipit-source-id: 9f7642c7b9207c5b48ffff0f4eb0333af00bc7d5
Summary: This is not needed, we can use the PrjfsChannel class directly where needed.
Reviewed By: chadaustin
Differential Revision: D23946259
fbshipit-source-id: eafcd38c0927fa282d62ada0986a7ef8b612174b
Summary:
This enables autodeps, and brings us one step closer to building EdenFS with
Buck on Windows.
Reviewed By: fanzeyi
Differential Revision: D23857794
fbshipit-source-id: c8587a6f7b9e4d9575a62f592c1d0737dff2a8f0
Summary:
Most of the non-tests callers have an ObjectFetchContext available, let's use
it instead of a static null context. This will help in understanding why some
files/trees are being fetched.
Reviewed By: chadaustin
Differential Revision: D23745752
fbshipit-source-id: b2e145d9e559bde0542adbc5b20ff56ccfc59ece
Summary:
Now that the win directory only contains the mount directory, we can rename it
to be more faithful to its intent. Since this is about ProjectedFS, let's
rename it "prjfs".
Reviewed By: chadaustin
Differential Revision: D23828561
fbshipit-source-id: cb31fe4652fd4356dc2579028d3ae2c7935371a7
Summary:
Since the Stub.h now only contains NOT_IMPLEMENTED, let's move it to its own
header outside of the win directory.
Reviewed By: genevievehelsel
Differential Revision: D23696244
fbshipit-source-id: 2dfc3204707e043ee6c89595668c484e0fa8c0d0
Summary:
This was only used in EdenMount.h, to declare a method that was not compiled on
Windows, let's ifdef that method instead.
Reviewed By: chadaustin
Differential Revision: D23693494
fbshipit-source-id: 1eda62f2ae3a38a30aa0b517911635ef3d3896c2
Summary: This file is not fuse specific, therefore, let's move it to a non-fuse folder.
Reviewed By: chadaustin
Differential Revision: D23464460
fbshipit-source-id: f70e94bb0ecc37bd74798fd230dee2058484f31b
Summary:
Scuba logging that tracks undesired file fetches has some blind spots i.e. a
lot of fetches have null pid and null cmd line. This diff fixes at least part
of the problem.
TreePrefetchContext which is used from TreePrefetchLease didn't logged client
pid at all (in fact, it logged almost nothing). This diff fixes at least one blind spot, however it doesn't look like this is the only one.
Reviewed By: kmancini
Differential Revision: D23417451
fbshipit-source-id: 107884e94c6b40de999328ec2ef78fe22174c1ca
Summary:
This is purely moving code around to be more similar between Unices/Windows.
At some point in the future, I'd like to see just one EdenDispatcher class in
the shared code with as few platform specific code (ie: ifdef) as possible,
this is small step in this direction.
Reviewed By: wez
Differential Revision: D23112345
fbshipit-source-id: feb276040aea106d5951af30c5d445b4cb654e01
Summary:
On Linux, all the notifications are sent to the strace logger, let's do the
same on Windows. Whenever possible, I've tried to use the name of the Linux
syscalls that correspond to the notification.
Reviewed By: chadaustin
Differential Revision: D23105536
fbshipit-source-id: 31c6e545cdec93b0f751682e34c34b894b2890c4
Summary:
We're still reading the entire file at a time, but this paves the way to not do
that.
While the change looks big, a lot of it is just moving code around. The main
gist of it is removing EdenMount::readFile and writing the proper future
combinator in EdenDispatcher::getFileData.
Reviewed By: wez
Differential Revision: D22361748
fbshipit-source-id: 6391a29d25a4c9e61b91952c40c21ad52e728c8b
Summary:
This would enable the enumeration callback to use ProjectedFS asynchronous
completion.
Reviewed By: chadaustin
Differential Revision: D22361747
fbshipit-source-id: b2d31533ee5128e9dd3da7f91d5225331cf8e926
Summary:
This adds logging for data fetches that come from the thrift checkout call to
help debug the cause of unexpected data fetches. (See D22448048 for more
motivation)
Reviewed By: chadaustin
Differential Revision: D22489504
fbshipit-source-id: 3b732a1e5627c2130f561ec0138a1df270e1925d
Summary: This is unused, no need to keep it around.
Reviewed By: wez
Differential Revision: D22361749
fbshipit-source-id: 3c353776437b59c6c7735652f7eb1ce052215e11
Summary:
Futures are intended to be chained together and not synchronously waited one
after the other. While this may be one of the goal, it also paves the way to
enable ProjectedFS asynchronous notification handling.
While doing this, a bunch of code was moved from EdenMount.cpp to the
dispatcher itself, the rationale behind this is to follow what the unix
EdenDispacher with the long term plan to merge the 2 as much as possible.
Reviewed By: wez
Differential Revision: D22361750
fbshipit-source-id: fa679a8b94ff6f8b5a33782fdb6b129ab066c4d8
Summary:
At mount time, EdenFS will try to create the .eden/config file which will
indicate to the client that this is an EdenFS repo, this config also contains
the location of the socket to talk to EdenFS.
On unices, and while the .eden directory is slightly different, the content of
this directory is written onto the overlay, but on Windows, the overlay is the
repository itself. What this means is that creating this config file will end
up triggering a ProjFS callback, which can potentially lead to deadlocks if
we're not careful.
A future change will simply prevent these recursive callbacks from happening
and thus a solution needs to be found for the .eden/config file itself. Since
the file itself is tiny (about 400B), and should only be accessed once[0], the
simple approach to keep it in memory and special case it works perfectly.
[0]: Once a file was read fully by ProjFS, it's present in the overlay and
EdenFS will no longer be requested to provide it.
Reviewed By: chadaustin
Differential Revision: D22310734
fbshipit-source-id: 6b2dba2164496ebd251104d7875b51569be2471f
Summary:
This function is only used in EdenDispatcher, and all of it can be achieved by
simply calling InodeBase::stat method to fetch the file size.
As a bonus, the code now uses future continuation, instead of individual calls
to `get`.
Reviewed By: chadaustin
Differential Revision: D22310733
fbshipit-source-id: c81be7dd114c3a9bebc6b1b5b0887ae3ade30602
Summary:
The WinStore was only used for 2 simple things: checking that a file is
present, and getting its blob. Since both of these are trivial to implement
with EdenMount, there is no reason for WinStore to exist.
Reviewed By: chadaustin
Differential Revision: D22288430
fbshipit-source-id: 90567ac39b5124039becd3599766fcd0cde3e9aa
Summary: This diff defines `Overlaychecker::ProgressCallback` to replace repetitive function type declaration.
Reviewed By: genevievehelsel
Differential Revision: D22243160
fbshipit-source-id: ea05e451817a760b5266879b956eaea48dc8d85e
Summary:
All the unification of the mount code lead to this point, where supporting
unmount is merely just removing the various #ifdef.
Reviewed By: fanzeyi
Differential Revision: D21797078
fbshipit-source-id: 7602eb85e3fa276b72daebda36ae269ec38cde66
Summary:
On unmount, the code relies on the channel (fuse/prjfs) to set a promise, while
not strictly required on Windows as the interface is synchronous, this will
enable unmount to be the same between on all platforms.
Reviewed By: fanzeyi
Differential Revision: D21797081
fbshipit-source-id: c3eac448d2b13ef49db7a7d1f36bb084130ea63a
Summary:
Most of what fuseMount does can be shared between fuse and prjfs, so let's
rename it and start using it on Windows.
Reviewed By: fanzeyi
Differential Revision: D21797079
fbshipit-source-id: b92a36ecc702ee72df8ebdf9534add42bea9b97e
Summary: This diff passes a callback function `progressCallback` from `EdenServer` to make `OverlayChekcer` print fsck messages to stdout.
Reviewed By: fanzeyi
Differential Revision: D21966060
fbshipit-source-id: 317b2c8954c718b51b5295c9f8d7698eb203906e
Summary:
The only reason tests need to set an FsChannel is for calling
cleanupPrjfsCache. For consistency sake, let's do the same as what we do on
unices: verify if we have a valid channel first.
Reviewed By: wez
Differential Revision: D21676316
fbshipit-source-id: 022ca91d32a7ab047a79acccd7d17dfd7906d527
Summary:
We want to use these on Windows too, so let's rename them to not be FUSE
specific.
Reviewed By: wez
Differential Revision: D21676317
fbshipit-source-id: 9f1bde9e0467d8828d4168a69581c8fea54669cd
Summary:
Instead of having 2 different functions for Windows and other unices, let's
only have one. This allows more of the mount code to be shared between both.
Reviewed By: wez
Differential Revision: D21517477
fbshipit-source-id: 185ce864aecdf9a7880a9c705df444ec1e39319a
Summary:
Next step in unifying the mount path, let's make the initialize the same in
Windows and unices. The only difference is now limited to the .eden directory
which we will be able to implement once regular users can create symlinks.
For the takeover code, the #ifdef is pushed down to the actual code that does
it, this allows the rest of the code to not have to bother about Windows vs
other platforms.
Reviewed By: wez
Differential Revision: D21517478
fbshipit-source-id: d40ca2694d23031ff98e319071e610efa306008f
Summary:
In the past, if an EdenMount got stuck while shutting down (inodes still referenced), the EdenMount would already be removed from the server's `mounPoint's` map while stuck. This resulted in `eden list` showing that the mount was not mounted, and then if we tried to remount the mount, we'd get a confusing error message about the not being able to grab the overlay lock. This also made it so if we tried to attach with gdb, or if we core dumped and tried to look at the data, we wouldn't be able to examine the object.
Instead, we should only remove the mount from the `mounPoints_` map after it has completely been shutdown. This will allow us to get better signal on the state of the mount (it will be in the `SHUTTING_DOWN` state in `eden list` instead of `(not mounted)`) and it will allow us to better debug these situations (where a mount is stuck shutting down) since we can get more information about the mount that is in this state.
Reviewed By: simpkins
Differential Revision: D21211544
fbshipit-source-id: 97ff11df67d0a80f47cf6b9f563876e17bd76622
Summary:
This diff updates the EdenDispatcher to fetch/update the FS info in EdenMount which is backed by InodeTree. This helps to store the FS state in the Inode structure, which are used for source control status and update operations.
Also added a custom formatter definition for RelativePathPiece to make it easy to log relative paths.
Reviewed By: simpkins
Differential Revision: D20480861
fbshipit-source-id: b4bf1da3eeebeaee46a4a187eea9193302182068
Summary: These API will handle the file system requests for each mount.
Reviewed By: simpkins
Differential Revision: D20480868
fbshipit-source-id: 84149af4449d258149fce97bb227d45d8d3d0e3a
Summary: The diff merge the Windows version of EdenMount into POSIX version. We don't start using the new all platform version of EdenMount in this diff. This switch needs few more changes and is done later in this diff stack.
Reviewed By: simpkins
Differential Revision: D20480878
fbshipit-source-id: 42bfaee8769beb7a2ac45cfcff5dda2a52a7dcb0
Summary: This diff ports TreeInode, FileInode, InodeMap and related classes to Windows. We don't build or test it here, there are more dependcies we need to port. The built script and the test are part of other diffs in this stack.
Reviewed By: simpkins
Differential Revision: D19956266
fbshipit-source-id: 9eb754233bca3d5a336f465c2400512a8593ca4f
Summary: Pass a valid ObjectFetchContext down into certain untracked requests.
Reviewed By: chadaustin
Differential Revision: D20243575
fbshipit-source-id: e7112c3bab1265803a26130c4d72905c25f2e729
Summary:
Add a fetch context interface to ObjectStore that allows tracing cache
hits, backing store fetches, and fetch durations in the context of a
diff or checkout operation.
Reviewed By: simpkins
Differential Revision: D19135625
fbshipit-source-id: d0d8f134b1c89f7ba4971a404a46a69a1704ba5c
Summary:
Add a config option to restrict the number of tree prefetches that can run in
parallel. Without this applications that recursively walk a checkout tree can
end up spawning a huge number of asynchronous prefetch operations. These
prefetch operations are quite expensive today, as we currently have to fetch
the full file contents in order to retrieve the file sizes. A recursive
directory walk can end up building up a huge backlog of in-progress tree
prefetches. This will slow down the directory walk, and it can take tens of
minutes to finish processing the prefetch backlog even after the directory
walk is aborted.
Reviewed By: chadaustin
Differential Revision: D19136685
fbshipit-source-id: cdc7a570d016fd7ca81a639cef83380b197acdfc
Summary:
EdenMount offered both getRootTree and getRootTreeFuture. Replace the
former with the latter and adjust remaining callers.
Reviewed By: genevievehelsel
Differential Revision: D18666585
fbshipit-source-id: 961361a1fb545e48db0ece970bb5e6ffbca8efef
Summary: Adds a helper function that can take a path to an Inode and load a file by simply getting the Inode then calling the existing `loadFile()` function
Reviewed By: simpkins
Differential Revision: D18647090
fbshipit-source-id: 89c25bef10a7a0d0ffd660293f85479ac0e9f9a9
Summary: Move the bulk of the gitignore loading to EdenMount, this is needed later in the stack to pass this function as a callback through the DiffContext to enable gitignore loading from the source control tree differ.
Reviewed By: simpkins
Differential Revision: D18565265
fbshipit-source-id: 062349d81bbfb78235ccb34d6ec72ad445a6e2b0
Summary:
Record the durations of each phase of checkout so it can be logged to
scribe in a future diff.
Reviewed By: simpkins
Differential Revision: D18632258
fbshipit-source-id: e447976af22cf25cffd640e79b95da2989b07448
Summary:
We're about to return more information from checkout, so
introduce a CheckoutResult struct.
Reviewed By: genevievehelsel
Differential Revision: D18630820
fbshipit-source-id: 78b68ef3551ca8c9e9968631de554084c333b4b9
Summary: changes `overlay_` to be stored as a `std::shared_ptr` in order to safely create and pass `std::weak_ptr`s of the overlay to OverlayFiles that are being created.
Reviewed By: chadaustin
Differential Revision: D18302732
fbshipit-source-id: 495b613914771714ba8a8d1beb1c7a0cd72b33d4
Summary:
add an enforceParent flag to signal if the diff call should only be run against its parent commit. setup work for `getScmStatusV2`
The only callsite for this version of diff() is in the testcase and in `async_tm_getScmStatus`, in which both cases they set `enforceCurrentParent = false`.
Reviewed By: chadaustin
Differential Revision: D17911708
fbshipit-source-id: 361c7586e1452544172cf9e79a0cdf0888154096
Summary: This makes a `diff()` function definition private inside EdenMount in order to solidify use of default parameters and to consolidate callsites.
Reviewed By: chadaustin
Differential Revision: D17962075
fbshipit-source-id: 2a6d4213f98828f4ebe0ba6db43211a96d1b2bc8
Summary: Keep track of `ResponseChannelRequest` in diff() in order to short circut function if client request is no longer active.
Reviewed By: wez, simpkins
Differential Revision: D17531102
fbshipit-source-id: 662333e5fbc2ed9b3b32b619b8d15db618ad06ec
