Summary:
We are unifying C++ APIs for accessing optional and unqualified fields:
https://fb.workplace.com/groups/1730279463893632/permalink/2541675446087359/.
This diff migrates code from accessing data members generated from unqualified
Thrift fields directly to the `field_ref` API, i.e. replacing
```
thrift_obj.field
```
with
```
*thrift_obj.field_ref()
```
The `_ref` suffixes will be removed in the future once data members are private
and names can be reclaimed.
The output of this codemod has been reviewed in D20039637.
The new API is documented in
https://our.intern.facebook.com/intern/wiki/Thrift/FieldAccess/.
drop-conflicts
Reviewed By: yfeldblum
Differential Revision: D22631599
fbshipit-source-id: 9bfcaeb636f34a32fd871c7cd6a2db4a7ace30bf
Summary:
The WinStore was only used for 2 simple things: checking that a file is
present, and getting its blob. Since both of these are trivial to implement
with EdenMount, there is no reason for WinStore to exist.
Reviewed By: chadaustin
Differential Revision: D22288430
fbshipit-source-id: 90567ac39b5124039becd3599766fcd0cde3e9aa
Summary:
This will allow adding custom MetadataImporters in different eden builds.
DefaultMetadataImporter provides a no-op version of the interface to be
used by default.
Reviewed By: chadaustin
Differential Revision: D21960834
fbshipit-source-id: aec8a3627ab1223f74466b92a0ebe3290b67b7ed
Summary: This diff defines `Overlaychecker::ProgressCallback` to replace repetitive function type declaration.
Reviewed By: genevievehelsel
Differential Revision: D22243160
fbshipit-source-id: ea05e451817a760b5266879b956eaea48dc8d85e
Summary: This diff made fetch threshold configurable, so we can change it later as repository size grows.
Reviewed By: fanzeyi
Differential Revision: D22337850
fbshipit-source-id: 4b46420cb4e7164a3f1080279d67fa5f90549cd8
Summary: This diff updated `ObjectStore` to send a `FetchHeavy` event to Scuba when the number of fetching requests of a process has reached 2000.
Reviewed By: fanzeyi
Differential Revision: D22292104
fbshipit-source-id: b7ac48412868216ea960c8681a5fb71c587952bc
Summary:
Recently the server team added an un-used directory to test that eden would not
fetch these as a test for the upcoming repo merge. They saw that these files
were fetched a non trivial number of times. We want to know why eden is causing
these fetches.
This adds the pid and interface through which the client is interacting with eden to
ObjectFetchContext for this purpose. This information will be logged in later
changes.
note: currently this is only for fetches through Fuse (thrift interface to follow).
Reviewed By: chadaustin
Differential Revision: D22050919
fbshipit-source-id: 49b93257a0e6d910f48b1e8ec6471527e056d22a
Summary:
This passes ObjectFetchContext into the backing store to prepare for adding
logging for the cause of server fetches.
In following changes I will add logging in the HgQueuedBackingStore.
Ultimately we will want to move this logging to be closer to the data fetching
(in HgDatapackStore and HgImporter), but I plan to temporarily add logging to
the HgQueuedBackingStore to simplify so that we can more quickly roll out.
Reviewed By: chadaustin
Differential Revision: D22022992
fbshipit-source-id: ccb428458cbf7a1e33aaf9be9d0d766c45acedb3
Summary:
In following changes I will be threading ObjectFetchContext into the backing
store importing process, since this will start to be used more outside of the
ObjectStore, I am moving this class into its own files.
Reviewed By: chadaustin
Differential Revision: D22022488
fbshipit-source-id: 1a291fea6e0fd56855936962363dfc9f6de8533d
Summary:
This diff makes `lookup` to use `RequestData` as `ObjectFetchContext` in `getattr` calls.
This will make sure we correctly record backing store fetches in `eden top`
Reviewed By: chadaustin
Differential Revision: D21780969
fbshipit-source-id: 468e2fadcebf4a00477bc5de434e6c658b99d1ce
Summary: This diff makes `InodeBase::stat()` to be able to accept `ObjectFetchContext` as a parameter.
Reviewed By: chadaustin
Differential Revision: D21780883
fbshipit-source-id: 9b1db2e2268cb98663bcf902ea61897da593ea05
Summary: This diff passes a callback function `progressCallback` from `EdenServer` to make `OverlayChekcer` print fsck messages to stdout.
Reviewed By: fanzeyi
Differential Revision: D21966060
fbshipit-source-id: 317b2c8954c718b51b5295c9f8d7698eb203906e
Summary:
The remotenames extension was enabled everywhere in production. It is
unsupported if remotenames is disabled.
Ideally the cpp tests should use the in-repo configuration instead of
maintaining its own hardcoded hgrc. For now I just did the minimal change to
unblock tests.
Reviewed By: fanzeyi
Differential Revision: D21986274
fbshipit-source-id: b632da074baf4026f331381ae8f2f8479a34e1b3
Summary:
The only reason tests need to set an FsChannel is for calling
cleanupPrjfsCache. For consistency sake, let's do the same as what we do on
unices: verify if we have a valid channel first.
Reviewed By: wez
Differential Revision: D21676316
fbshipit-source-id: 022ca91d32a7ab047a79acccd7d17dfd7906d527
Summary:
This moves it out of EdenServer and into EdenMount directly. This is closer
to what is happening in the Linux code and thus better.
Reviewed By: fanzeyi
Differential Revision: D21663501
fbshipit-source-id: 84420fcb37586cc2695568f0167b39f91c9bfb63
Summary:
Instead of having 2 different functions for Windows and other unices, let's
only have one. This allows more of the mount code to be shared between both.
Reviewed By: wez
Differential Revision: D21517477
fbshipit-source-id: 185ce864aecdf9a7880a9c705df444ec1e39319a
Summary: The thrift team is looking to retire `stopWorkers()` and `stopListening()`, so this refactors our code to stop using both of those methods.
Differential Revision: D21851442
fbshipit-source-id: 6d63d435cbd4a51c855bf71a3b2165e35431a4d7
Summary:
Next step in unifying the mount path, let's make the initialize the same in
Windows and unices. The only difference is now limited to the .eden directory
which we will be able to implement once regular users can create symlinks.
For the takeover code, the #ifdef is pushed down to the actual code that does
it, this allows the rest of the code to not have to bother about Windows vs
other platforms.
Reviewed By: wez
Differential Revision: D21517478
fbshipit-source-id: d40ca2694d23031ff98e319071e610efa306008f
Summary:
Fairly straightforward, one change to the test was necessary as reading an
unlinked file cannot be done on Windows.
Reviewed By: wez
Differential Revision: D21312681
fbshipit-source-id: c8cca5eeca7825983176ef618007c514b8e02140
Summary:
This brings it closer to folly::writeFile which should help in avoiding ifdef
whenever we want to use it.
Reviewed By: wez
Differential Revision: D21319020
fbshipit-source-id: 80fbf7fba671b18b5ef68375910e1a2a8869f590
Summary:
While EdenFS does not use a separate privhelper process on Windows, it still
defines a stub PrivHelper class. However, this class was previously defined
in a separate win/utils/Stub.h header file, which led to awkward `#ifdef`s to
include the correct platform-specific header file.
This diff moves the definition of the dummy PrivHelper class in Windows into
the same `PrivHelper.h` header file used on POSIX platforms. This results in
a few more `ifdef`s in the PrivHelper files, but fewer `ifdef`s in the calling
code, and will make it easier to start unifying more of the `EdenMain` logic
on Windows and non-Windows platforms.
Reviewed By: xavierd
Differential Revision: D21332568
fbshipit-source-id: c63bf2b4a8b7e767d7db7dcda28675f735c23bf8
Summary:
Move the `UserInfo` code from `fuse/privhelper` to `utils`, and also unify the
POSIX and Windows implementations of this class.
This code was originally put in the `fuse/privhelper` directory since it was
written at the same time as the privhelper. However, it's really a
lower-level library that doesn't depend on FUSE or any of the other code in
the `fuse/` subdirectory.
Reviewed By: wez
Differential Revision: D21296594
fbshipit-source-id: f58682f6ce86bba0328472c491bb4c0dc3370319
Summary:
Enable the unit tests under eden/fs/utils on Windows.
This does comment out a few tests related to `AbsolutePath` that are broken on
Windows. The AbsolutePath constructor does not enforce that the input path is
actually absolute. Additionally the `canonicalPath()` function ends up doing
weird things and returning paths that start with `/` followed by a drive
letter (e.g., `/C:/foo`). These issues should ideally be addressed in
subsequent diffs.
Reviewed By: xavierd
Differential Revision: D21239886
fbshipit-source-id: ef08d62353ba83b96d9fd79bd4636f4a0f961373
Summary:
The operation originally wanted to operate on the fuse `Attr`
structure which we don't have on Windows, so I repurposed the
`InodeBase::getattr` into `InodeBase::stat` and moved the conversion
of `struct stat` to `Dispatcher::Attr` to the `EdenDispatcher::getattr`
method (and a couple of other adhoc places that were doing a similar
conversion).
Reviewed By: chadaustin
Differential Revision: D20562459
fbshipit-source-id: 6b538110038352e9b5590fcb5ff5c33fe84ac1d8
Summary:
While I had to disable a bunch of them at first, this allow them to compile
properly and run. Future diffs will attempt to enable the disabled ones.
Reviewed By: pkaush
Differential Revision: D21188118
fbshipit-source-id: 154fec49c76563b0856fa36e78b2bbd09f0f2381
Summary: Now we are building the merged EdenMount on Windows, so we can also start using the new TestMount code.
Reviewed By: simpkins
Differential Revision: D20966556
fbshipit-source-id: a726b13250d6c29872349e1a2f562c719420f3f1
Summary: This diff is merging the Windows version of MountTest with the POSIX version. The merged MountTest is build and tested later in this stack with the Merged EdenMount.
Reviewed By: simpkins
Differential Revision: D20480864
fbshipit-source-id: 65e9402f1b03c81166835a6a605053a1bf011ddc
Summary:
This is a rough pass that resolves a linker issue on MSVC by
switching to inline static member functions.
Reviewed By: chadaustin
Differential Revision: D20529163
fbshipit-source-id: 578ed440758c685091d3e039e261638e027db17a
Summary: This diff bumps the open call from FUSE to High priority (which is higher than any other blob open request atm). This has shown improvement on the user experience of EdenFS when it's importing many other things from other channels (thrift, etc.)
Reviewed By: chadaustin
Differential Revision: D20287389
fbshipit-source-id: 319bc44ef8be5c904d7cf0db7cc2f8be28b4760a
Summary: This diff adds `Priority` added in the previous diff to the `BackingStore` interface with the default value set to `Priority::Normal`.
Reviewed By: chadaustin
Differential Revision: D20197071
fbshipit-source-id: a92f1b49bb82e3478042e5e3b79b047d834755ea
Summary:
Migration from Future-returning executor-erasing collectX forms to
SemiFuture-returning forms, that are less risky in particular with coroutines.
Earlier diffs added SemiFuture and Unsafe versions. This codemod migrates
collect versions to the Unsafe versions to allow the basic collect versions to
be made safe.
Reviewed By: simpkins
Differential Revision: D20331206
fbshipit-source-id: efc8dff487d45f7d53ee55e8c4696bd3eed0e6da
Summary:
Add a fetch context interface to ObjectStore that allows tracing cache
hits, backing store fetches, and fetch durations in the context of a
diff or checkout operation.
Reviewed By: simpkins
Differential Revision: D19135625
fbshipit-source-id: d0d8f134b1c89f7ba4971a404a46a69a1704ba5c
Summary: Easier to zero initialize structs with braces, which defines that even padding is zeroed.
Reviewed By: wez
Differential Revision: D19655675
fbshipit-source-id: 2fd12383324029646707e93008cf9ad34e9f1dce
Summary:
This diff turns the return type of `BackingStore::getBlob` from `folly::Future` into `folly::SemiFuture` to prevent executor leaks.
This also enable us to remove the need of holding `serverThreadPool` from backing stores.
----
**Changes**
* `ObjectStore` now needs to hold a `folly::Executor::KeepAlive` that is used to turn `SemiFuture`s it gets from backing stores into `Future`.
* Signature changes of the implementations of `BackingStore` class.
* For tests, I chose to use `QueuedImmediateExecutor` in place of `UnboundedQueueExecutor` as it will basically execute tasks inline. I'm concerned introducing thread pool executor in tests may turn tests flaky.
Reviewed By: wez
Differential Revision: D18669664
fbshipit-source-id: 0cae89f365dcf8b345b49d64469a530cf25d4ac5
Summary:
EdenMount offered both getRootTree and getRootTreeFuture. Replace the
former with the latter and adjust remaining callers.
Reviewed By: genevievehelsel
Differential Revision: D18666585
fbshipit-source-id: 961361a1fb545e48db0ece970bb5e6ffbca8efef
Summary:
The last remaining user of ObjectStore::getBlobMetadata was a debug
Thrift call. Remove it and update the Thrift call to use getBlobSize
and getBlobSha1.
Reviewed By: pkaush
Differential Revision: D18663376
fbshipit-source-id: 86baefc9004a07aac4ddf5849870431be04c75f2
Summary: This diff contains the tests for hg status. This is not comprehensive list of tests but the tests that we have working at this time. More tests will be added as we test and fix issues.
Reviewed By: simpkins
Differential Revision: D18454851
fbshipit-source-id: 80e3ff4d24873016dc420f202fbfe53fcffc24f2
Summary: Adds a function which takes both the manifestID and the commitID to get a Tree. This will be used in `checkOutRevision()` and this allows us to skip looking up the manifestID since the caller can just pass it in themselves.
Reviewed By: wez
Differential Revision: D18719405
fbshipit-source-id: 919f0a7c84bff4a2f0bc20110c45bd272f9e9107
Summary: Adds a helper function that can take a path to an Inode and load a file by simply getting the Inode then calling the existing `loadFile()` function
Reviewed By: simpkins
Differential Revision: D18647090
fbshipit-source-id: 89c25bef10a7a0d0ffd660293f85479ac0e9f9a9
Summary:
Allocate and hook up a StructuredLogger at startup if the EdenConfig
has the appropriate values set.
Reviewed By: simpkins
Differential Revision: D18071821
fbshipit-source-id: 3996b6644bbf0c16bb3b9950d57a79d4619a1656
Summary:
This saves the version number (like 20191014-154333) in the EdenServer. This will be returned to the user in a getScmStatusV2 call later up the stack. This is useful for being able to warn the user if EdenFS is running on an old or known-bad version.
I ended up splitting up `getEdenfsVersion()` and `getEdenfsBuildName()` because I don't think we need to store "edenfs " with every version number in the server / return this from the thrift call of `getScmStatusV2()`. Not sure of the use case of having "edenfs " prepended to the version number returned from the current `getEdenfsBuildName()` so didn't want to change that, but if it seems fine we could remove that prepended edenfs (or if it actually makes sense to return the string with that prepended from the ScmStatus call).
Reviewed By: chadaustin
Differential Revision: D17919528
fbshipit-source-id: 5abdf7851371c4614da862657350a8cc263eb014
Summary:
When we implemented FUSE_NO_OPEN_SUPPORT and FUSE_NO_OPENDIR_SUPPORT,
we forgot to remove the FileHandle and FileHandleBase code.
Reviewed By: pkaush
Differential Revision: D17991710
fbshipit-source-id: dfeb26d512f017cef7710929ccff1f6940cf8641
