Summary: The InMemory overlay does not persist any of its state so on restart it can leave the repo state in a weird state. This change allows us to rebuild the overlay without having a root inode.
Reviewed By: kmancini
Differential Revision: D49441198
fbshipit-source-id: b703a95147fff595ed963bbd6de2a2fb835100ff
Summary: InodeMap, in subsequent diffs, will report metrics when looking up inodes (hits, misses, errors). Adding EdenStats to it initalization to enable this.
Reviewed By: chadaustin
Differential Revision: D49477891
fbshipit-source-id: 86f7c357d5cf011897d1ccb57e42f80f55890ba0
Summary: In preparation for merging ProcessSimpleName and ProcessName functionality into ProcessNameCache renaming now The new cache will cache ProcessInfo object which will include both names, and the parent pid.
Reviewed By: genevievehelsel
Differential Revision: D48855222
fbshipit-source-id: 4cb10df7b6cc32efc0d655771d01d58e7ba57dd5
Summary:
EdenFS behavior appears to be slightly different from Mercurial (and I presume
git) when it comes to files ignored by a gitignore but known to source control.
In particular, Mercurial appear to not look at the gitignore when comparing 2
commits, or when comparing the working copy to another commit.
For instance, if we try to manually reproduce the
diff_added_tree_top_level_not_materialized_with_gitignore, here is what we see:
```
% hg status --rev 4694c115b9bba800b6d2cf47e10f556f3e8e5b7e
A root/src/foo/r.txt
% eden debug thrift getScmStatus "/data/users/xavierd/fbsource" False 4694c115b9bba800b6d2cf47e10f556f3e8e5b7e
ScmStatus(
entries={},
errors={})
```
That is: EdenFS doesn't report any changes, while Mercurial does.
Fixing this issue is just a matter of removing all the gitignore handling code
from store/Diff.cpp and fixing the callers.
Reviewed By: quark-zju
Differential Revision: D48705684
fbshipit-source-id: 5526e28fe3c7b3a329119a245c09b1fc11372e97
Summary: It looks like this was inadvertently added in D47738790
Reviewed By: genevievehelsel
Differential Revision: D48844923
fbshipit-source-id: b2ce55500a29cbe8dc360175671199672c18cbcb
Summary:
# The problem
Users were running into issues with files not showing up as the correct type. For example, a user expected file `foo/bar` to be EXECUTABLE, but in reality it was REGULAR. This happened because of the following scenario:
Commit A has a file `foo/bar` which is a REGULAR file
Commit B updates foo/bar to EXECUTABLE
Any update from Commit A to after Commit B would cause foo/bar to exist in the working copy as a REGULAR file instead of an EXECUTABLE file.
Users would not notice this issue, the file would become materialized, and then they would accidentally commit the change that caused `foo/bar` to become a REGULAR file again (Commit C).
This commit would add even more opportunities for people to update from A -> B or B -> C or some combination, and this would add even more chances for the file's mode to become changed.
# The bug and solution
During checkout, we fail to check if a file's mode has changed. We do a check to see if the contents of the file change, but we don't check to see if the file mode has changed. This caused us to fail to update modes across commits that changed file modes.
The solution is to avoid returning early when the modes of oldScmEntry and newScmEntry differ while processing a checkout entry.
Reviewed By: xavierd
Differential Revision: D48667112
fbshipit-source-id: cb424f11cb95df46239ee49d4989baa7c869bc94
Summary:
In a world where Tree does not know its own ID, getRootTree must
return the root tree's ObjectId, which is independent of the passed-in
RootId.
Reviewed By: kmancini
Differential Revision: D47738790
fbshipit-source-id: 2ad19b131225ef8e6c9fdfb3e8f20bdb79d48ac7
Summary:
Similar to manual scripts ran on Linux and macOS, this powershell script measures duration of `hg update` commands based on several inputs.
Defaults params will remove any existing test clone (c:\open\test\fbsource) and cone anew. It will then crawl the file system starting from //admarket and working its way downward until it reaches the $TotalBytes amount of file data (default 1MB).
Finally, it alternates checking out an old commit (~6mo, 2.6M commits ago as of 8/15/2023) and a new commit (8/15/2023) and records the time it takes. It does this for $Iterations times.
Reviewed By: genevievehelsel
Differential Revision: D48438882
fbshipit-source-id: 93ef982639cbf0c769032315e9deece7db2127c2
Summary: Previously restarting EdenFS on Windows could get the wrong type for symlinks after a restart, which this diff fixes.
Reviewed By: xavierd
Differential Revision: D48306746
fbshipit-source-id: c4a5016dcb824a3b27cd34e68e972267c54a2222
Summary:
InMemory overlay did not override the method `maintenance`. This posed no issues in testing, but was discovered while using on Windows machine.
This change impelments a stub version of the method.
Reviewed By: xavierd
Differential Revision: D48284745
fbshipit-source-id: 7eca6294f7a6f307fc08472ef7d6fabd7b7e4f17
Summary: Portability.h does not need to include SysTypes.h. This avoids bringing in basetsd.h and some non-portable typedefs in typical code that just wants to use Folly.
Reviewed By: Gownta
Differential Revision: D42044843
fbshipit-source-id: 27ba1c5298d936c85a03f0b5159259f8f31fc693
Summary:
Cleaning up some code where I learned about OptionSet features - containsAllOf.
Looking good now. :)
Reviewed By: genevievehelsel
Differential Revision: D48214786
fbshipit-source-id: 4ba4d38e0fbc72ae163da34889546192cc0ba0c6
Summary:
To enable consistent and simple configuration of a clone configuration and preferences around cloning:
* Refactored code to simplify InodeCatalogTypes (aka Overlay types) to three: Legacy, Sqlite, and InMemory
* Added new InodeCatalogOptions to support different variants for each type (currently only Sqlite has options)
This will allow CLI to read the users preferences from EdenConfig for overlay type, to control the defaults via Chef + GK, and to persist the configuration in CheckoutConfig. This change better unifies the settings that already existed in EdenConfig and CheckoutConfig.
Reviewed By: genevievehelsel, kmancini
Differential Revision: D48014211
fbshipit-source-id: 56e8f00d1680c717e736c84e5b5c24fb02e36b5d
Summary: We now know that these events are occurring quite frequently. Let's log some additional info here so we can tell if the InodeMetadataTable entry is completely corrupted, or if just a small part of the entry is incorrect.
Reviewed By: xavierd
Differential Revision: D48164675
fbshipit-source-id: 8bb0991a8a916278c4add1f7feab7cd2348e4b0a
Summary: Previously running `ls` on dirs other than the root directory would fail if there was a symlink directory there, which this diff fixes.
Reviewed By: xavierd
Differential Revision: D48081347
fbshipit-source-id: 959ba37edd11dc81b63f91a5cafe3b8868d3897f
Summary:
* Add setting to EdenConfig to specify which inode catalog to use - default to current behavior.
* Add setting to CheckoutConfig to specify which inode catalog to use - default to current behavior.
Reviewed By: genevievehelsel
Differential Revision: D47601732
fbshipit-source-id: ca137d5f38bb3253dd66a5f476a07602f09dc9ef
Summary:
This allows hg on EdenFS to be able to create symlinks with absolute paths on Windows as well as check out commits with absolute path symlinks.
There are a few caveats here:
- Symlinks with absolute paths will only be usable on the same type of OS (Windows vs. POSIX) they were created.
- Checking out commits with absolute paths created on POSIX (i.e., starting with `/`) aren't usable on Windows
- Symlinks with absolute paths created on Windows will keep the same drive letter across different checkouts.
- E.g., if the root of the EdenFS checkout is somewhere at `C:\` and there is a symlink pointing to somewhere like `D:\foo\bar.txt`, and then that same commit is checked out in a different Windows machine with that same repo mounted at `E:\`, that symlink will still point to `D:\foo\bar.txt`
If there seems to be some UNC wonkiness in this commit, it is because UNC usage on symlinks is quite inconsistent on Windows:
- Things that use UNC for absolute path symlinks:
- Windows' `mklink` command (creates symlinks with UNC paths)
- Python's `os.symlink` (same as above)
- Hg commits
- EdenFS (when stored as Inodes; some functions like `realpath`)
- Things that don't use UNC for absolute path symlinks:
- `PrjWritePlaceholderInfo2`
- `PrjFillDirEntryBuffer2`
- `std::filesystem::read_symlink`
Reviewed By: xavierd
Differential Revision: D47227590
fbshipit-source-id: b92c8462ad3f13c2724b76ee61f8c095695027cd
Summary:
This diff makes it possible to check out commits with directory symlinks on Windows. Previously checking out commits with symlinks would make these symlinks appear as symlinks to regular files instead of symlinks to directories, making them unusable.
In order to solve this, each time a new symlink is to be created by ProjFS, we check the type of the directory with the equivalent of the [`stat()` function](https://pubs.opengroup.org/onlinepubs/000095399/functions/stat.html). This recursively reads symlinks until the target directory is a symlink or a regular file. Note that this populates all symlinks in the chain and materializes the final file / directory.
For avoiding cycles, paths are marked as "visited", so that if a cycle is found, we mark that entire cycle as a cycle of symlinks pointing to regular files. This was decided arbitrarily; the type of the cycle doesn't matter as symlinks cycles are useless anyways.
Reviewed By: xavierd
Differential Revision: D47144225
fbshipit-source-id: d521ab0d561cfe452675ae7249476879a1d3bedf
Summary:
Due to using optional fields in a struct, it wasn't easy to understand which
fields were supposed to be set in which state. To make it more clear, we can
use a `std::variant`.
Reviewed By: kmancini
Differential Revision: D47606369
fbshipit-source-id: 6fa136c6a4da1c8083aa44f71f3f372faabb0e8f
Summary:
Some tools seem to query EdenFS directly about the type of certain files. If queried directly, EdenFS might answer that some files are symlinks regardless of whether symlink support for Windows is enabled or not. This adds some additional gating to make sure that EdenFS only answers that files are symlinks if symlink support is enabled for a checkout.
Split between this diff and D47829752 for clarity
Reviewed By: xavierd
Differential Revision: D47326227
fbshipit-source-id: 6e60b3d434ba45dd529286436526350ee0ac9dcf
Summary:
Some tools seem to query EdenFS directly about the type of certain files. If queried directly, EdenFS might answer that some files are symlinks regardless of whether symlink support for Windows is enabled or not. This adds some additional gating to make sure that EdenFS only answers that files are symlinks if symlink support is enabled for a checkout.
Split between this diff and D47326227 for clarity
Reviewed By: xavierd
Differential Revision: D47829752
fbshipit-source-id: e66c7d774b5fd037fd26538f945e2ff077f3975d
Summary:
In preparation for removing the blob ID from Blob, update all of the
call sites that previously read it.
Reviewed By: xavierd
Differential Revision: D39188082
fbshipit-source-id: 7bf0b652a929cb24957e617e499fc8228807f99d
Summary: Moved InodeCatalogType to a namespace level type. This was done so that we can deserialize from a config (TOML) file or a command line.
Reviewed By: xavierd
Differential Revision: D47648635
fbshipit-source-id: 7feeda41f8a40091a26f161d05eec17f7e12dfcf
Summary:
* Refactored some SqliteInodeCatalog scanning logic to enable other InodeCatalogs to work on Windows
* Moved some type alias from OverlayChecker to InodeCatalog
* Implemented MemInodeCatalog::scanLocalChanges from SqliteInodeCatalog
Reviewed By: xavierd
Differential Revision: D47601733
fbshipit-source-id: 139f1d963594329107e90896efdf4088e41548ad
Summary:
* Initial implementation of `MemInodeCatalog` using simple sychronized map for underlying storage and atomic for inode numbers.
* Integrated minimally with `Overaly` and `TestOverlay`
Reviewed By: xavierd
Differential Revision: D47601735
fbshipit-source-id: 2ff9a6b25c4d2781f8807e8406e4cd2cfb949f86
Summary:
Adding an in-memory `InodeCatalog`. This will be used to see if we can succuessfully speed up various scm operations on Windows (e.g., `hg update` and `hg rebase`). It might also be useful for Sandcastle builds where persistence is not the aim.
This first commit introduces this new class - `MemInodeCatalog`. Subsequent commits will add implementation, tests, and integration with `Overlay`.
Reviewed By: xavierd
Differential Revision: D47601736
fbshipit-source-id: 751c79b7d24fc6dae1253ed3cb621e5fc020568f
Summary:
This has been enabled for many months everywhere, let's just remove the
fallback code.
Reviewed By: kmancini
Differential Revision: D47571146
fbshipit-source-id: 96d396651041740d0da40be35c6dd9cffd5b86de
Summary:
When EdenFS is interrupted mid-checkout either due to a crash, or simply via
`eden stop`, the information is recorded on disk to allow the next checkout to
be resumed. There is however a subtle race in this case if 2 FORCE checkout are
started at the same time. They both will detect that a previous checkout was
interrupted, and both will start a checkout operation. This is not the desired
behavior as one of the checkout should stop and warn that an existing checkout
is ongoing instead!
The fix is trivial: the checkoutPid should simply be updated when checkout is
starting.
Reviewed By: kmancini
Differential Revision: D46037833
fbshipit-source-id: a711a1575dcbc671185fde7d4114ee1633e53408
Summary:
When creating a new symlink manually on Windows, `PrjFS` used to categorize the newly created file as a regular file instead of a symlink, which this diff fixes.
Since symlink creation on Windows is racy, this diff double checks the type of file for regular files in case they are symlinks, in a similar way to what is done for directories.
Reviewed By: xavierd
Differential Revision: D46860405
fbshipit-source-id: 87aa8b3909553baabe8480f69e996682674c69e7
Summary: With the changes made in D44218035, files would not be listed as symlinks when running commands such as `ls / Get-ChildItem` until those files were populated. This diff changes that behavior so that files are recognized as symlinks from the moment they are listed.
Reviewed By: xavierd
Differential Revision: D45276694
fbshipit-source-id: 82a83c80ec5d4474fdf8d226de2783292b520b00
Summary: Sometimes on repos without symlinks enabled files that were supposed to be symlinks would appear as modified. This was caused by using the actual type of the files regardless of whether symlinks were enabled for the repo or not.
Reviewed By: xavierd
Differential Revision: D47597262
fbshipit-source-id: 7e6fc75db049c00bc4e28d90e6329a3b5dd7e14f
Summary:
The quick_check pragma is documented in
https://www.sqlite.org/pragma.html#pragma_quick_check as performing similar
checks to the integrity_check pragma, but omitting a couple of check allows it
to be much faster. We've seen engineers getting many minutes hangs at startup
due to this, let's switch the the faster pragma to help them.
Reviewed By: genevievehelsel
Differential Revision: D47531252
fbshipit-source-id: fa887a24f1e0c3ba5ce75db46d0b962fb687e4ae
Summary: When looking at the getdeps builds I noticed some simple warnings.
Reviewed By: xavierd
Differential Revision: D47490053
fbshipit-source-id: 4d720aded376d34a63e7e3c39d50db682f0bdc17
Summary:
off_t is not suitable on Windows because it is a 32-bit signed
type. Introduce a new FileOffset type for use in EdenFS.
Reviewed By: kmancini
Differential Revision: D47406784
fbshipit-source-id: 9cc9585b4b118c54a1ca148c2ba7e7ccfb6f278c
Summary:
Partially enables symlink support on Windows by making symlinks appear on Windows as actual symlinks as opposed as regular files containing only the place where the symlink would point in other systems.
Creating new commits with symlinks also works (after editing `fscap.py` on hg's side as well as the requirements file for the current directory for enabling symlinks on Windows EdenFS) when the symlinks are in the same directory.
Reviewed By: xavierd
Differential Revision: D44218035
fbshipit-source-id: 0e3094dc5a13cabef1cd24f8fe18cc73ca40d4a8
Summary:
The methods mentioned in the title changed a bit on Windows, now allowing them to respond that a TreeEntry can be a symlink.
In order to gate this change, on Windows now there is a helper method that is used for reverting the changes for `TreeEntry:: getType` to its previous behavior, in order to be able to tell whether symlinks are enabled for the current Eden checkout or not. Where possible, (which is in most places) the config for whether symlinks are enabled or not on the Eden checkout is passed down.
The non-gated changes are kept since in those cases we actually want to register that a TreeEntry is actually a symlink, even when symlinks are "not enabled".
Originally this was intended to be part of D44218035, but was split for clarity.
Reviewed By: xavierd
Differential Revision: D47326228
fbshipit-source-id: be6cfae6626bf3a32aa119d25bf8b5fe6a549898
Summary:
clang-tidy had some automated suggestions for our code. Apply the ones
that make sense.
Some of them didn't, like removal of all uses of `volatile`. I
manually reverted those changes.
Reviewed By: genevievehelsel
Differential Revision: D41051052
fbshipit-source-id: 3fe22a91e929d3bb8e6346126c2c7bf9f027eb32
Summary:
Now that ProcessId exists, we should use it instead of pid_t in
ObjectFetchContext.
Reviewed By: genevievehelsel
Differential Revision: D42037216
fbshipit-source-id: 34cd89f78be35a15d73b26edc840e917fd642723
Summary:
In preparation for unifying unmount() in FsChannel, return SemiFuture
from EdenMount::unmount.
Reviewed By: kmancini
Differential Revision: D45300725
fbshipit-source-id: 4eeb7a19d4140f9985c8a46dd357f7631ff9d490
Summary:
Aiming towards moving unmount() into FsChannel, move the knowledge of
how to unmount an NFS mount into Nfsd3.
To support unmounting on Windows, we can eventually add an invocation
of unmount.exe to the Windows PrivHelper implementation.
Reviewed By: kmancini
Differential Revision: D45296963
fbshipit-source-id: 55fa7fe0f6190d3708caa21a0cd4b3868f464f8b
Summary:
Aiming towards moving unmount() into FsChannel, move the knowledge of
how to unmount a FUSE mount into FuseChannel.
Reviewed By: kmancini
Differential Revision: D45296899
fbshipit-source-id: 3f270bcc21386224cf7472eb88c851ca1f718e8a
Summary:
picking up from D37112756.
given thrift library `foo`, where before we'd generate target `foo-rust` (containing a `types.rs` module) we now generate two targets, `foo-rust` & `foo-rust-types` where the `types.rs` module is in the latter crate which is depended upon by the former. the motivation is to enable greater build parallelism (see D37112756 for more).
## buck
### files and symlinks in buck-out/.
single-file-thrift-library-foo multi-crate {F1041073024}
multi-file-thrift-library-foo multi-crate
{F1041073062}
## autocargo
### files in target/.
single-file-thrift-library-foo multi-crate {F1041073095}
multi-file-thrift-library-foo multi-crate {F1041073125}
## deployment plan
- diff (1) land autocargo changes (D47342888)
- will break autocargo workflows not using `arc`
- diff (2) initiate an autocargo bump
- https://www.internalfb.com/intern/msdk/bump/autocargo
- diff (3) (D46573767 - this diff) rebase all remaining changes on (2) and land it
- thrift compiler, compiler wrapper, rust thrift build defs
- regenerated 'Cargo.toml', 'thrift_build.rs' and 'thrift_lib.rs' files
- will break autocargo thrift builds (bad "thrift1"; invoking incompatible thrift compiler)
- diff (4) initiate a thrift1 bump (now (3) has landed) and land it
- https://www.internalfb.com/intern/msdk/bump/thrift1
Reviewed By: zertosh
Differential Revision: D46573767
fbshipit-source-id: b2db8e3bf307a6498f92cdb8a64c26b8871a0a1d
Summary: We needed a way of making EdenFS aware that some mount supports symlinks on Windows while at the same time make hg aware that that same mount / working copy supports symlinks.
Reviewed By: xavierd
Differential Revision: D45442009
fbshipit-source-id: 33640dbe1b6508092b622a977c357fb172c11d4c
Summary:
Provided implementation for EdenMount::chown on NFS (MacOS). Utilized exisiting code to get all referencered Inodes and then loaded and invaliated them.
Ran into some issues compiling Ranges - something to learn more about - where certain iterator traits needed to be defined. In subsquent diff should be able to address (if desired), but for this diff just used a vector instead.
Reviewed By: MichaelCuevas
Differential Revision: D46946414
fbshipit-source-id: b405f16712d6d6315d43d6973e7615aced7e5c7d
Summary: Tuning particular numbers (like page size) did not show performance improvement when benchmarking, but enabling these two modes did show slight improvement.
Reviewed By: chadaustin
Differential Revision: D46325226
fbshipit-source-id: 6e0cc424f4d29c76be3237178a8a73f03066f67b
Summary:
Serializing and deserialzing the OverlayDir has similar runtime to copying the structure directly but is more memory efficient. For the benchmark I was running on Linux
```
I0509 16:33:45.357141 2489454 BufferedSqliteInodeCatalog.cpp:303] odirSize: 900, serializedSize: 254
```
Reviewed By: mshroyer
Differential Revision: D46325184
fbshipit-source-id: 8d5c063002e4601f81592941ee8791f239c85682
Summary: Adding a possibility to configure the default blake3 version that could be used if config is not set.
Reviewed By: xavierd
Differential Revision: D46934813
fbshipit-source-id: 6d6032eec1204800f06d5794c5150db7e139a6fa
Summary:
eden doctor can discover when an inode is missing for a file, but can't
remediate the issue. restart usually remediates the issue, but it would be
better to have doctor remediate since restart can be very slow.
We could do something similar to our remediation for phantom inodes (performing
a filesystem operation). However, messing with the filesystem leaves us open to
races with concurrent modifications. The point of the filesystem io is to make
eden see a notification about a certain path and match it's state to the
filesystem. So we can directly do that instead.
We can more directly do this by introducing a thrift call to make eden match
it's internal state to the filesystem.
We could replace the other remediation with this thrift call. I'll leave that
for a follow up.
Reviewed By: xavierd
Differential Revision: D46243633
fbshipit-source-id: a1df5929428dc4f6c8fd71d826fe1e7371ebf283
Summary:
In the case where PrjfsChannel::start failed to initialized, EdenFS would crash
due to the stopPromise_ not being fulfilled. That promise is being set in the
PrjfsChannel::stop method, but unfortunately, when start throws, the caller
wouldn't catch the exception.
To fix this, we can simply move all the code in the initialize method, which if
it throws, the caller already properly tears down the mount thus solving the
crash mentioned above.
Reviewed By: genevievehelsel
Differential Revision: D46210460
fbshipit-source-id: 1524e8e0fc74ce3119b1b6dcd35b5419efcebdf3
Summary: Expose blake3 via file attributes and changing some integration tests accordingly. Added blake3_sum so that it could be used to verify blake3 hashes and updated the tests to work with blake3.
Reviewed By: chadaustin
Differential Revision: D46307686
fbshipit-source-id: 6f2a4e8e25757862ef17d56f92b90a95c7f5a474
Summary:
Adding blake3 support into TreeEntry and backing stores.
Note: Http and RE CAS stores don't provide blake3 hash so far. While it would be pretty easy to add support for RE, not sure how hard it would be for the http store.
Reviewed By: chadaustin
Differential Revision: D46268715
fbshipit-source-id: db66e63fe0348eb582a8050f22cdc0ff720ccf85
Summary:
Adding blake3 into the `Entry` and `Info` objects to make it possible to retrieve blake3 hashes.
The FileNode part will be integrated in the next diff
Reviewed By: chadaustin
Differential Revision: D46268719
fbshipit-source-id: b787b583e5abb3ed306fd1870f96bbdbc9fc0711
Summary: allow using `SqliteOverlay` on non-Windows, since OverlayChecker has now been updated to work with it.
Reviewed By: chadaustin
Differential Revision: D44105360
fbshipit-source-id: 5f78e6c9b5323faeee394da4a51d7e84f8d44a2f
Summary:
Extends `FsckTest` to test `SqliteOverlay`. Two major changes were made to non-test files:
1) `loadInodeInfo` was moved from `OverlayChecker` into `InodeCatalog`
2) `getAllParentInodeNumbers` was added to facilitate `OverlayChecker::readInodes`
Reviewed By: chadaustin
Differential Revision: D45447172
fbshipit-source-id: 9ad291a0c4c34f1e44e1f05c1fdbf2ac7ea7d309
Summary: This will no longer be tied to the fscatalog, so it makes more sense to live in a more general location
Reviewed By: chadaustin
Differential Revision: D44980396
fbshipit-source-id: 3cabf00d87df7eb00671c2b82b04c8f12283e394
Summary:
For tools that want to take advantage of the same fast-path logic when
directories don't change across updates, expose a semistable ID they
can use to cache derived data or get a rough understand when a
directory has changed its contents.
Reviewed By: kmancini, xavierd
Differential Revision: D45974142
fbshipit-source-id: 7b2b482876b07e73514a936e198de2dc31ed1597
Summary:
In both the ObjectStore and in the hg BackingStore, copies of the unique_ptr
were being made. For large blobs this is particularly inefficient as
potentially several MB (if not more) of data needs to be copied. Let's fix this
by changing the BackingStore API to return a shared_ptr.
In order to make the code easier to read and write, also define 3 types:
TreePtr, BlobPtr and BlobMetadataPtr and use them in the BackingStore code.
Future changes should be done at a later point to convert the whole codebase to
using these.
Reviewed By: chadaustin
Differential Revision: D45967102
fbshipit-source-id: 6086f95456232db48a5cbec47b7cf8b14e4424ed
Summary:
We have several legacy conditional includes that are no longer
necessary. Remove another.
Reviewed By: xavierd
Differential Revision: D45927567
fbshipit-source-id: 05bea9a94b249104c26893cfc0017fa472643ce5
Summary:
I want to use EdenMountHandle in EdenServiceHandler without pulling in
EdenServer.h.
Reviewed By: xavierd
Differential Revision: D45924297
fbshipit-source-id: 2043374d513c01423cd6f785b2c4e2b36d937676
Summary:
I ended up in a crazy situation where the CMake build was failing to
link because `target_link_libraries` had a cycle. Fix that by
preventing cycles in the first place. CMake is the gift that keeps on
giving.
Reviewed By: xavierd
Differential Revision: D45927359
fbshipit-source-id: 6a378f7935195e42b3446249091898370d6e2ee5
Summary:
enums can contain arbitrary values, so gcc (and older clang?) complain
about the possibility that none of these cases match.
Reviewed By: xavierd
Differential Revision: D45927298
fbshipit-source-id: 0cd0a946d0faf7442a1a8d5de388a7d4b6dbb274
Summary: Add a function to query the (optional) ObjectId from VirtualInode.
Reviewed By: kmancini
Differential Revision: D45673165
fbshipit-source-id: 72dadaf4cbd5120c75548734aca2bb8ae715b8e2
Summary:
Add a function to query the (optional) ObjectId from an arbitrary
inode.
Reviewed By: kmancini
Differential Revision: D45673053
fbshipit-source-id: 23630638ba94bbd1925a39193fa125df04abebad
Summary:
The EdenMount::getObjectStore gave the assumption to callers that the
ObjectStore lived longer than the EdenMount. Since this assumption is implicit
and not explicit, there is no way to guarantee it by just reading the code in a
local fashion. It forces the reader/auditor to understand the code in its
entirety. Unfortunately, EdenFS contains more of these than can be kept in
memory which leads to bugs and use-after-free.
This can be avoided by breaking the implicit lifetime assumption and simply
returning the shared_ptr. If the caller needs to keep the ObjectStore around,
they can do so explicitely which makes reading the code easier.
Reviewed By: chadaustin
Differential Revision: D45831168
fbshipit-source-id: 8bdb8e708308a8a220e3130d2fc51854c8556cb1
Summary:
This will be used in a later diff outside of EdenMount.cpp, thus let's move it
to its own file.
Reviewed By: chadaustin
Differential Revision: D45831170
fbshipit-source-id: 1cc55e1ae580ee7f02e3d9c6a717da4751c955e6
Summary:
ServerState.h is a widely-included header, as it's the dependency
injection seam used by EdenFS's inodes layer.
It included several unnecessary headers. I noticed this by seeing too
many files recompile when changing PrivHelper.h.
Replace the includes with forward-declarations.
Reviewed By: kmancini
Differential Revision: D45260059
fbshipit-source-id: 31a73030c9eb592107e8786be026e32eea80ceb4
Summary:
EdenMount used to track the set of bind mounts for a checkout. That's
long been removed, and bind mounts are now managed by the CLI (and
rectified against the mount table).
Move these vestigial EdenMount methods into EdenServiceHandler,
simplifying lifetimes and moving us closer to breaking EdenMount's
dependency on PrivHelper.
Reviewed By: kmancini
Differential Revision: D45259970
fbshipit-source-id: e9f1bb1c44fb74e49ece08416fbe1913dd1a6ea1
Summary:
We use privhelper for NFS too, so move fs/fuse/privhelper into
fs/privhelper.
Reviewed By: kmancini
Differential Revision: D45243273
fbshipit-source-id: cfc93b324768809c87075b0085905e873c7ea88d
Summary:
It's unfortunate that changing NFS code causes most of the inode layer
to recompile. One benefit of FsChannel is we can begin to decouple
EdenMount from FUSE, NFS, and PrjFS. Remove the Nfsd3.h include from
EdenMount.h.
Reviewed By: kmancini
Differential Revision: D45167461
fbshipit-source-id: 41650390cbd053233dfe3baf051c592bc203de56
Summary:
waitForPendingWrites is general to any userspace filesystem with
asynchronous write notifications. This could even include FUSE in
writeback caching mode. To remove another ifdef, move
waitForPendingWrites into FsChannel.
Reviewed By: kmancini
Differential Revision: D45167206
fbshipit-source-id: 4daec70845864d90b94739fe2011e8ed90a6a200
Summary:
Even though PrjfsChannel has no definition on unix, we can still offer
the function. This removes an ifdef. Any attempt to PrjfsChannel
remains limited to Windows.
Reviewed By: kmancini
Differential Revision: D45167158
fbshipit-source-id: b2769ba003c3decf964c9cea06b8b4238ed5887c
Summary:
There's no need setting a test FsChannel needs to be prjfs-specific,
so remove the ifdef and introduce setTestFsChannel.
In the future, it would be nice if the FsChannel was specified at
construction so EdenMount can just be given an appropriate one during
tests.
Reviewed By: kmancini
Differential Revision: D45167055
fbshipit-source-id: 4274e827a8cf0578b88b63214558236aceb12604
Summary:
The only remaining use of EdenMount::getMountProtocol() is for
stringifying into the FinishedMount StructuredLog event. We can use
FsChannel::getName() instead. The only difference is that
FsChannel::getName() is lowercase, but I did not see anything using
the logged value.
Reviewed By: kmancini
Differential Revision: D45159419
fbshipit-source-id: 08b6a21d910bdf9c27f91c3e021c7ca8c902d478
Summary:
I'm about to make some changes to VirtualInode. In advance, refactor a
few things I noticed.
Reviewed By: kmancini
Differential Revision: D45672797
fbshipit-source-id: 7edf67ac14fb9b98324d3ed20eecbebb5ff903c6
Summary:
I'm about to make some changes to VirtualInode. In advance, clean up
the visits with the new `match` function.
Reviewed By: mshroyer
Differential Revision: D45629562
fbshipit-source-id: f7bab01366f88c2e5a8f3d2a6d8eecef4f769d5a
Summary: The SpawnedProcess constructor can throw in some cases. In these, we want to make sure `EdenMount::performBindMounts` returns a failed `SemiFuture`, not throw an exception as throwing an exception will lead to a mount failure while it is instead desirable to continue the mounting process.
Reviewed By: chadaustin
Differential Revision: D45759407
fbshipit-source-id: 9c190bf5e29ae4b4d94089b6d469506deb4572f6
Summary: VirtualInode included several things it never used. Remove them.
Reviewed By: genevievehelsel
Differential Revision: D45629434
fbshipit-source-id: db937ddba1886c0fa2461ec9f1e5842462275be9
Summary:
Staging an update to the latest fmt version triggered lots of build errors due to non-`const` methods on custom formatters. This fixes the `format()` methods to be `const` as they don't mutate any state anyway, as well as `parse()` methods that don't need to mutate internal state. This mitigates many future build errors.
Updates were identified and executed by using regular expression search/replacements such as:
`(constexpr auto parse\(ParseContext& [^)]*\)) \{` -> `$1 const {`
`(constexpr auto parse\(ParseContext& [^)]*\)) ->` -> `$1 const ->`
`(auto format\(.*, FormatContext& [^)]*\)) \{` -> `$1 const {`
`(auto format\(.*, FormatContext& [^)]*\)) ->` -> `$1 const ->`
Any changes to third-party code was then reverted. Some small changes detected from subsequent build errors were then applied.
Reviewed By: vitaut
Differential Revision: D45420948
fbshipit-source-id: 211bf7f3b59616eda3cd228caf4e93ff81d4fc2b
Summary: Newer versions of fmt require that you pass an iterator rather than a direct memory buffer as an output to the `format_to()` family of functions. This does that to unblock an upgrade to a newer fmt version.
Differential Revision: D45555419
fbshipit-source-id: a4e2217299ada0ab5e56942dacc9107153e4260d
Summary:
A primary advantage of existentials over sum types is that they allow
they breaking dependencies. Now that channel_ is an FsChannelPtr, we
no longer need to include PrjfsChannel.h in EdenMount.h.
Reviewed By: kmancini
Differential Revision: D45109931
fbshipit-source-id: 7774804fe4056ef5b897de294707496ff722db28
Summary:
On Windows, macOS, and Linux, EdenMount::channel_ is an
FsChannelPtr. Code that wants to branch on a specific implementation
can use getFuseChannel or getNfsChannel or getPrjfsChannel, though
those will be removed later.
Reviewed By: kmancini
Differential Revision: D45109907
fbshipit-source-id: d4b3b46bd78b9487909bf1cb7c1c754e48b6e9a9
Summary:
FuseChannel already had an asynchronous initialize(). Lift that to
FsChannel and unify initialization of Prjfs and NFS.
Reviewed By: kmancini
Differential Revision: D45106889
fbshipit-source-id: 632ddfa275c732fc30efc1984bda43f61c37fd5e
