Summary:
When edenfs is stopped, `eden restart` fails on Windows:
```
PS C:\> eden stop
Stopping edenfs daemon (pid 6232)...
edenfs exited cleanly.
PS C:\> eden restart
Found an existing edenfs daemon (pid 6232 that does not seem to be responding to thrift calls.
Use `eden restart --force` if you want to forcibly restart the current daemon
```
This is due to a boolean flip error.
Reviewed By: xavierd
Differential Revision: D33698942
fbshipit-source-id: 487b587cd278a77e96e2e3a7a0560caa094875a4
Summary:
Currently the only place we log which channel type we use is to the local
on disk log. This is a start, but does not allow us to track the global numbers
of people using the different channel types (ex. FUSE vs. NFS).
Reviewed By: xavierd
Differential Revision: D33671457
fbshipit-source-id: 6db11d4567127f3ad0fc1d9446f178f249aa1dae
Summary:
We've started seeing hangs during update on Windows due to EdenFS trying to
invalidate files and directories that do not have a placeholder already. When
that happens, ProjectedFS will send a lookup callbacks to EdenFS to write a
placeholder to disk. This triggers a recursive callback to EdenFS, which
unfortunately ends up deadlocking EdenFS.
The reason EdenFS is trying to invalidate these files and directories, is due
to the original diff eagerly loading more inodes to speedup startup time. Since
this original diff is causing so much issues, let's revert it.
Reviewed By: kmancini
Differential Revision: D33697483
fbshipit-source-id: 6c15c9cfbd784e1033c2096e1794929b888bbbd7
Summary:
On Windows, the parentLock needs to be acquired on PrjFS callbacks in order to
walk the Tree hierarchy. Unfortunately, since checkout holds this lock for the
duration of checkout operation, the callbacks will be blocked. This is usually
not an issue as update is expected to be relatively fast, but in some cases, an
update may trigger these callbacks. One of these can happen when EdenFS is
killed midway through a checkout operation and checkout tries to resolve
conflicts during a force checkout, the current code will iterate through many
unloaded inodes and issue recursive callbacks.
While the above issue can be fixed, we can also rethink the reason for holding
the lock for the duration of checkout. The main one being to prevent concurrent
checkout, and this can be achieved by storing a boolean protected by the lock.
That way we can detect the pending checkout, and not hold the lock for too
long.
Reviewed By: chadaustin
Differential Revision: D33668308
fbshipit-source-id: 634c60d4684c755d38e4b26826e3b14afbc6083a
Summary:
When an update removes a directory but a materialized ignored file/directory
was present, EdenFS will rightfully not remove that ignored file/directory.
However, the parent directory would have been a placeholder on disk. And since
placeholder directories can never be made full[0], the directory keep its
placeholder. Listing this directory would thus issue an opendir callback, but
EdenFS would fail due to that directory not being present in the manifest.
Since this case can only happen in this case, let's simply return an empty
directory and let ProjectedFS merge it with the on-disk directory content.
[0]: https://docs.microsoft.com/en-us/windows/win32/projfs/cache-state
Reviewed By: genevievehelsel
Differential Revision: D33676621
fbshipit-source-id: bdbf769175a6471a1a709bdb0b03a45bd60ef366
Summary:
When EdenFS is stopped, users have attempted to remove the root directory of
the mount point, in the hope that it deletes their repository. Cloning a new
repository at that location gives them an unhelpful error message claiming that
the directory isn't empty:
error: [Errno 41] Directory not empty: 'C:\open\foobar'
Let's improve this error to help guide the user towards using `eden rm`.
Reviewed By: kmancini
Differential Revision: D33574906
fbshipit-source-id: 3f2ae567dd9b2af2493e6e52c52c85a8216c993b
Summary:
Now that notifications are running in a serial executor, and are also issuing
disk IO, notifications are significantly slower than they used to be. While
writing to the working copy is an overall anti-pattern, some workflows (like
Unity) do and it's thus critical that their performance isn't affected
negatively.
In order to solve this, we can simply move the handling of notifications to the
background and answer the notification immediately, since notifications can no
longer fail, we shouldn't need to send an error back to ProjectedFS which would
anyway be ignored.
The drawback is of course that applications are no longer blocked while the
notification is being processed in EdenFS, and thus any operation that needs to
get a synced up inode hierarchy will need to wait on all the pending
notifications.
Reviewed By: chadaustin, genevievehelsel
Differential Revision: D32480993
fbshipit-source-id: 7ad6b07f540f7d9a52a35a0ff3b94911ef5267af
Summary:
ProjectedFS is... weird. It can sometimes send notification out-of-order for
things like directory/file creation and removal and sometimes even send
duplicated removal notifications. This makes is really hard in EdenFS to keep
track of the changes in the inode hierarchy to behave similarly to NFS/FUSE
where the inode hierarchy always matches with what a user sees.
Previously, only out-of-order directory creation was handled correctly by
simply creating the parent directory if these weren't known by the inode
hierarchy, but this scheme fall over when handling directory removal. Consider
the following operations from the point of view of a user:
mkdir a/b/c/d
rmdir a/b/c/d
mkdir a/b/c/d
EdenFS can receive the notifications in the following order:
mkdir a/b/c/d
mkdir a/b/c/d
rmdir a/b/c/d
If EdenFS blindly follow the received notification, EdenFS will lose track of
the directory `d`.
Since deletion notification have been observed to also be sent multiple times
by ProjectedFS, EdenFS has no choice but to check the state of files on disk
and updating the inode hierarchy based on it. In order to avoid concurrency
issues, this is also done in a single thread.
For now, this is performed while servicing the notification and thus is
expected to slow down every writes, a future diff will return early from the
notification without waiting for the inode hierarchy to be updated to mitigate
this.
Reviewed By: chadaustin, fanzeyi
Differential Revision: D32329808
fbshipit-source-id: 33e01936fa49d2d8efa4ea3712f07dd2a132f3a4
Summary:
Adding more redirection-handling code in the next diff, so moving some redirect code to its own file.
Also was annoying to detangle some changes I made at the same time so they are bundled in this diff:
- Changed the deserialize_with handler of redirections to not return an `Option` and just return an empty `BTreeMap` instead
Reviewed By: kmancini
Differential Revision: D33161964
fbshipit-source-id: 5452d5026cca781710793486ce2b243588483f9b
Summary: Used in later diff D33162049 to get redirections disk usage.
Reviewed By: fanzeyi
Differential Revision: D33161819
fbshipit-source-id: 6790cfbb4b5435941f9760d2ecc71bf7821e9603
Summary: This was working fine for the usage counts we've aggregated so far, but when you try to aggregate the redirections, this gets stuck in infinite loops due to symlinks. Using `symlink_metadata` makes it not follow symlinks (symlink directories no longer pass the `is_dir()` check and are treated as a file). This matches what is done in the python version of `eden du`.
Reviewed By: fanzeyi
Differential Revision: D33160356
fbshipit-source-id: 81cdc638a262cad26aee29cbc93b1efbd26f23c5
Summary: Moving this to utils because I use it in later diffs in a different file other than `du.rs`. Also rename from `du_` prefix to `eden_` prefix.
Reviewed By: kmancini
Differential Revision: D33160357
fbshipit-source-id: 2e2a1de9288fca8f60191771e1fe2bcc539c3c4a
Summary:
While adding support for unmounted stale NFS mounts I realized that
`eden doctor`'s stale mount detection and fixing on macOS are broken.
Lets fix that.
Reviewed By: genevievehelsel
Differential Revision: D33290981
fbshipit-source-id: 69502a9b858b130882f01fc40e0ec64aeb11555e
Summary:
The _LPDWORD creates a nullptr, and thus GetExitProcess would fail to write to
it, raising an OSError which would pretend that the process is no longer alive.
This caused `eden stop` to not really wait for EdenFS to exit. Similarly,
`eden restart --force` would sometimes fail to start the new EdenFS due to the
old one not having released the lock file.
Reviewed By: genevievehelsel
Differential Revision: D33438953
fbshipit-source-id: 756ff4541ace5be0d081c8d4902838a50d2b134f
Summary:
The request being handled is for a tree, not a blob, thus we need to get the
tree request.
Reviewed By: kmancini
Differential Revision: D33411533
fbshipit-source-id: 50ffd3be5e3050f71d25a1fe69475443164e0885
Summary:
In order for EdenFS to correctly invalidate all the on-disk placeholder and
full file that are present on disk, EdenFS relies on the loaded and unloaded
inodes. On startup, no inodes are loaded and thus EdenFS needs to fill up the
InodeMap with all the files present on disk. Unfortunately, doing so is very
expensive as checking if a file is present on disk is an IO operation, and thus
can take a really long time: we have seen users with several minutes InodeMap
initialization.
In order to reduce this time, EdenFS can try to be slightly less precise by
assuming that all files present in a on-disk directory have placeholders, and
thus not check them. This can cause more invalidation on update, but these are
much cheaper, and also concurrent, they are also not a bottleneck.
Since all directories on-disk must have an overlay entry, we also don't need to
check the working copy and just test the overlay.
Reviewed By: fanzeyi
Differential Revision: D33414475
fbshipit-source-id: efe0719bbac6cbba3aa4300fe4810dbee2e43841
Summary:
You can't run Autocargo right now, it fails with:
```
Error: File CargoTomlPath { file: PathInFbcode("configerator/structs/data_access_policies/upf/eval/Cargo.toml"), dir: PathInFbcode("configerator/structs/data_access_policies/upf/eval") } is covered by both ctp and pxl projects
```
This is affecting me (on Buck2) since some of our dependencies use Autocargo
for their Cargo.tomls and those are now missing some dependencies (notably,
`below` needs `nix`). This also affects people trying to vendor crates:
https://fb.workplace.com/groups/rust.language/permalink/7625318880849994/https://fb.workplace.com/groups/rust.language/permalink/7577471608968055/
This diff fixes it, and regenerates Cargo.tomls as a result.
Note: I'm not sure exactly what "pxl" is, but it seems to be owned by the team working on data access so that's a better home for that crate than ctp.
Reviewed By: ahornby
Differential Revision: D33428045
fbshipit-source-id: f3feab3ae04069672040423c145c69a58445ef96
Summary:
With Facebook having been renamed Meta Platforms, we need to change the license
headers.
Reviewed By: fanzeyi
Differential Revision: D33407812
fbshipit-source-id: b11bfbbf13a48873f0cea75f212cc7b07a68fb2e
Summary:
Now that EdenFS is packaged as a universal binary, we can start it as a ARM64
one. Unfortunately, since `edenfsctl` is an Intel binary, the Intel version of
EdenFS will always be chosen, and thus we need to manually specify the ARM64.
Reviewed By: kmancini
Differential Revision: D33198960
fbshipit-source-id: 1bb3d7469fb6830a2cf1d20927519840f40c44fd
Summary: This diff makes symlinks behave similarly to bind mounts. Instead of failing if a user attempts to add a symlink redirection that already exists, we simply print out an acknowledgment and return success.
Reviewed By: kmancini
Differential Revision: D33258881
fbshipit-source-id: d2d343b9d91b1c4e1f0aec5a5b83a35b10733f93
Summary: I thought it existed anyway, so I thought I might add it if no one objects.
Reviewed By: kmancini
Differential Revision: D33247766
fbshipit-source-id: 610fa019bd5887dab286778e16ee200f79d6d3d0
Summary: Warn users when `eden redirection add` overwrites an existing directory. Users can prevent overwriting existing directories by adding the --strict flag to the command.
Reviewed By: kmancini
Differential Revision: D33246406
fbshipit-source-id: 1c08e35437060a626220f99324b59ff3c9e7ef81
Summary: We currently just fail silently when we read a malformed config. Instead, we should catch the exception and warn the user about which config format we use.
Reviewed By: kmancini
Differential Revision: D33286752
fbshipit-source-id: 7823cae71a3f248659aa7277c0e5c87d019aefbd
Summary:
It feels a bit weird for getDaemonInfo().status's meaning to diverge
from fb303's getStatus(), but fb303 is being restructured so remove
the custom getStatus implementation.
Reviewed By: praihan
Differential Revision: D33176576
fbshipit-source-id: 5a7b7f3394b123b1073191508c2c0c1ebcda5bb5
Summary: xavierd was seeing some overlapping between the pending counts and the table: P471839300. I suspect this is because the pending counts were outputted via `println` and the table via a stdout handle. Seems like a race between these two. Let's add the output to just the stdout one stdout handle.
Reviewed By: xavierd
Differential Revision: D33176273
fbshipit-source-id: 4c4c91e1521323bf51359a503153295ab752f377
Summary:
The time of these can be useful to confirm whether EdenFS crashed or not, the
minidump may also be useful to debug why EdenFS crashed, thus let's upload it
too.
Reviewed By: chadaustin
Differential Revision: D33147862
fbshipit-source-id: b47253c521336c2fb08be2e41183c5c7f7918b5d
Summary:
These diagnostics traces contain full backtraces for all threads, plus other
useful information. When EdenFS crashes, these can be very useful in
understanding why, thus they can be useful to collect.
Reviewed By: chadaustin
Differential Revision: D33146311
fbshipit-source-id: f420583e9bef43d9a3a29226edf8ac77fd54de33
Summary:
Use ThriftServer::setAllowCheckUnimplementedExtraInterfaces to silence
the warning instead.
Original commit changeset: 84ff38a00f15
Original Phabricator Diff: D33146694 (c56347e76d)
Reviewed By: praihan
Differential Revision: D33178139
fbshipit-source-id: ac151b0184bf058035c6732210119a9e00b8db13
Summary: I want to use this function elsewhere so moving it to utils
Reviewed By: xavierd
Differential Revision: D32971287
fbshipit-source-id: 243d8ca962311139d32e7dc860768097276f9355
Summary:
In D32655834 (aa573849fb), I incorrectly removed the code that prints the error message.
We can't simply revert back because a) `.exit()` function never returns b) we want to collect telemetry.
So this diff re-implemented the `exit()` function without terminating
Reviewed By: genevievehelsel
Differential Revision: D33149361
fbshipit-source-id: 2d36b4e4f25ebc85a9ee15f59aafc78b82c37d3e
Summary: 0.27 onwards has support for large tuples so can remove patch
Reviewed By: farnz
Differential Revision: D33100620
fbshipit-source-id: 68348b16113043826dfb253acc64048d01bba7d1
Summary: The change is in upstream slog-syslog release 0.13.0 so we can remove the patch
Reviewed By: jsgf
Differential Revision: D33099611
fbshipit-source-id: a30b89179d18cbd7b3d9dbe6ac8a461b143b01c0
Summary: Instead of taking a uint64_t, we'll take in an InodeNumber. This allows us to avoid the conversion from InodeNumber -> uint64_t -> InodeNumber.
Reviewed By: chadaustin, xavierd
Differential Revision: D33135685
fbshipit-source-id: c9e4317b1f5f97ad924dce5da322860c91c64115
Summary:
Silence the "FacebookBase2 is going away! To drive the migration,
Thrift server will send a fake getStatus RPC at server start."
warning, because it does not apply to EdenFS.q
Reviewed By: genevievehelsel
Differential Revision: D33146694
fbshipit-source-id: b33480c0d3c1255efe50ff43aaa838c100d26085
Summary: don't include this header if we're on Windows
Reviewed By: xavierd
Differential Revision: D33111628
fbshipit-source-id: 8bf6c7db4f023f7c348f1e1c439c15856acbf476
Summary: Pass use_readdirplus config option to the priv helper. If the option is enabled, we should use the appropriate options when mounting.
Reviewed By: kmancini, xavierd
Differential Revision: D32777711
fbshipit-source-id: b4717fdb52e7a36f9ec1b5ba4158f34a855f3a9b
Summary: Enables the use of readdirplus in Eden. This diff simply adds the option. The next diff will include the logic needed to use this option.
Reviewed By: xavierd
Differential Revision: D32775492
fbshipit-source-id: afeb705ee0bfb4f7e975c0e9a96f01cf64942f75
Summary: Before this diff, we were stubbing out the stat data with an empty post_op_attr struct. This diff adds the functionality for inserting the stat data into the NfsDirList struct so we can return the stat data to the kernel.
Reviewed By: xavierd
Differential Revision: D32771629
fbshipit-source-id: 27f7e5ac369cc804eacab9f7549e230118aaeaab
Summary: In the next diff, we need to use statToPostOpAttr() in another file. We will move these utility functions so we can access them from any file.
Reviewed By: xavierd
Differential Revision: D32771540
fbshipit-source-id: f4d6e2819da3ef66248134337f98eb0a938b6edb
Summary: This method is needed so that the NfsDispatcher layer can fill in missing stat data for each DirList entry.
Reviewed By: kmancini, xavierd
Differential Revision: D32770593
fbshipit-source-id: 2510adf8b8738259d5e6b8714c6492fd4f75f5d3
Summary:
We account for args.dircount in this implementation (dircount is the max number of bytes we can use for holding directory information)
We *don't* account for args.maxcount. We should probably do that.
Reviewed By: kmancini, xavierd
Differential Revision: D32767072
fbshipit-source-id: c07d0f3aec5a13ed2bb3fa6960ea65e95438dff3
Summary:
The current readdir implementation uses NfsDirList to hold readdir results. Our new readdirplus implementation should use the same struct. To do this, we will add the ability to use an entryplus3 XdrList instead of an entry3 XdrList.
We also add a new constructor so we can specify which XdrList to initialize when we create an NfsDirList.
Reviewed By: xavierd
Differential Revision: D32708688
fbshipit-source-id: e2b324dd1dffc8620be89dca471043aba9b167a5
Summary:
Calculating the size of materialized files and printing that in json.
Todo in future diffs:
- print in non-json mode
- print failed files to check
Reviewed By: fanzeyi
Differential Revision: D33070717
fbshipit-source-id: 2d306f30bcc57b46b1ce1116c586f8d0f0b301b4
Summary: In order to be most accurate in which files a job uses, we should not record file accesses/fetches that come from a prefetch request.
Reviewed By: xavierd
Differential Revision: D33071618
fbshipit-source-id: 7516348996c9f94214135184bb192f39871ba7c0
Summary: The default for normal prefetch profiles should be set to "ON" since it is supported on all platforms. This will allow us to delete some managed configs, we can always turn it off using configs if needed.
Reviewed By: xavierd
Differential Revision: D32964459
fbshipit-source-id: 2db28ec08586d79daa8b6601e2457f3bd2afe9e9
Summary:
In the following diffs I am going to use this in multiple files of eden doctor
so let's separate this out into a utils file. More to be added to utils in the
following diffs
Reviewed By: xavierd
Differential Revision: D32971283
fbshipit-source-id: a3f49ca0ef9afcaede40b5cff2332397c7219845
Summary:
This option is only ever used for tests, let's remove it and simplify the code
a bit.
Reviewed By: singhsrb
Differential Revision: D33103247
fbshipit-source-id: e822de6dba541152452984f6aad5258a5585db02
Summary:
On Windows, even though a process died, its pid can still be queried, thus
issuing a `eden stop` followed by an `eden restart` would try to kill an
already dead process and fail with a permission denied that would be displayed
to the user.
Let's test if the process is still alive before claiming that a given PID is an
EdenFS process to solve this.
Reviewed By: genevievehelsel
Differential Revision: D33076917
fbshipit-source-id: 062ec9e99ba2014720e3aa8d873309f5e581185d
Summary:
Like it says in the title. In Buck2 this doesn't quite work as `__FILE__` is
relative to fbsource but now tests are relative to fbcode.
This might be something we have to fix in some way, but until then let's just
fix the test.
Reviewed By: xavierd
Differential Revision: D33092380
fbshipit-source-id: f54fabe795f909111cc7f2269359a4117232e630
Summary:
On clone, the CLI tries to open an .arcconfig file at the given revision for
the solve purpose of printing a slightly different log message. Since this log
isn't necessary to be present, let's avoid the `hg cat` and simplify the code a
bit on top of making it faster.
This should make cloning a couple of seconds faster on Windows due to Mercurial
taking roughly a second to start.
Reviewed By: chadaustin
Differential Revision: D33070099
fbshipit-source-id: 81c5cba8348d0c8d82c57a6656e6eb79c2b1f878
Summary: Last users are gone, lets remove these old crates
Reviewed By: wqfish
Differential Revision: D33061743
fbshipit-source-id: bea14f6f199a3f805d76318d3c8070739abc6085
Summary: The directory map keys should always be paths and are always converted to `PathBuf` by usages, so let's just convert them to `PathBuf`s in the directory map getter method.
Reviewed By: fanzeyi
Differential Revision: D33001349
fbshipit-source-id: aa08c6e6756cca556dc92c3babf585072b87a45b
Summary: The existing code doesn't throw an error if the directory map does not exist. I missed this part when re-implementing the code which causes the integration test to fail in the case where eden hasn't started yet.
Reviewed By: fanzeyi
Differential Revision: D33001350
fbshipit-source-id: 603690d49f4273c5ef6cb2ba634d2f8c2c94cf60
Summary:
Drop all the remaining instances of `IOBufQueue::clear()`.
(Note: this ignores all push blocking failures!)
Reviewed By: yfeldblum
Differential Revision: D32941992
fbshipit-source-id: c7d983e215a78014ffda74a944b628e1886c35c8
Summary:
This refactor is necessary so we can get the exit code of the Python process.
Now we are opening a subprocess on Linux with the Python edenfsctl instead of `execing`.
Reviewed By: xavierd
Differential Revision: D32655834
fbshipit-source-id: 3e9d05eccdf566052f101f121bf15eab696f86e5
Summary:
These checkers do verify that a user is on the VPN, or that their certificates
are not expired. Since we didn't bundle these, we weren't checking these
things...
Reviewed By: chadaustin
Differential Revision: D33007790
fbshipit-source-id: e4e7f8b625443b4f2f9bfd0fe181b7a1858cd97c
Summary: collections was unused and the linter was warning about it. Remove it so the linter is satisfied :)
Reviewed By: kmancini
Differential Revision: D32922201
fbshipit-source-id: c4142196d2f5ad4247f0ecfb0bb4c811d501824a
Summary:
eden doctor should detect when a user has nested checkouts and warn them that it could cause issues. No action should be taken at this time because the nested checkouts could be intentional.
In the future, we may attempt to fix the issue for the user.
Reviewed By: chadaustin
Differential Revision: D32517950
fbshipit-source-id: 15c0e60149d0513a8bee72adc254b0fae369edfe
Summary: FakeEdenInstance is missing many key methods and fields. Add get_mounts() so that D32517950 can pass the doctor tests.
Reviewed By: kmancini
Differential Revision: D32922051
fbshipit-source-id: 42631bfae43259dfbd271cba9e367910be00bdf1
Summary:
By default, we do not want to support nested checkouts. Their behavior is not well defined and can cause issues.
This diff disallows nested checkouts unless specifically requested by users. They can use the --allow-nested-checkout or -n flags to force nested checkout creation. If these flags are not used, the clone is aborted and an error message is printed.
Reviewed By: chadaustin
Differential Revision: D32514269
fbshipit-source-id: d04a7ef82c6d31993801648da42e4182167f37e6
Summary:
Define a new function that searches for nested checkouts in a defined order. The order is currently defined by sorted(), which should sort the checkout paths alphabetically.
This also means for all paths with the same parent directories (e.g. /home/cuev and /home/cuev/fbsource), the shorter path will be searched first.
Reviewed By: chadaustin
Differential Revision: D32699824
fbshipit-source-id: b01b07ab5b3b51e55b33c30b8d28d1ee8f2cc11d
Summary: Adds description for `eden glob` and `eden prefetch` to provide more information on the commands' inner workings.
Reviewed By: malanka
Differential Revision: D33000302
fbshipit-source-id: a780fd10e6bc7db5181cfd517ba7974296c138fa
Summary:
For the same reason that we shouldn't prefetch blobs when SCS is enabled, we
also shouldn't prefetch them if the aux metadata is enabled.
Reviewed By: genevievehelsel
Differential Revision: D33000240
fbshipit-source-id: a194d30b419d9a6a63faeee75c941f3471431840
Summary:
Instead of using unix commands, use Python as a portable way to achieve the
same goal.
Reviewed By: chadaustin
Differential Revision: D32999556
fbshipit-source-id: 8299c36126cf837ab9434402b478455dd643c9ea
Summary:
From the Thrift documentation:
* Ensure that all inflight working copy modification have completed.
*
* On some platforms, EdenFS is processing working copy modifications
* callbacks from the platform in an asynchronous manner, which means that by
* the time a write/creat/mkdir/unlink/etc syscall returns from the kernel,
* EdenFS may not have updated its internal state.
*
* Thus, an application making changes to the working copy and quickly
* requesting EdenFS to perform an operation on it will race with EdenFS
* updating its internal state and may thus get stale data.
*
* To avoid this, EdenFS queries need to internally synchronize the working
* copy before performing the query itself. This structure defines how EdenFS
* will do this.
*
* Applications that care about synchronizing EdenFS up to a certain point in
* time are expected to set a non-zero syncTimeout once to synchronize EdenFS
* and then issue all their thrift requests with a syncTimeout of 0.
Reviewed By: chadaustin
Differential Revision: D32900351
fbshipit-source-id: c242d881b2f2db6cbca544288a0eac5959469fc5
Summary:
static_assert sizeof doesn't show the actual size in clang or gcc, so
implement a CheckSize template.
Reviewed By: xavierd
Differential Revision: D32998926
fbshipit-source-id: 432fbf19eae5b4572e5a4908f9007b7242a3ae69
Summary:
On Windows, even when a process is terminated, its handle can still be queried,
for instance to get its exit code. Thus merely checking if the handle can be
opened isn't a good way to test if a process is alive. Let's instead query the
exit code for the process.
As a bonus, also removed the bit about EdenFS dropping root privileges since
EdenFS no longer starts as root on macOS or Linux.
Reviewed By: chadaustin
Differential Revision: D32995121
fbshipit-source-id: 9ac7946dbb53b3fabb163c98e3f72f9132b13a21
Summary:
While we'd like to change this, Trees require an ObjectId today. Don't
provide a default, and make every ObjectId explicit.
Reviewed By: genevievehelsel
Differential Revision: D32895654
fbshipit-source-id: b45f348101269d80ac7e0dfd6f651be2a49ecc63
Summary:
On Windows, printing paths by default use backward slashes, but forward
slashses are also acceptable. For the sake of making tests pass, let's always
use forward slashes.
Reviewed By: chadaustin
Differential Revision: D32960607
fbshipit-source-id: c1424c4a090d805bab337fc6d540563e8205e5dc
Summary:
On Windows, paths are by default not separated by /, thus use os.path.join to
make these tests portable.
Reviewed By: chadaustin
Differential Revision: D32960608
fbshipit-source-id: 6a7052f0214c2066acb9b99899595eb362ca616c
Summary:
We were always returning the errored Try despite building the correct directory
listing. Let's add it back.
Reviewed By: fanzeyi
Differential Revision: D32964916
fbshipit-source-id: 60faa4534c712287bf0192fd7efbb68d783985d8
Summary: As suggested in D32810877, adding a util for converting &[u8] utf8 bytes to a PathBuf.
Reviewed By: fanzeyi
Differential Revision: D32884767
fbshipit-source-id: 02d79350ed02594d9d3a5c945886b540f5ce74b8
Summary: Not totally necessary, but I think this is cleaner.
Reviewed By: fanzeyi
Differential Revision: D32884768
fbshipit-source-id: e01f860a49310d1dc4c30eb0f343daa96b2521be
Summary:
On Windows, PIDs can be recycled, but the CLI isn't checking if the the PID
stored in the pidfile under ~/.eden/pid. This can lead to some confusing error
messages to users when they try to see if EdenFS is alive after restarting
their computers.
Reviewed By: chadaustin
Differential Revision: D32885862
fbshipit-source-id: deb4a7f3026465acd1421b710ce91290cd0c12aa
Summary: There are cases where ObjectId was created with an empty hash, and in this case, when Eden will consider it needs to materialize, there is nothing to materialize but the backing storage does not contain the content for the hash.
Reviewed By: chadaustin
Differential Revision: D32451346
fbshipit-source-id: 723219a70f4da2127d1a60d82731b08f4befba46
Summary:
Same setup as D32368654 (400c0caa31)
I realized in starting to implement `eden du` that `eden list` was really a subset of `eden du` so doing this as a stepping stone towards `eden du` made sense.
Reviewed By: fanzeyi
Differential Revision: D32810898
fbshipit-source-id: 82012702f97e1a96d6371d7713f5cec15496ebca
Summary:
I was using this to debug graceful restart locally, and had it at a high debug
level. DBG1 is a bit verbose this should probably be DBG7 with the rest of the
debug logging I added.
Reviewed By: genevievehelsel
Differential Revision: D32902321
fbshipit-source-id: 4b7db7cbe96db8ef5b1035932c06a1de264f8502
Summary: This will be used in a subsequent diff.
Reviewed By: chadaustin
Differential Revision: D32837579
fbshipit-source-id: 9a32be6a9c912faaf43064c98f61c9fbec17c7ba
Summary:
I ran into a rough situation this morning where EdenFS was running but
all of my mounts were unmounted. The sequence of events was that,
overnight, the system attempted a graceful restart. EdenFS crashed
during the restart, and left dangling mounts in the mount table. Then
a new EdenFS process started up and attempted to remount, but failed
because the `statfs` in filesystem validation returned ENOTCONN.
I grepped for ENOTCONN in linux/fs/ and it looks like the only
filesystems where this matters are already in the valid filesystem
type list, so allow mounting over an ENOTCONN filesystem.
Reviewed By: xavierd
Differential Revision: D32891776
fbshipit-source-id: 073134326822f417d9148e79bcd5590867884e18
Summary: As now ObjectId can be an arbitrary length of string, we can remove the proxy
Reviewed By: chadaustin
Differential Revision: D32345391
fbshipit-source-id: e5d1db6f0c8e93c6d8a5f5b3d499f4caac9fceb2
Summary:
The file system is not always FUSE and could be prjfs or nfs. Let's remove FUSE from these headers.
Note, this was hardcoded to read FUSE in python as well but maybe `eden top` in python only supported FUSE file systems?
Reviewed By: xavierd
Differential Revision: D32839825
fbshipit-source-id: f407839b11b54bce4973bad5bbecbf483c3e6604
Summary:
Duration::new() takes in a `u32` for the nanoseconds parameter but `from_nanos` takes in a u64. This hopefully fixes some out-of-range errors that xavierd was seeing:
```
Error: out of range integral type conversion attempted
Caused by:
out of range integral type conversion attempted
```
Reviewed By: xavierd
Differential Revision: D32813694
fbshipit-source-id: 9c05ac32d6c0d2ab8dfda94b101185122e0be064
Summary:
Some scripts are using the return status of edenfsctl to decide whether to
continue or abort their operation. We need to make sure that when an error is
thrown by edenfsctl.real.exe, the Rust CLI exit with the same error code to
make sure that these scripts continue working as expected.
Reviewed By: skotchvail
Differential Revision: D32818993
fbshipit-source-id: 21df2153d469d3a48bfba046a07360726498806c
Summary:
Both of these applications are known to crawl the repository without taking
into account that files are virtual. Let's not allow them to read the
repository at all.
Reviewed By: fanzeyi
Differential Revision: D32809237
fbshipit-source-id: 173857f34596b07f7f594a5efffe9c5b876cc5da
Summary:
When the backingstore throws an error, EdenFS doesn't currently log any trace
to the edenfs log, which makes debugging hard. In particular, we've seen
several cases where outdated certificates is causing EdenFS to hang on
fetching, before finally falling back to the Mercurial importer. This will help
surface these issues more easily.
Differential Revision: D32781950
fbshipit-source-id: 7524b3b640beaa80276ccc73365fc9f31a9f0874
Summary:
There was a subtle bug in D31782916 (87ef50135e) where a wildecarded enumeration may send
the same directory entry multiple times to ProjectedFS. This can happen if the
first directory entry returned to ProjectedFS is not where iter_ initially
points to, which is almost always the case when a wildecard is being used. Thus
the first call to advanceEnumeration will simply find the first element sent to
ProjectedFS and send it again on the next readdir call.
Reviewed By: fanzeyi
Differential Revision: D32742433
fbshipit-source-id: 12904a5d2df4407072b8d775950f92ea93344a73
Summary:
We have not used version one of takeover in year(s?) at this point.
Let's just delete this code now because it makes switching to capabilities
only easier.
Reviewed By: genevievehelsel
Differential Revision: D32226548
fbshipit-source-id: 00b91162ee902b10285a0730c0eeb9ac558ef628
Summary: minor changes to InodeLocks documentation sentence structure and grammar. Also use title case in headings (to match style of other documentation).
Reviewed By: genevievehelsel
Differential Revision: D32625917
fbshipit-source-id: 7635103e1ff8660c7ca9e6c0c01dbbcfa0900f9a
Summary: Other docs use title case for headings. This one should too!
Reviewed By: genevievehelsel
Differential Revision: D32623259
fbshipit-source-id: a569ce1538ad6c963e41a4fa7aef4127f1e2b730
Summary:
fix several small typos and comma placements
Created from CodeHub with https://fburl.com/edit-in-codehub
Reviewed By: genevievehelsel
Differential Revision: D32620443
fbshipit-source-id: e14352534153c40a0c2bb2e4cd868f7cfaa08116
Summary:
For invocations of `ensure!` where the argument is one of the comparison operators `==` `<=` `<` `!=` `>=` `>` and the lhs and rhs of the operator both have a `Debug` impl, we now render both sides as part of the error message.
### Example:
```
ensure!(flags.len() <= 40);
```
```
ensure!(kind == Kind::File);
```
Before:
```
Condition failed: `flags.len() <= 40`
Condition failed: `kind == Kind::File`
```
After:
```
Condition failed: `flags.len() <= 40` (99 vs 40)
Condition failed: `kind == Kind::File` (Symlink vs File)
```
Reviewed By: zertosh
Differential Revision: D32598267
fbshipit-source-id: 70fdab6d8f0e3ceceb98e651eab0a73795cadaeb
Summary:
The TreeOverlay doesn't allow removing a directory that isn't empty, except
when calling the loadAndRemoveOverlayDir. Let's use this one in handleGCRequest
since this function tries to remove the directory before its children have been
removed.
Reviewed By: kmancini
Differential Revision: D32533928
fbshipit-source-id: f6daf3c379730953a4635f40e30efdc8e25a2b90
Summary:
This release helps catch places that the `anyhow!` or `bail!` macros are invoked with a missing fmt argument, such as:
```
- anyhow!("unsupported redirect operation {:?}")
+ anyhow!("unsupported redirect operation {:?}", op)
```
All instances of this in fbcode were fixed in D32266567 (2f440eac7e).
Reviewed By: wqfish
Differential Revision: D32564287
fbshipit-source-id: e9cf5a1376e6c83343de3f3c6ba2925d88334a30
Summary: Mimicking what is done in `eden top`. I'm guessing it's because `FUSE TIME` is typically < 1s and thus becomes not useful if we showed them in seconds. And I'm guessing `FUSE LAST` (last access time) can't also just be in nanoseconds because 1 second passes by quickly considering the default refresh rate is 1 second.
Reviewed By: fanzeyi
Differential Revision: D32367513
fbshipit-source-id: c90e3a64f2a7e9c886f7df2edaee96e0d8355480
Summary:
When compiling with clang, these are triggering warnings, let's silence them to
avoid polluting the console.
Reviewed By: fanzeyi
Differential Revision: D32565964
fbshipit-source-id: ee2dfb71e5bb56d361584b721bd837f9cd249fad
Summary:
Now that takeover is implemented on nfs, we can enable the integration tests
for graceful restart on NFS.
This last test relies on file invalidation, so we implement this as fully as we
can on an nfs mount in order to enable this test.
Reviewed By: xavierd
Differential Revision: D32225243
fbshipit-source-id: c6ddbba54eb71edf6bfe70285a775d438a9d0acd
Summary:
We've seen several reports of users getting confused by the output of `df` and
thinking that EdenFS was using 100+ GBs of disk space. In reality, EdenFS is
simply reporting the stats from the underlying filesystem, and thus isn't a
true representation of the disk space usage.
To avoid this confusion, let's simply pretend that EdenFS doesn't use any disk
space. This isn't exactly true as the overlay and some internal datastructures
are using space already, but `eden du` is the right way to see these.
Reviewed By: fanzeyi
Differential Revision: D32536715
fbshipit-source-id: ec9b27084340d2e69cc5a14d6aa71def5fb4465d
Summary:
Provide a no-arg constructor, for C++20
This avoids the following errors:
```
eden/fs/nfs/rpc/Server.h:234:63: error: no matching constructor for initialization of 'facebook::eden::RpcTcpHandler::State'
explicit StateWrapper(folly::EventBase* evb) : evb_{evb}, state_{} {}
^ ~~
eden/fs/nfs/rpc/Server.h:221:5: note: candidate constructor not viable: requires single argument 'state', but no arguments were provided
State(const State& state) = delete;
^
eden/fs/nfs/rpc/Server.h:223:5: note: candidate constructor not viable: requires single argument 'state', but no arguments were provided
State(State&& state) = delete;
```
Differential Revision: D32521305
fbshipit-source-id: 70d53db2093b41b58d4abc0b20389b0169b7a2e3
Summary:
For restart for NFS we now have the pieces to transfer nfs mounts.
The last piece of restart is transferring the mountd socket. This extracts the
socket from the current mountd and provides it to the takeover server, as well
as implements reading the mountd socket from the takeover client and starting
the mountd with this server socket.
Reviewed By: xavierd
Differential Revision: D31802786
fbshipit-source-id: 1727ea0913bfe1dcb02fdbe27c8cc3b9d62a8464
Summary:
For restart for NFS we now have the pieces for the nfs channel to
perform takeover and for the takeover code to transfer the data.
This commit plumbs everything through so that we are able to stop and nfs
channel on takeover. this means extracting the connected socket and plubming it
to the takeover server.
Reviewed By: xavierd
Differential Revision: D31357367
fbshipit-source-id: 91973f7c149ae9209e85aa40111f41e7753db63f
Summary:
For restart for NFS we now have the pieces for the nfs channel to
perform takeover and for the takeover code to transfer the data.
This commit plumbs everything through so that we are able to start and nfs
mount during takeover. This means starting the NFS channel from an existing
connected socket.
Reviewed By: xavierd
Differential Revision: D31802788
fbshipit-source-id: 9034c1cc93e6bb64c09a203d265cf3bdedd1e160
Summary:
We are working towards supporting graceful restart on NFS so that NFS users will
have this available when we start the migration.
This commit is teaching the Server at the core of the nfsd and mountd to
support takeoverShutdown and takeoverStartup. This means stopping processing
requests when we call takover shutdown and passing the sockets (both server
socket and connected sockets) up the stack.
For now we are only supporting takeover once everything has been initialized
and not during mount or unmount.
This means for the nfsd we will only care about the one connected socket which
is being used to communicate with the kernel. we will pass through the
shutdown promise for the channel. For the mountd we only care about the server
socket which we will return from the takeoverShutdown method itself.
Then we also add being able to initialize itself from a server socket or
a connected socket.
Reviewed By: xavierd
Differential Revision: D31802790
fbshipit-source-id: 06ef4da48cd864bfa20ea0fb98485792255d98cd
Summary:
Now that we have NFS mounts we need to support takeover for those mounts.
This commit teaches the takeover server and client to serialize and deserialize
NFS information.
NFS information includes:
- the mountd socket. Mountd is used to register mount points, and should keep
the same socket across graceful restarts.
- a connected socket for each of the NFS mount points.
- All the rest of the mount point information that we send for fuse mounts (with
the exception of the fuse negotiated parameters.
Future commits will teach the NFS components to graceful stop and yield this info
as well as restart with this info.
Reviewed By: xavierd
Differential Revision: D31802787
fbshipit-source-id: 667625589b77eed79f7e17390013a818d4be7068
Summary:
D32399852 (826c926a47) has a small bug for the Windows case. Was not caught during testing because I tested on Mac/Linux. This should fix it.
Created from CodeHub with https://fburl.com/edit-in-codehub
Reviewed By: kmancini
Differential Revision: D32507465
fbshipit-source-id: 5b79b3931d632b59ce9c3d4361e57569e87da02a
Summary: This doesn't compile on Windows due to the symlink method not being defined.
Reviewed By: chadaustin
Differential Revision: D32502813
fbshipit-source-id: 884c8c902db3ad22aaf7d4ce7debae9fdd2c1eb8
Summary: We no longer support python2, thus we can use python3 constructs directly.
Reviewed By: chadaustin
Differential Revision: D32502816
fbshipit-source-id: d3277ae5a8938ee02a689ffd3bfcf33657b7587b
Summary: Instead of fetching the file size and sha1 of a file separately, we use the new FileInode::getBlobMetadata() method to get them at the same time.
Reviewed By: xavierd
Differential Revision: D32401043
fbshipit-source-id: 7fcedeaf9070931e55d3fd8b23544b8b56118d1a
Summary: adds FileInode::getBlobMetadata() method. Instead of fetching sha1 and filesize separately, we can use this method to get them both at the same time
Reviewed By: xavierd
Differential Revision: D32399852
fbshipit-source-id: 6b4e271136492a7fa836a49c3794347a5ebab9ef
Summary: Used codemod to modify all references to eden/fs/store/BlobMetadata.h. Changed them to eden/fs/model/BlobMetadata.h and moved the file to the model director
Reviewed By: genevievehelsel
Differential Revision: D32397203
fbshipit-source-id: 0c17b96d1aa33ba32fefe43ea5c29a645c577765
Summary:
Adds logic for new thrift endpoint that allows users to request attributes for a given list of files.
Requested attributes are obtained for each file and stored in a FileAttributeData struct. Each file is then mapped to a FileAttributeData struct (or an EdenError). If a requested attribute cannot be found for a given file, an EdenError is returned instead of a FileAttributeData.
Integration tests cover too many cases, so I've commented out the excess cases and will turn them into unit tests in a later diff. For now, they are available for use, but not included by default.
Integration tests cover the following cases:
- Requesting all attributes for 2 files
- Requesting attributes for a non-existent file
- requesting just file_size for a file
Commented out Integration tests include:
- Requesting attributes for empty string path
- Requesting attributes for a file that contains dot components
- Requesting attributes for a directory
- Requesting sha1 for a symlink (should file_size for a symlink also return an EdenError?)
- requesting all attributes for no files
- requesting no attributes for a file
- requesting just sha1 for a file
Reviewed By: kmancini
Differential Revision: D32179305
fbshipit-source-id: 6c23bb7780154159fdd9626d332c641311afbe57
Summary:
update thrift interface to include a function that returns attributes for a given list of files.
to support this, we add several structures that define the functions parameters and return type.
- FileAttributes: a "bitmask" enum. Each attribute is a power of 2 to enable bitwise ORing attributes together to decide which attributes to fetch from files.
- FileAttributeData: struct to hold the attributes of an individual file
- FileAttributeDataOrError: union that is either a FileAttributeData or EdenError thrown when accessing the file attributes
- GetAttributesFromFilesParameters: self explanatory.
- GetAttributesFromFilesResult: self explanatory. Map of file -> FileAttributeDataOrError
- getAttributesFromFile(): actual function that will be implemented in EdenServiceHandler.cpp
Reviewed By: chadaustin
Differential Revision: D32155168
fbshipit-source-id: 6dd768f3bcee1e32a2362df231feebfb1297dbf6
Summary:
Cast EDEN_MICRO to something fmt::format can handle: char* (from std::basic_string<char8_t>); but that means this is no longer a literal, so cannot concat. Move to its own fmt argument.
This avoids the following errors:
eden/fs/service/EdenServiceHandler.cpp:227:60: error: invalid operands to binary expression ('basic_ostream<char>' and 'std::basic_string<char8_t>')
Differential Revision: D32486517
fbshipit-source-id: c799bdd73be1b46cc938b2076ebc78d0bdb3dded
Summary:
We should all be migrating to platform010, for the improved performance of its
generated code and for its improved diagnostics/portability.
reinterpret_cast another u8"..." literal to const char*
This avoids the following errors:
```
stderr: In file included from eden/fs/utils/test/Utf8Test.cpp:9:
In file included from folly/portability/GTest.h:32:
third-party-buck/platform010/build/googletest/include/gtest/gtest.h:1545:11: error: invalid operands to binary expression ('char8_t const[4]' and 'const std::basic_string<char>')
if (lhs == rhs) {
~~~ ^ ~~~
third-party-buck/platform010/build/googletest/include/gtest/gtest.h:1572:12: note: in instantiation of function template specialization 'testing::internal::CmpHelperEQ<char8_t [4], std::basic_string<char>>' requested here
return CmpHelperEQ(lhs_expression, rhs_expression, lhs, rhs);
^
eden/fs/utils/test/Utf8Test.cpp:43:3: note: in instantiation of function template specialization 'testing::internal::EqHelper::Compare<char8_t [4], std::basic_string<char>, nullptr>' requested here
EXPECT_EQ(u8"\uFFFD", ensureValidUtf8("\xff"));
```
Differential Revision: D32484896
fbshipit-source-id: b271292eb121e0e183bec56ab51fc6acb791b052
Summary:
On Windows, directory removal notifications can be received out of order, and
sometimes duplicated. The next diff will rework the way EdenFS handle
notifications on Windows by looking at the on-disk state and applying it to the
inode hierarchy. In some cases, an entire directory hierarchy will need to be
removed which the added function will make it easier to do.
Reviewed By: chadaustin
Differential Revision: D32329806
fbshipit-source-id: ad9af389c993420184c0e59dd3d25188adddb77a
Summary: Have constructor do init() instead. I can't think of any valid reason to have this API other than confusing everyone who decides to use it.
Reviewed By: praihan
Differential Revision: D32302228
fbshipit-source-id: 136e0d2919291acd60be953337309cd006bd674b
Summary: Sha1 computation show up in many profilers, we can skip this step entirely if we are using metadata from hg.
Reviewed By: xavierd
Differential Revision: D32396312
fbshipit-source-id: 928f8458a91bba4d18da6730d3fc244eb5499489
Summary:
This diff allows eden to fetch blob metadata from hg cache.
Combined with previous diffs in stack, it allows to entirely skip scs metadata import, saving whole additional network round trip for operations like ls -lr or parsing buck files.
This diff adds hg:use-aux-metadata config that controls this behaviour.
When config is set, we instruct hg backing store to fetch file aux metadata along with get tree request.
Later when blob metadata is requested we are checking hg cache to get the metadata.
For trees that were imported prior to this config we fallback to reading scs metadata cache.
In terms of performance effect, this diff reduces traffic on laptops during listing/stating files significantly, because x2p client sends significant amount of extra data.
It does not improve end to end speed significantly though, because scs queries run in parallel with tree request.
Before
```
ls -lR fbsource.eden/fbcode/admarket 0.97s user 8.02s system 3% cpu 4:47.42 total
Traffic usage: 77Mb 25Mb down, 52Mb up
```
After
```
ls -lR fbsource.eden/fbcode/admarket 0.98s user 8.19s system 3% cpu 4:33.65 total
Traffic usage: 40 Mb, 22Mb down, 18Mb up
```
On laptops with worse network this likely to improve speed too, because some networks have much lower uplink latency
Additionally, this allows to efficiently use metadata on windows (where previously entire blob needed to be downloaded in order to calculate blob size)
Reviewed By: xavierd
Differential Revision: D32371710
fbshipit-source-id: 8ac962ea0136bc3bc4e30d0bf31692e65c35c43f
Summary:
When unwrapping Try it is convenient to be able to build an ImmediateFuture
directly from the .exception() call.
Reviewed By: chadaustin
Differential Revision: D32333774
fbshipit-source-id: 0066ca7eddc4a57010f60974c234b57691b73d77
Summary:
For some reason, in some cases the compiler wants to use the folly::collectAll
function instead of the ImmediateFuture one. Adding the namespace solves this
issue.
Reviewed By: chadaustin
Differential Revision: D32329807
fbshipit-source-id: 22ee9b20cf33e5718c9bb0270dd93f3f857b0967
Summary:
Previously it raised
``` raise TTransportException(
type=TTransportException.NOT_OPEN, message="eden not running"
)
```
even if it wasn't thrift and it wasn't eden.
Now it will raise OSError (as regular sockets do), and translate it to TTransportException if it's used as a thrift socket
Reviewed By: DurhamG
Differential Revision: D32171878
fbshipit-source-id: 290368acf8def5aec812abc88c23a946e7a0f1f4
Summary:
So that I can re-use it in the next diff.
I considered putting this in a util/ subfolder but when doing the `use` imports, this caused issues since there was another crate called util and would require renaming all those use statements.
Reviewed By: fanzeyi
Differential Revision: D32331290
fbshipit-source-id: 78f9bf94bcf6426f1f0f23761061161e3e392832
Summary: As fanzeyi pointed out, the Rust compiler can correctly infer that this code is unreachable (since compiler warnings pop up) and we don't need to explicitly write unreachable.
Reviewed By: fanzeyi
Differential Revision: D32295516
fbshipit-source-id: d8562fab7a4d012db5fe91c4947558531e512144
Summary:
This is because the save / restore cursor position sometimes behaves differently depending on the command that I run minitop with (e.g., it sometimes will re-print the first row repeatedly if the saved cursor position was somehow wrong).
I also explored calculating how much to truncate by getting the screen size, but this was simpler.
We can add this back in if we switch to an "alternate screen".
Reviewed By: fanzeyi
Differential Revision: D32289941
fbshipit-source-id: 3edda78ab2aa82958f6a40efd03e302520f84f4b
Summary:
There is a race in here. We read the config, then mutate it, then put the new
config in place. However, if something else comes and mutates the config
between the time where we read it and write it again, we'll clobber its
changes.
Concretely, if you're running lots of `eden clone` or `eden rm` concurrently,
then the current mechanism will sometimes "forget" some directories or add back
directories you removed, which will e.g. break `eden list` (and cause those
commands to sometimes fail as well).
This diff fixes that by wrapping the critical section with a file lock.
Reviewed By: chadaustin, xavierd
Differential Revision: D32251727
fbshipit-source-id: 9eee57c7a824286bb4f0e7911e5ccd9d769c6daa
Summary:
There was confusion in previous diff with some proxy hash interface taking optional<> and some taking nullable pointer, resulting in cretion of optional with null pointer inside.
To avoid this confusion I am just going to use optional<> everywhere.
Reviewed By: chadaustin
Differential Revision: D32324008
fbshipit-source-id: 2f0df1f50d032dd0c8823a1671a69ff74435eb3b
Summary:
On takeover, the new EdenFS sends the current mount point over to the
privhelper, and due to being a suid root process, the privhelper needs to
double check that the given mount point is valid before accepting it. In order
to do this, a `stat` is ran on the mount point.
Unfortunately, this `stat` call may need to perform a lookup and getattr FUSE
calls, but since EdenFS hasn't yet initialized FUSE, it may hang, leading to
EdenFS not being able to restart.
To fix this, we can simply start FUSE prior to advertising the mount point to
the privhelper, this way the lookup and getattr FUSE calls can be answered.
Reviewed By: kmancini
Differential Revision: D32322661
fbshipit-source-id: 4ea1e44ce362e04ed7eebaff1f2a8f381f8bff60
Summary:
Do not hide cursor because when you stop the process, it leaves the cursor as disabled.
Even when this was disabled, you could still type anyway. Better to use alternate methods to disable this.
Reviewed By: fanzeyi
Differential Revision: D32254638
fbshipit-source-id: ce76b8943aea5f15eb35a263afc1b67593c2ba12
Summary:
I noticed that I'm hitting this assertion. Likely because the running process changes what commands it's running. Let's remove this assertion.
This is the same as the python `eden top`'s behavior where they just continue to aggregate and never update the `cmd`.
Reviewed By: fanzeyi
Differential Revision: D32173253
fbshipit-source-id: 72bc4710e585fa800d7d96c04fa9a88f13a8557d
Summary:
The QueuedImmediateExecutor can have surprising effect, let's instead move the
futures to the global CPU executor, but only if the ImmediateFuture isn't
ready, ie: when there is still some computation to be performed. The use of
folly::detachOnGlobalCPUExecutor also makes it obvious that the intent is to
run the future elsewhere in the background.
Reviewed By: chadaustin
Differential Revision: D32155651
fbshipit-source-id: 8ae5be48230ff44c58a5697d3824825eeb8235dc
Summary:
In order to compute the sha1 of materialized files in the working copy, EdenFS
on Windows will open and read the file out of the working copy directly. In the
case where EdenFS is wrong about the state of the file, this can lead to a
ProjectedFS callback to be triggered which may need to take the same lock that
the sha1 computation holds. This obviously results in a deadlock and to protect
against this, EdenFS would bail early when detecting recursive callbacks.
With read-only callbacks having been switched to only take into account the
underlying Mercurial data, and not the inode state, these callbacks will never
attempt to take any inode locks. This also means that the deadlock described
above can no longer happen: the ObjectStore layer will never attempt to read
anything from the working copy, thus recursive callbacks are impossible. These
checks are thus unecessary.
In theory, these checks could stay in place as a safety measure, but EdenFS
will soon need to list directory entry and call PrjGetOnDiskFileState in
response to file/directory creation/removal to handle the case where these can
happen out of order.
Reviewed By: genevievehelsel
Differential Revision: D32149411
fbshipit-source-id: 73b228bb8c73416b38f7f2042c25964e1aef3c4e
Summary:
We already had a collectAllSafe for argument packs, let's have a similar
function when the argument is a vector.
Reviewed By: chadaustin
Differential Revision: D32323573
fbshipit-source-id: 2cfa3da565a7d9bb88746e3cc6b10d1d8890c05e
Summary:
This config has been enabled for many months now, let's simply assume it is set
to true on Windows.
Reviewed By: genevievehelsel
Differential Revision: D32192585
fbshipit-source-id: 76829a031f859594a5139dd09a6479cabeda2089
Summary:
This diff introduces new format for serializing trees in local store.
Unlike currently used git serialization format, new format supports variable length object ids.
Long term this will be replaced with using mercurial as a source of truth, but for now we need this to roll out proxy hash replacement.
Reviewed By: chadaustin
Differential Revision: D31998935
fbshipit-source-id: 9aacfbea631e75c0ea1421094a0b1ae255adb04a
Summary: If toml values are maps, we were overriding the previous values without merging them. This makes `eden-config` to incorrectly discard values that should have been kept.
Reviewed By: genevievehelsel
Differential Revision: D32288269
fbshipit-source-id: e9c9f6a1aed3b0de719a046e56344cab763a3e7e
Summary: I wasn't sure why I didn't make them a test when I wrote them. Let's make them a test and continuously run them to make sure `stack-config`'s implementation is correct.
Differential Revision: D32288267
fbshipit-source-id: 988f65993de0947ce408a6809b32b3ece5e9b3a3
Summary: When debugging, it's nice to know what the kernel thinks the mount table is.
Reviewed By: xavierd
Differential Revision: D32288215
fbshipit-source-id: d4c15b16689f20846a45f682c4089e2fa8250661
Summary:
In ProjectedFS terminology, the "backing store" refers to the state of files
and directories in the source control backend. For the longest time, EdenFS
assumed that the Inodes hierarchy was the "backing store", but this lead to
subtle bugs. For instance, renaming a non-hydrated placeholder would render the
file unreadable due to the read callback from ProjecedFS callback using the old
file name, not the new one
(https://github.com/microsoft/ProjFS-Managed-API/issues/68). It also meant that
files just created would always get a tombstone written on disk when being
removed as ProjectedFS queries EdenFS when removing a file via the access
callback. Returning true from this callback means that this file is present in
the backing store and thus ProjectedFS needs to create a tombstone on disk.
In order to fix these issues, EdenFS needs not to query the Inodes hierarchy,
but the Tree hierarchy. The latter matches with the expected behavior from
ProjectedFS.
For modification callbacks, these are by design acting on the Inode hierarchy
and thus are unchanged.
Reviewed By: chadaustin
Differential Revision: D32022639
fbshipit-source-id: bef18a0709a7eff9ef0a48a90126d1339be0023e
Summary:
Update tracing-subscriber crate to 0.3.1. Also add some features that David Barsky suggested.
Also update tracing-appender since 0.1.x is incompatible.
Reviewed By: zsol
Differential Revision: D32175169
fbshipit-source-id: ad714db1c1c67b508be1c97d3dc960dd7d809cc6
Summary:
We should all be migrating to platform010, for the improved performance of its
generated code and for its improved diagnostics/portability.
add an explicit constructor, now required for C++20, see https://en.cppreference.com/w/cpp/language/default_constructor
This avoids the following errors:
eden/fs/utils/test/ImmediateFutureTest.cpp:335:17: error: no matching constructor for initialization of 'Uncopyable'
# Context:
The cost of delaying a platform upgrade is now estimated at about $3M/week.
For the preceding one, it was measured at [**$1M/week**](https://fburl.com/jfmpsqmy).
Differential Revision: D32250064
fbshipit-source-id: 8c6990489cafb09629c03e56bdbae92395d6f85f
Summary:
This diff introduces config store:embed-proxy-hashes.
When this config is set, we store HgId directly into ObjectId, instead of using proxy hash object.
This allows to bypass proxy hash rocks db storage when reading files.
**Compatibility notes**
This diff is compatible with previous versions unless store:embed-proxy-hashes config is set.
Once config is set, new ObjectId format is used and serialized into inodes. Once this is done previous versions of eden fs won't be able to read overlay inodes created by this version.
This means we need to be careful with setting this config - once set we won't be able to roll back eden fs version easily, it will basically require re-creating eden checkout.
Inodes created prior to this config being set will remain written in old format, only when new inode is written is when new format is used.
**Git tree format issue**
We use git tree serialization format in the LocalStore to serialize trees.
This format assumes 20-byte hashes and is not compatible with variable length ObjectId.
In this diff we bypass this issue by not storing trees into local store. This seem ok in terms of correctness, because tree information can always be fetched from mercurial.
However, this seem to impose performance penalty on some work loads (see below).
We can solve this by either introducing new format that supports var length object id(short term), or by getting rid of tree cache and efficiently getting the data directly from mercurial(long term).
**Performance numbers**
Hot file access time is reduced by 50%:
```
$ fsprobe.sh run cat.targets
Before:
lat: 0.2331 ms, qps: 4, dur: 28.697384178s, 123092 files, 217882490 bytes, 1641 errors, rate 7.59 Mb/s
After:
lat: 0.1611 ms, qps: 6, dur: 19.835917353s, 123092 files, 217882490 bytes, 1641 errors, rate 10.98 Mb/s
```
However, we do not see improvement with arc focus, most likely due to bypassing tree serialization, so we will need to figure out that issue.
We can still merge this diff and see if enabling this feature on other workloads like sandcastle is benefitical.
Reviewed By: chadaustin
Differential Revision: D31777929
fbshipit-source-id: fc4b678477d0737c9f242968f0be99ed04f4f58a
Summary:
Hide the cursor and disable the line wrap
Some issues I've still noticed:
- If you resize the window to be smaller it messes things up (you can resize the terminal to be larger and it works fine)
- Sometimes it will decide to continuously print instead of overwriting the old text. Not sure why exactly this happens. It happens more often when I run `buck run` rather than the underlying binary.
- You can still type even though the cursor is disabled
I suspect to address this, I need to rely on the `event` module in the `crossterm` crate.
Reviewed By: fanzeyi
Differential Revision: D32114319
fbshipit-source-id: 037dc1b7e4fbba6e624b00210b4cb07b8d0644c9
Summary:
This diff gets access counts and prints them out in a simple table laid out with tabs.
Still a lot to do, but wanted to get this code out there to get feedback earlier. Some things I need to do:
- format last access time
- sort by last access time
- laying out in a nicer table (I plan on using comfy-table: https://fburl.com/code/k353fp6i)
- updating every `refresh_rate` seconds (I plan on using cross-term: https://fburl.com/code/ahuoqy7t)
Reviewed By: fanzeyi
Differential Revision: D32000885
fbshipit-source-id: 6fc60d13e9519ed45ad6e18628013f9b3ac88bc7
Summary:
The makeImmediateFuture is merely a glorified try/catch, and thus the entirety
of the readdir callback would run immediately, and would block on every size
sequentially. This is inefficient and may cause files to be fetched one at a time.
By reworking the inner logic, we can collect all the sizes beforehand prior to
servicing the callback, allowing for sizes to be fetched concurrently, and for
completing the callback asynchronously.
Reviewed By: chadaustin
Differential Revision: D31782916
fbshipit-source-id: d6315347492e969ffa79037dc2a4f275f4b95a8d
Summary:
Since the CWD wasn't specified when gathering the Buck version, the buck
version returned by `buck --version-fast` could be different from the one in
use in the subdirectory where `buck kill` would be called. This would lead to
Buck failing to stop itself, which would interrupt `eden stop`.
Reviewed By: kmancini
Differential Revision: D32193032
fbshipit-source-id: cbe4983df785b4439645037f9d42084cf745a423
Summary:
The original bug has been fixed for over a year now, let's remove the
workaround.
Reviewed By: kmancini
Differential Revision: D32193033
fbshipit-source-id: 29de7a52e47f8fe070b45b785756b4c93e42151c
Summary: The same as the option that is available in `eden top`.
Reviewed By: fanzeyi
Differential Revision: D31863321
fbshipit-source-id: 3571cc4866fc43e5f43dc5f390cf737b33a7c0b4
Summary:
C++20 changed the rules around char8_t. Cast to char* as
necessary. Also, fix references to some removed std::allocator
associated types.
Reviewed By: fanzeyi, meyering
Differential Revision: D32155762
fbshipit-source-id: dc72e5bc58e8c08612ea1a1eb2568861701287c6
Summary: It is conflicts with CTIME from libc when sys/ioctl.h is imported
Reviewed By: johnny90
Differential Revision: D32017643
fbshipit-source-id: 0346674a7a215524ccaf7cce0e5dfafb5ea9f288
Summary:
On Windows, the path returned to the PRJ_GET_PLACEHOLDER_INFO_CB callback must
be canonical and may differ from the passed in path. Currently, this callback
is implemented by walking the Inode hierarchy, but this is being changed to
walk the Tree hierarchy. In the case of Inode, the getPath method can be used,
not for trees. Therefore let's add a method that walks the Tree hierarchy and
returns the proper cased path.
Reviewed By: chadaustin
Differential Revision: D32022637
fbshipit-source-id: fbf14ef806d3ca0d356a0a4d5374d4a48663d005
Summary:
On Windows, the expected behavior from ProjectedFS callbacks is that they query
the "backing store" state. On EdenFS, we have so far assumed that the "backing
store" was EdenFS Inodes, but EdenFS's backing store state is actually what
needs querying. In order to do so, the ProjectedFS dispatcher must be able to
query the backing store from a path, which is what getTreeOrTreeEntry does.
This is similar to the EdenMount::getInode but acting on the backing store
Tree.
Reviewed By: chadaustin
Differential Revision: D32022636
fbshipit-source-id: df1dfe09740dcf3ab36d318a850e9e7e4942be60
Summary:
Only the TestMount calls this function for now, let's move the null context
creation over there.
Reviewed By: chadaustin
Differential Revision: D32022638
fbshipit-source-id: 9ddec23c4c94640bde90e1e8ba197d0118546171
Summary: Bring the config in line with the oss branch names
Reviewed By: mzr
Differential Revision: D32109171
fbshipit-source-id: f87a0070b0ef06a49c86cacdfdc7f5c8b02f3fbf
Summary:
EdenFS NFS does not support filesystem getxattr calls. This means for now tools
that rely on sha1 will need to use thrift. To understand how big of a regression
this is I am benchmarking the difference between reading files via thrift and
filesystem.
We already have a benchmark for measuring thrift sha1 accesses, so I am
extending this to support xattrs through the filesystem as well.
Reviewed By: chadaustin
Differential Revision: D31161699
fbshipit-source-id: 9f6b1d98de18c9ca720cfa18c7094d322ae4161a
Summary: This fixes the recursive call we saw on Windows for edenfsctl.
Reviewed By: xavierd
Differential Revision: D32113829
fbshipit-source-id: d87c6628afe668dd63b6ee908f178470656b9630
Summary:
The privhelper can be tricked to mount or umount in places of the FS that the
user doesn't actually have access. This can be misused to mount or unmount to
privileged paths.
When accepting mount points from EdenFS, let's verify that the user actually
have access to them first.
Reviewed By: chadaustin
Differential Revision: D31875209
fbshipit-source-id: 939bf8f519adb63ce54f91c7b0757a7486cabf51
Summary:
AsyncServerSocket works like this:
1) a socket is accepted from the fd
2) the socket is pushed into the socket queue
3) the connectionAccepted() callback is called with the socket is dequeued
Things happening before connectionAccepted() are like black box to us currently. For example the time taken for a socket to sit in the socket queue is completely unknown. Adding this AcceptInfo into the callback could allow us to pass more information to downstream for instrumenting.
This diff only pass timestamp of before the socket is being pushed into the queue.
Reviewed By: yfeldblum
Differential Revision: D31633582
fbshipit-source-id: 66072955318388c2410b48313d3d54078edce478
Summary:
This commit is introducing capabilities for the takeover code. This will make it
easier to evolve the protocol. Which I am about to do to support NFS.
Why is capabilities worth adding?
Currently takeover uses a versioning based approach to evolve the takeover
protocol. This works and is pretty simple. The downside is that it leads us to
making lots of checks in the code base like: if this version then do x.
There are two downsides of such checks:
1. reading the code is not self contained. You have to know or have already
read which version introduces which features for the checks to make sense. This
generally makes the code harder to read and harder to check in code review.
2. introducing new versions sometimes mean we need to update those checks we
previously added. For example we make this check in the server on wether to
send a ping message before sending data:
```
shouldPing_ =
(protocolVersion_ == TakeoverData::kTakeoverProtocolVersionFour);
```
if we add a kTakeoverProtocolVersionFive where we still want to send the ping (
which we likely do because the ping is a part of the protocol correctness under
failures), then we need to update this time to
```
shouldPing_ =
(protocolVersion_ >= TakeoverData::kTakeoverProtocolVersionFour);
```
then if we replace the ping with something else we would have to again update
the code to
```
shouldPing_ =
(protocolVersion_ >= TakeoverData::kTakeoverProtocolVersionFour
&& protocolVersion_ <= lastTakeoverVersionWithPing);
```
The first of these code updates is very easy to skip accidentally if you are
making an update to the protocol that has nothing to do with ping. This would
silently introduce a regression.
Using capabilities instead makes this check
```
shouldPing_ =
(protocolVersionCapabilities_ & TakeoverCapabilities::PING);
```
This is more readable. and this code never has to be updated; we just don't use
the ping capability any more.
Eventually, if we go down the capabilities route we will want to kill off
versions completely. This will mean bumping the version of the protocol, where
we agree on capabilities after agreeing to that version. After this version is
stable (released to production and a good track record indicating we don't need
to rollback prior to the change), we can delete the older versions of the
protocol and all the version based stuff in the code. This stabilizing process
will take a while, so I do not want this to block the NFS evolution I am doing
in the next change. I will comeback to this after we get NFS support in graceful
restart.
Reviewed By: xavierd
Differential Revision: D31965080
fbshipit-source-id: f8413d07da42a92125d46aa3cfbd3fa458b87b46
Summary:
When running the Eden tests using buck2, this flag is used because the
privhelper is not immediately next to the EdenFS binary.
This diff updates the approximation of the privhelper path introduced in
D31870880 (710f03501d) accordingly so that it also works in this case.
Reviewed By: xavierd
Differential Revision: D32070038
fbshipit-source-id: c111cc533bb9e9d111bfe9bad71e817655b3d613
Summary:
The privhelper has "fuseTakover" methods but there is really nothing fuse about
it. This commit just renames the "fuseTakover" things to "takover" things to
sound generic. We will use these for NFS mounts in a later commit.
Reviewed By: xavierd
Differential Revision: D31802785
fbshipit-source-id: 77263d5e4a55541db1aa9d295b5ad93912d8488b
Summary:
Takeover data needs to include fuse data for fuse
mounts and NFS data for NFS mounts.
Fuse related data is already wrapped up in
FuseChannelData. we are making an NfsChannelData to
mirror this for NFS. These structs will be used in
TakeoverData in a later commit.
Reviewed By: xavierd
Differential Revision: D31740475
fbshipit-source-id: f64367a5881175b9e20d261e43ec7570c498f380
Summary:
Some compiler (msvc, old gcc) can't seem to figure out that these switch cases
are exhaustive and thus warn about them. Let's help it to silence the warnings.
Reviewed By: chadaustin
Differential Revision: D32022198
fbshipit-source-id: 67f89d52211db45a09e8a2fc0ed64da667b8cbbc
Summary:
`socket.setblocking` is required by HTTPConnection http client in Mercurial. Commitcloud is using it when using the x2pagentd.
Commitcloud on x2pagentd is gated behind `--config auth_proxy.commitcloud_use_uds=True` which isn't rolled out anywhere right now.
Reviewed By: fanzeyi
Differential Revision: D31860864
fbshipit-source-id: 7b8864bb81afb30aa32a37f6dfe91f5e7806d8ee
Summary:
On Windows, overlay data is never recursively removed, causing it to grow
unbounded. This may be the root cause of slow startup that some users have
reported. Since there is nothing Windows specific about this code, let's enable
it.
Reviewed By: fanzeyi
Differential Revision: D31834657
fbshipit-source-id: 852968cf803119a494f0eb71ba77f392b64437c3
Summary:
As the name implies, it loads an overlay directory and removes it at the same
time. This is done to bypass calling removeOverlayData when recursively
removing a directory as removeOverlayData would check if the directory no
longer has childrens and thus would fail.
Credits for this diff should go to chadaustin.
Reviewed By: genevievehelsel
Differential Revision: D30843607
fbshipit-source-id: 370491e454990ca5c8215501cd7e1d5e85526cce
Summary:
Some users have reported slow startup, this log will allow us to more easily
identify if time is spent initializing the InodeMap.
Reviewed By: genevievehelsel
Differential Revision: D31806744
fbshipit-source-id: f8b3122909058ab8f6859c661ef9615f390966e2
Summary:
To avoid any risk of SSLKEYLOGFILE being used to unexpectedly write
files as root, ensure the privhelper does not even link against
OpenSSL.
Reviewed By: fanzeyi
Differential Revision: D31944839
fbshipit-source-id: 3280657a10f4a468ddb63a874d38198bd333e01c
Summary:
Futures can only be waited on once, but this code can potentially try to wait
on it more. If for instance the size future were to fail, the continuation
would never be called, and thus the next call to getSize would obtain an
already moved future.
For now, let's simply use a FutureSplitter to close this bug. In the future,
the readdir code will be rewritten to be more asynchronous and to avoid this.
Reviewed By: genevievehelsel
Differential Revision: D31782917
fbshipit-source-id: 9e78fdec62dbccbc2796722175936d563a20ebcd
Summary:
Ignoring the return value of this method is equivalent to dropping the future
entirely, which isn't what is usually desired. Let's make sure the compiler
helps us in finding these cases as code is converted.
Reviewed By: chadaustin
Differential Revision: D31945362
fbshipit-source-id: dd646365e00d3ebb04ed9b6803c561fc11ff4a14
Summary: This completes the conversion to ImmediateFuture for Prjfs.
Reviewed By: chadaustin
Differential Revision: D31746629
fbshipit-source-id: 2587452da9032514c86f54a16194edbda2cf546d
Summary:
This moves all the conversions to folly::Future one level up. This allow
removing all folly::Future code from PrjfsDispatcherImpl.
Reviewed By: genevievehelsel
Differential Revision: D31746626
fbshipit-source-id: cb9b78c9ea2f09045f4bc921ea820a77b5832ddb
Summary: This simply moves the conversion to folly::Future to the caller.
Reviewed By: genevievehelsel
Differential Revision: D31746632
fbshipit-source-id: e7a5606f6027cde071c3fed5ab1fc51dcfeb1258
Summary: This simply moves the conversion to folly::Future to the caller.
Reviewed By: genevievehelsel
Differential Revision: D31746631
fbshipit-source-id: b9542ed81564d4e9ff255b7cb4a3fe74a6b3046e
Summary: This simply pushes the conversion to the caller.
Reviewed By: genevievehelsel
Differential Revision: D31746635
fbshipit-source-id: d562f139eb6bff910fcc934a0177582d1ae5c479
Summary:
In order to execute the edenfs_privhelper, EdenFS needs to be a suid root
binary, but due to the large amount of dependency, it's hard to fully control
what runs before main and thus what executes with root privileges.
To reduce the attack surface, let's only make the edenfs_privhelper a suid root
binary. In tests, we still start EdenFS with sudo, and thus the
edenfs_privhelper will have the proper privileges.
Reviewed By: chadaustin
Differential Revision: D31870880
fbshipit-source-id: 3f06d22ecd4f3bc65e3e955c5a1de811b1cf328f
Summary: This simply moves the conversion to folly::Future one level up.
Reviewed By: genevievehelsel
Differential Revision: D31746628
fbshipit-source-id: b64c17eefe949623f6b532d8d86aa703b65c99ad
Summary: This simply move the conversion to folly::Future to the caller.
Reviewed By: genevievehelsel
Differential Revision: D31746627
fbshipit-source-id: 944686bb5995d337e6cf66c13a3cd630ace74a6b
Summary: This moves the conversion to folly::Future up to the caller.
Reviewed By: genevievehelsel
Differential Revision: D31746630
fbshipit-source-id: 522169af5f7b978b88ebc8596558bd5c9b3c2482
Summary: This moves the conversion to folly::Future one caller up.
Reviewed By: genevievehelsel
Differential Revision: D31746633
fbshipit-source-id: 1af5f4f36c33da9248138ada112e277db239b0b1
Summary:
This merely moves the conversion from ImmediateFuture to Future in the
PrjfsChannel code.
Reviewed By: genevievehelsel
Differential Revision: D31746634
fbshipit-source-id: 9978c834e20f17c456322a32deaaa4c13e0cb270
Summary:
The timeout based notification mechanism appears to be at the root of a lot of
EdenFS crashes. It also doesn't detect all of the network failures and is being
replaced with a more robust mechanism which will handle these cases.
For now, let's just get rid of the timeout based mechanism.
Reviewed By: fanzeyi
Differential Revision: D31739683
fbshipit-source-id: d6baa03fe3579dc36c9f435bd16df39799e51ef9
Summary: This removes a bunch of conversion to a Future.
Reviewed By: genevievehelsel
Differential Revision: D31703355
fbshipit-source-id: 229b5fd55551a6d0b1346b8452fd1e57cb40e940
Summary: This merely pushes the conversion to a Future to the caller.
Reviewed By: genevievehelsel
Differential Revision: D31703354
fbshipit-source-id: 16446b4a03f18168af97134f08d374526c81adcd
Summary: This allows pushing the transformation to a Future to the caller.
Reviewed By: chadaustin
Differential Revision: D31703356
fbshipit-source-id: 7e39b5d5862a7c8a2de646de515c5846a81c458a
Summary:
This simply bubbles up the ImmediateFuture to the caller, who already wanted an
ImmediateFuture, saving on conversion.
Reviewed By: kmancini
Differential Revision: D31703353
fbshipit-source-id: 698b3fe0e10a3fce919abd744a4ca3a4f2ff3878
Summary: This allows to remove some conversion from ImmediateFuture to folly::Future.
Reviewed By: chadaustin
Differential Revision: D31674457
fbshipit-source-id: 3e044776a3e9eb467131e309353f24774a13e549
Summary: As discussed in previous diff, this removes constructor from hex string and adds ObjectId(fbstring) constructor
Reviewed By: chadaustin
Differential Revision: D31841234
fbshipit-source-id: c36ae315ad3a6eaecfd47889588c2bd18928aafb
Summary:
This mimics the one eden/fs/utils/Future.h behavior where all the Try are
unwrapped to simply return a tuple.
Reviewed By: kmancini
Differential Revision: D31668773
fbshipit-source-id: f4b9b6b2564cc1e2c43aa5b03bfb06048f0a39c1
Summary:
In some places, EdenFS uses folly::collectAll passing futures to it. In order
to replace it, we need an equivalent for ImmediateFuture.
The implementation relies on the fact that ImmediateFuture's constructor will
extract the value from a SemiFuture if it is ready, and thus the code doesn't
have to special case when all the passed in ImmediateFuture are immediate
values.
Reviewed By: chadaustin
Differential Revision: D31667897
fbshipit-source-id: ea680e1c74528d1eb34210cacb58efda8a26fee0
Summary: This diff supports variable length hashes in serialized tree metadata. See comments to serialize/deserialize functions for description of the format
Reviewed By: chadaustin
Differential Revision: D31701196
fbshipit-source-id: 24d7630d4574842e3888b56b7291e313d8e35e55
Summary:
This diff modifies ObjectId structure to support storing more information directly in ObjectId, avoiding proxy objects like HgProxyHash and ReCasDigestProxyHash.
Storing this information directly in ObjectId allows to avoid additional db access for loading/storing those proxy hash objects, reducing file access latency in hot case.
**New ObjectId format**
ObjectId can now content variable length hash.
The variable length can be used in following cases:
(1) To replace ReCasDigestProxyHash, extra 8 bytes can be used to store size portion of `remote_execution::TDigest`
(2) To replace HgProxyHash, extra 1 byte can be used to identify whether ObjectId represents ProxyHashId or an HgId.
In the future, ObjectId can contain path or other information needed to implement ACL
**Compatibility notes**
For compatibility reasons, we only currently initialize ObjectId with 20-bytes content.
This is essential to allow smooth migration to a new format
(a) Until new hash format is actually used(e.g. we switch HgBackingStore to store HgId inside ObjectId), this code is backwards compatible with other EdenFs versions. This revision can be safely rolled back and previous version of EdenFs can still read inode data written by this version
(b) When we introduce support for new ObjectId into HgBackingStore, we can gate it's usage behind config, allowing controlled slow roll out of new ObjectId format.
**ToDo**
Just to track progress of removing proxy hashes, few things are still left:
* We need different format for SerializedTreeMetadata
* We need to support new format for thriftHash
* We need to actually switch HgBackingStore to embed HgId information into ObjectId, instead of using ProxyHash
Not planned:
* Migration of ReCasDigestProxyHash - I don't know how to test it, so someone else should probably do that
Reviewed By: chadaustin
Differential Revision: D31668130
fbshipit-source-id: 720127354c648651bb35e850beb8dd252a5566b2
Summary:
Previously we had magical hashFromThrift that determined input format based on the length
With variable length object ids that does not longer work, so we introduce separate functions and pick whichever to use based on the use case
Reviewed By: chadaustin
Differential Revision: D31812539
fbshipit-source-id: dce7265bb434889fd2a64ff72d81ff104df88b57
Summary:
As a first step, as I ramp up on Rust and the edenfsctl codebase and in order to more quickly support basic top functionality in windows, we're adding a minitop subcommand that will have a subset of the `eden top` output.
This diff adds scaffolding for the minitop subcommand.
Reviewed By: fanzeyi
Differential Revision: D31841478
fbshipit-source-id: 7020e81a0c8021bad45de835f2e279c049b76200
Summary: Since we have a rust wrapper around the python version of edenfsctl (as we are working on moving to be completely rust), we can strip these environment variables from the fallback python edenfsctl command. Without stripping them, this breaks edenfsctl for some users who have custom PYTHONHOME and PYTHONPATH variables that don't contain the modules the python fallback needs to import.
Reviewed By: fanzeyi
Differential Revision: D31810981
fbshipit-source-id: d357254d7b122d4c501470e1701b034883fca174
Summary: Generated by `sed -i 's/edition = "2018"/edition = "2021"/' common/rust/cargo_from_buck/project_configs/*.toml` and `cargo autocargo` following the autocargo code change in D31830985 and its corresponding MSDK rebuild in D31832311.
Reviewed By: zertosh
Differential Revision: D31832698
fbshipit-source-id: db79024cad311467d5b7c13b7abedc6464f09366
Summary: This crate is a dependency of autocargo. The update is necessary in order for autocargo to be able to emit 2021 edition Cargo.tomls.
Reviewed By: zertosh
Differential Revision: D31824537
fbshipit-source-id: 1261e34a2f49d407afbb55a39acb73a4b4b237f6
Summary: This will unblock making ObjectStore::getBlobMetadata use ImmediateFuture.
Reviewed By: kmancini
Differential Revision: D31667900
fbshipit-source-id: 8f0ac170aa115655f19a8ac12719c0f9fb121b4a
Summary:
This allows the ImmediateFuture to be used as is in FileInode::getSha1 to
reduce the transition from folly::Future to ImmediateFuture
Reviewed By: chadaustin, kmancini
Differential Revision: D31667901
fbshipit-source-id: 3985e256d3c29c334d7e542ea8a2ce4e89528773
Summary: This should reduce the overhead of obtaining sha1 from EdenFS.
Reviewed By: kmancini
Differential Revision: D31667896
fbshipit-source-id: 4ae9b309b54eec56cc8fc1050279ced22e19c40e
Summary:
During Buck build, getting the sha1 of files is a very common operation, and
the various Future overhead are showing up in profiles, let's switch the code
to use ImmediateFuture to speed it up.
Reviewed By: chadaustin
Differential Revision: D31667899
fbshipit-source-id: a87de70ac113d7fb2fe65ad6534c53de0d0170c8
Summary:
With ObjectStore::getTree and getOrLoadChildTree both returning an
ImmediateFuture, we no longer need to convert these to folly::Future to collect
their result.
Reviewed By: chadaustin
Differential Revision: D31452295
fbshipit-source-id: eea46af2d961fcee788a286708cb0cb44945e16a
Summary:
Now that we have three future types, write down the rules about why
each one exists and when to use them.
Reviewed By: xavierd
Differential Revision: D31752167
fbshipit-source-id: 602fdcfaf4dbcf4f90fa199006e3d9424c885d0d
Summary:
ObjectId::toString was too ambiguous of a name which makes it hard to
reason about the correctness of all of its uses as we generalize the
meaning of object identifiers.
Rename toString to toLogString, and change a bunch of call sites to
use the iostream, folly::to, or fmt formatters.
For the one place that knows the format of the ID but wants its hex
representation, use asHexString. This gives us the freedom to change
the log representation of ObjectIds in the future without worrying
about breakage.
Reviewed By: andll
Differential Revision: D31776897
fbshipit-source-id: 8160ab581f260b3bb955afbad65d6dbe8486e3bc
Summary:
HashRootIdCodec is intended for BackingStores that use a 20-byte hash
for commits or root IDs. Therefore, don't support arbitrary hex
strings.
Reviewed By: andll
Differential Revision: D31775704
fbshipit-source-id: a8ed7e644c952c3f63fb97182802d5bb24807451
Summary:
Since this method first queries an in-memory cache, it can greatly benefit from
not allocating a folly::Future.
Reviewed By: chadaustin
Differential Revision: D31452293
fbshipit-source-id: 237bb9503af75f954bcc2d3e22052345bdf11852
Summary:
This will cut some of the boiler plate of returning an ImmediateFuture in the
error case.
Reviewed By: chadaustin
Differential Revision: D31769752
fbshipit-source-id: 42be026aee6a6071bbb2c1f8af918b107d593eab
Summary:
The number of worker threads handling downloading content from source control servers is currently hard-coded. We want to make that value configurable.
This diff removes all hardcoded values and adds a numBackingstoreThreads option to the edenfs config. This value defaults to the original hard-coded value, 32. The minimum number of threads is 1.
As a result, we remove the numberThreads parameter from the HgQueuedBackingStore constructor and modify all code that previously used that parameter.
Reviewed By: kmancini
Differential Revision: D31739881
fbshipit-source-id: fc1706a9a4b7b7b4f4b5d87ac428ddc025616d40
