Summary:
To support better telemetry and logging in watchman we want to use Eden's components. Lets migrate and detangle the needed pieces.
This change moves SystemError from eden to edencommon.
Reviewed By: MichaelCuevas
Differential Revision: D54343729
fbshipit-source-id: 7861e3effc9d242fbeda34333078c14c4d021a80
Summary:
To support better telemetry and logging in watchman we want to use Eden's components. Lets migrate and detangle the needed pieces.
This change moves Utf8 from eden to edencommon.
Reviewed By: fanzeyi
Differential Revision: D54341648
fbshipit-source-id: b1461a7f92963100914733d69de6eb574af3e1f9
Summary:
To support better telemetry and logging in watchman we want to use Eden's components. Lets migrate and detangle the needed pieces.
This change moves String from eden to edencommon.
Reviewed By: fanzeyi
Differential Revision: D54341192
fbshipit-source-id: 8f0763313e5585efeb336038f41c646d84986ec3
Summary:
To support better telemetry and logging in watchman we want to use Eden's components. Lets migrate and detangle the needed pieces.
This change moves Memory from eden to edencommon.
Reviewed By: fanzeyi
Differential Revision: D54340336
fbshipit-source-id: cd209be7f39452123ffc41312cf370a48e4923bc
Summary:
To support better telemetry and logging in watchman we want to use Eden's components. Lets migrate and detangle the needed pieces.
This change moves CaseSensitvity from eden to edencommon.
Reviewed By: fanzeyi
Differential Revision: D54339283
fbshipit-source-id: f96a421f4390578e5d2281b307532c62e22935d3
Summary:
To support better telemetry and logging in watchman we want to use Eden's components. Lets migrate and detangle the needed pieces.
This change moves ImmediateFuture from eden to edencommon.
Reviewed By: fanzeyi
Differential Revision: D54337631
fbshipit-source-id: 2e09a9e244ef8b8b662959c1b14db3e103f559d0
Summary:
This is the start of the codemod for https://fb.workplace.com/groups/rust.language/posts/25780668961555042.
It follows the same approach as {D53374511} by adding a comment above the re-exports to identify the correct Buck target to use instead. This way I can codemod all uses by locally changing the comment into a real `#[deprecated = "..."]` attribute, compiling all Rust targets with `-Awarnings -Wdeprecated` in rustflags, parsing the deprecation warnings, and programmatically applying the correct fix.
Note that the deprecation needs to be on a `pub mod` rather than a `pub use` because deprecation on re-exports is not supported by Rust.
Reviewed By: zertosh
Differential Revision: D54332814
fbshipit-source-id: 16358f211032a7b1a701c08d909078b86b53d65f
Summary:
To support better telemetry and logging in watchman we want to use Eden's components. Lets migrate and detangle the needed pieces.
This change move DynamicEvent from eden to edencommon.
Reviewed By: kmancini
Differential Revision: D54046154
fbshipit-source-id: de9466c73c0b2a2a16302cc0114e4c2a12426b30
Summary:
# Background:
Rendering and parsing FilteredObjectIDs was previously broken. This is because we were naively stringifying all the bytes of the FilteredObjectIDs, some of which were whitespace/invalid when converted to utf-8. This made it impossible to use endpoints like `eden debug inode` and `eden debug blob` since the FOIDs weren't rendered/filtered correctly.
# This diff:
This diff changes the way we render/parse FOIDs so that they are rendered in a readable way, and they are copy/pastable for use in other debug endpoints. Eden will properly parse the copy/pasted FOIDs and convert them into FilteredObjectID structures in code.
Reviewed By: genevievehelsel
Differential Revision: D54150449
fbshipit-source-id: 9f8a4fcbf913bfd24a2c12a8220298ac3c856306
Summary: There's validation of CAS digests based on length, as well as other implicit assumptions about the length of the hash. However, a new hashing algorithm is now being used that produces longer digests, and creating Eden mounts for CAS is broken.
Reviewed By: genevievehelsel
Differential Revision: D47922565
fbshipit-source-id: ff29880f98ef09356f1645ac27c48363f266fe53
Summary:
As the name implies this is unsafe as it causes futures to run inline. Let's
instead use an ImmediateFuture to remove this unsafetyness.
Reviewed By: genevievehelsel
Differential Revision: D51393145
fbshipit-source-id: f22c20eb4c5f23928f11a8636a06fdedff621b84
Summary:
Using `.get()` blocks the Thrift executor which can prevent other parts of
EdenFS from making progress. By futurizing them this issue goes away.
Reviewed By: kmancini
Differential Revision: D51860079
fbshipit-source-id: f5259646acd413e6f76c7930812a7920a529dacf
Summary:
When parsing a TOML file and there's an error, we're just raising an exception without providing the user useful information.
This diff will show the user the specific line that is likely to have the problem in the TOML file and for the cause of incorrectly quoted backslashes, a hint about the problem.
Reviewed By: jdelliot
Differential Revision: D54075769
fbshipit-source-id: 061b85341a2d834ad8aad8c8ea18d7af2491499c
Summary:
# Context
When serving a request for a file or directory, EdenFS will check its own local caches before falling back to requesting Sapling, the Source Control client, for that file/directory object. When falling back to Sapling, it will check in the Sapling local cache before downloading the object from the remote server.
Sometimes the local cache file is corrupted or for other reasons we would like to force a pull from the remote server. This will be read back through the local cache to refresh it.
# This Diff
This Diff implements the debug CLI for FetchRemote. This is done by adding a call to getBlobRemote to the existing stub
# Technical Details
- Implement call to getBlobRemote
- Fix for FetchMode ffi
Reviewed By: genevievehelsel
Differential Revision: D54139848
fbshipit-source-id: 6aac3139d046c59de1f012732323b763afc902dd
Summary:
# Context
When serving a request for a file or directory, EdenFS will check its own local caches before falling back to requesting Sapling, the Source Control client, for that file/directory object. When falling back to Sapling, it will check in the Sapling local cache before downloading the object from the remote server.
Sometimes the local cache file is corrupted or for other reasons we would like to force a pull from the remote server. This will be read back through the local cache to refresh it.
# This Diff
This Diff implements the c++ side skeleton for remote only file fetch. It does not implement any business logic. A followup PR will implement these functions.
# Technical Details
- Add new FetchMode RemoteOnly
- change localOnly bool to directly using FetchMode enum
- add skeleton for get_remote_XXX function
# Discussion Points
Reviewed By: genevievehelsel
Differential Revision: D53833131
fbshipit-source-id: 788459bca68d94fb1d14d997eb690619e454a371
Summary:
`HgBackingStore::periodicManagementTask` is only used in `HgQueuedBackingStore::periodicManagementTask`
As we are planning to fold HgBackingStore into HgQueuedBackingStore, we can remove this method in `HgBackingStore` and implement it inside the caller
Reviewed By: genevievehelsel
Differential Revision: D54141055
fbshipit-source-id: 955ea9fbcdf4fa56df003d800182406e8cd27c5c
Summary:
`nullptr` is typesafe. `0` and `NULL` are not. In the future, only `nullptr` will be allowed.
This diff helps us embrace the future _now_ in service of enabling `-Wzero-as-null-pointer-constant`.
Reviewed By: meyering
Differential Revision: D54163067
fbshipit-source-id: 3473ba42193c8dd3cd61b8f170d20c7279f88521
Summary:
To support better telemetry and logging in watchman we want to use Eden's components. Lets migrate and detangle the needed pieces.
This change extracts DynamicEvent from LogEvent so that it can be moved to edencommon in a later diff.
Reviewed By: kmancini
Differential Revision: D54046152
fbshipit-source-id: b50bae2b155f8640a8b319f9b0353e7f44110100
Summary:
To support better telemetry and logging in watchman we want to use Eden's components. Lets migrate and detangle the needed pieces.
This change moves Throw.h and it's related tests from eden to edencommon.
Reviewed By: genevievehelsel
Differential Revision: D54046153
fbshipit-source-id: 669d702c13e70536d9c0b58ff8ff17f826237851
Summary: The correct format of the .hg/sparse file is `%include {filter}`. This didn't show up in tests because hg only complains if you try to change the filter.
Reviewed By: fanzeyi
Differential Revision: D54134450
fbshipit-source-id: 5e3b784f5aaa7b6f232f3fed6b932fb9d1e570cb
Summary:
All future reads to these files will error until the file metadata caches
are cleared. This diff triggers a delayed invalidation after a file is
read with the incorrect size. This makes the file accessible again.
It's possible (though it doesn't always happen) that EdenFS's internal state
could be inocorrect for the file. a future `eden doctor` run will be able
to fix it up.
Reviewed By: MichaelCuevas
Differential Revision: D52278628
fbshipit-source-id: 580746bb91d21c79f87d16d5e442843617ac779c
Summary:
Currently, EdenFS crashes if ProjFS requests to read more bytes than we have
available for a file. I am able to repro this, by blocking the read so that
it starts before an `hg checkout` that shortens file contents and completes
after.
I am making eden return an error when we don't have enough data available.
Unfortunately, it seems there is no error code that indicates to projfs it's
view is stale and make it flush it's caches and retry. All error codes seem to
be propagated to the user. These ones are the most promising to make PrjFS do
something we want: https://learn.microsoft.com/en-us/windows/win32/projfs/provider-overview#callback-return-codes
But all those still result in errors to the user. I tried a few more common
HRESULT return codes: https://learn.microsoft.com/en-us/windows/win32/seccrypto/common-hresult-values
But all errors still get propagated to the user. Invalid argument seems to be
the most accurate reflection of the issue, so I'm going with that for now.
It would be better if we could just return the amount of data that we have, and
not have to error. However, PrjFS will still think the file is the longer
length and will add null bytes. This will leave the file in an inconsistent
state and leave `hg status` dirty. This seems worse than erroring because it
causes EdenFS to corrupt the file.
Ideally we would fix this by preventing the length from getting out of sync
from the contents we should read the data from. As far as I see there are two
options, but they are both more involved:
- Failing the checkout when we try to invalidate the contents and
it fails. This would be a bit disruptive to the user as background reads
can interfere with a checkout. but it matches the what a native filesystem would
do.
- Keeping the version of the file to read in the placeholder so the version and
length are never out of sync. The file contents would then be the old contents.
This would leave the file modified and make status dirty, which is still not a
great user experience, but at least the file would be in a consistent state.
I've reached out to ProjFS about other solutions -- maybe there is or could be
a particular error that eden can send that will make ProjFS retry and
gracefully deal.
For now error is better than crash
Reviewed By: mshroyer, MichaelCuevas
Differential Revision: D50193962
fbshipit-source-id: 6ca05a4195c29996a0cd255bfd461a98f7dd5fa6
Summary:
Now that we have the ability to kill processes from the previous diff, this one adds the logic to retry if the process kill is successful.
(note: Replacing a previous diff that was giving me some rebase pain)
Reviewed By: MichaelCuevas
Differential Revision: D54070610
fbshipit-source-id: 7a085c7eccba2bf1544de55c3b5dca7a53533ef3
Summary: This code was removed, I assume this file was just missed in the cleanup
Reviewed By: jdelliot, kmancini
Differential Revision: D53976398
fbshipit-source-id: 922e0f4aa1bb453a1da59e3a5638ddbfb2088b81
Summary:
Unfortunately the more time I've spent with this the more I've realized how poorly the whole `polars` import in `fbsource` (`third-party/pypi:polars` + `third-party/rust:polars` + `third-party/rust:polars-py`) has been done. To be fair it is quite a complicated use case and definitely the largest dep footprint Rust/Python module in all of `fbsource`. For more context see T179532760 but this is the 1st step of many to fix everything and hopefully move `polars` out of `third-party/rust` completely.
That said, the work here allows me to land `1.76.0` without having to upgrade `pyo3` + `polars` + `libcst` + tons of other crates (stack @ D53776888) and do that with more time on my hands post land.
Reviewed By: dtolnay
Differential Revision: D53858402
fbshipit-source-id: 3644897f2b569ebc23463dd35aba77bd5c59fca6
Summary:
This diff adds the functionality of asking the user if they want to terminate processing preventing eden rm from succeeding.
Also some refactoring done, specifically create prompt.py for the prompt utility because having in cmd_util (which imports config, which imports cmd_util) was causing a circular reference problem when trying to import from file_handler_tools.
Reviewed By: MichaelCuevas
Differential Revision: D53688726
fbshipit-source-id: cf1c6f32ec9e026fa29932554fb552740ad2c5a5
Summary:
# Context
eden rage is a tool used to gather diagnostic output for debugging issues with eden. One of the things it does is collect a variety of logs and sends them to paste.
# This Diff
This diff adds 2 new log sources to eden rage,
1) edenfs_upgrade.log - written by a scheduled python script that restarts EdenFS on a weekly basis to ensure users are running with the most current release
2) edenfs_config.log - written by a scheduled python script that delivers EdenFS configs to user machines
# Technical Details
The logs are read on disk and sent to paste. The link to the paste is displayed in the rage output.
Reviewed By: genevievehelsel
Differential Revision: D53717899
fbshipit-source-id: d18545dfe224e6c4ad37688e75720159f1901400
Summary:
# Context
Purgeable space is an APFS concept where certain types of data is retained when "deleted" to be cleaned up at a later point in time by the OS or manually. Certain workflows lead to the creation of alot of purgeable space. We would like to create a command for users to easily see and clean out this space.
# This Diff
This diff adds the --purgeable flag to `eden du`. This command only works on macOS. When running `eden du` on macOS without the flag, the summary output will hint to the user that the flag exists. When running with the flag, the user will see the space available in purgeable space and ask the user if they want to clear it.
# Technical Details
- --purgeable flag added, should only work on macOS but exists on other platforms for compiler reasons. Has no effect on non-macOS platforms
- When run with the flag performs the following commands
-- Reports Amount of purgeable storage
sudo /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -G $HOME
-- Purges purgeable storage
sudo /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -P -total <value from -G> $HOME
- displays usage space in summary output
- user will be prompted to clear space or not
- user will be asked for sudo passsword
# Discussion Points
1. Text output is open to adjustment
Reviewed By: MichaelCuevas
Differential Revision: D53591104
fbshipit-source-id: 38f4d92d027394bc0d29e4a066ba06ac83f3fc72
Summary:
`-Wextra-semi` or `-Wextra-semi-stmt` found an extra semi
If the code compiles, this is safe to land.
Reviewed By: MichaelCuevas
Differential Revision: D53776196
fbshipit-source-id: f83dda6df32622f1ced24d051eadeaa8ebfda4e8
Summary:
This document is not up to date with the EdenFS Config Manager, and would be nice to reference when getting questions about how everything works.
VSCode formatted the file for me, so some of this diff is just formatting change as well.
Reviewed By: kmancini
Differential Revision: D53589632
fbshipit-source-id: 389c2ee18f71533ca5c7bb4f922a36c63aa431e4
Summary:
This diff moves the code that manages handler.exe to its own module and abstract the functionality into a base class, even though for now only Windows is implemented.
It also calls this new handle in the branch that deletes a non-Eden directory, because it's very convenient to test.
Reviewed By: genevievehelsel
Differential Revision: D53594227
fbshipit-source-id: 309efab3f89893f745a82fe5acd128ecd702d42e
Summary:
Refactored a few things:
- Removed ODS tracking code
- Re-named interfaces to not be SCS specific
- Renamed CASCountersManager to `StorageDemandControl`
Logic wise nothing has changed, so this should be a safe land.
Reviewed By: Skory
Differential Revision:
D53598493
Privacy Context Container: L1141686
fbshipit-source-id: 32d6ba623af134c6de54908b9f09622306c7afd0
Summary:
# Problem
The `eden clone` and `hg clone` command currently allow you to specify that you want to use FilteredFS, but they don't allow you to specify the filter to active after the clone completes. Instead, the user has to manually set the filter themselves by running `hg filteredfs enable {path_to_filter}`.
This is annoying for many reasons, and it prevents us from easily rolling out filters on Sandcastle/ODs.
# Solution
We should allow users to pass in the Filter they want to enable at clone time. This will allow them to skip the `hg filteredfs enable {path_to_filter}` step. Even better, we'll allow passing this filter via an Hg Config. That way we can easily enable FilteredFS + use a specific filter just by setting a config value.
# This diff
Adds the option to pass a --filter-path to `eden clone`. This will be used in the next diff to allow passing the filter path via config to `hg clone`.
Reviewed By: genevievehelsel
Differential Revision: D53599526
fbshipit-source-id: 1e5d8e8f7e66157f345d49ff51a36b977b19b96d
