Summary:
Exposes the ability to construct `NodeIpc` from a Windows HANDLE, not a
msvcrt/ucrt's fd.
Reviewed By: zzl0
Differential Revision: D46811167
fbshipit-source-id: 94f48dc0d059bca109c1792dceb0b733e4737adf
Summary:
In the next change I'm adding "raw ..." concept. Let's rename `raw_fd` to
`libc_fd` to make it obvious that the raw something is different from the raw
"HANDLE" on Windows.
Reviewed By: zzl0
Differential Revision: D46811165
fbshipit-source-id: e0fe48633fd8d928ce7d7ae0652f041a90aec922
Summary:
While most PrjFS operations only communicate about the filesystem object they're modifying via the [`PRJ_CALLBACK_DATA->FilePathName`](https://learn.microsoft.com/en-us/windows/win32/api/projectedfslib/ns-projectedfslib-prj_callback_data) member, notifications via the [`PRJ_NOTIFICATION_CB`](https://learn.microsoft.com/en-us/windows/win32/api/projectedfslib/nc-projectedfslib-prj_notification_cb) have a `destinationFileName` argument as well.
This is documented as:
> If notification is PRJ_NOTIFICATION_PRE_RENAME or PRJ_NOTIFICATION_PRE_SET_HARDLINK, this points to a null-terminated Unicode string specifying the path, relative to the virtualization root, of the target of the rename or set-hardlink operation.
While the individual STRACE `NotificationArgRenderer` functions handled the extra path, `PrjfsLiveRequest::formatTraceEventString` does not.
This forwards the string to that object, allowing it to be rendered if needed, so that it shows up in the `eden trace fs` output.
Reviewed By: xavierd
Differential Revision: D46809287
fbshipit-source-id: 781db1d911a975902d0b27d9ba4f553cc3b85299
Summary: We were forbidding \n but not \r. A couple Python spots validate against "\r" and "\n" in filenames (scmutil.checkfilename and manifest._checkforbidden), so let's do it in Rust as well.
Reviewed By: zzl0
Differential Revision: D46646011
fbshipit-source-id: 41b5cbad057e2b966e3c00011d75cd6cd6e804fc
Summary:
This test needed significant changes to use remotefilelog (unrelated to Rust status).
In particular:
- "hg cat" doesn't report "no such file" for non-existing files. Previously that came from py manifest, but Rust manifest walk doesn't support.
- Kill tampered bundle check. I tried for a while to get this working, but everything fails really early with the Rust manifest since bad paths are immediately rejected by the RepoPath type.
- Various update-over-symlink tests now work since the Rust vfs will remove symlinks in order to write the new file (thus avoiding the "xxx traverses symbolic link yyy" audit error).
Also this test revealed the need to process directories in the Rust status to catch a file being replaced by a directory. I changed the (Watchman) code path to no longer skip directories returned from watchman.
Reviewed By: zzl0
Differential Revision: D46646010
fbshipit-source-id: a2f8a92e85a97611212d917b2a6118f0dd1f5eed
Summary:
The x2p auth advice header is handled/displayed by the network doctor in practice now.
I'm making tests support the Rust status, and opting to remove this one instead of update it.
Reviewed By: zzl0
Differential Revision: D46646012
fbshipit-source-id: 72bd074b0e8d76ecbc4206920f6c8867d6d794cd
Summary: I had to tweak the case collision auditor to exclude untracked files. The Rust status (using Watchman) seem so to add untracked files to the dirstate earlier than Python, which was causing extra "possible case-folding" warnings.
Reviewed By: quark-zju
Differential Revision: D46832704
fbshipit-source-id: fa0e30df757804804ca437df94a4fb4b56dcbeda
Summary:
Some Python commands depend on the "bad" matcher callback getting called by dirstate.status(). Tweak things so this happens when we use the Rust status as well. This also adds support for the "clean" option when using the Rust status since clean files and "bad" are somewhat coupled.
I moved invalid file type detection from filesystem.py into dirstate.py so it covers Rust status as well.
Reviewed By: quark-zju
Differential Revision: D46646006
fbshipit-source-id: 28278af15e02e0f794aefc7bde850ce58b319ac6
Summary: It aleady passes for these, so turn it on.
Reviewed By: quark-zju
Differential Revision: D45510980
fbshipit-source-id: f45a2dbe11186411adc259863b88c6ff5ee1914e
Summary: It's annoying to have to remember when this setting is and isn't enabled by default.
Reviewed By: quark-zju
Differential Revision: D45510982
fbshipit-source-id: 7d54c0c95d7edf61f358692b1a7c487e5889b5fa
Summary:
add option for consistent routing of single tree requests and single blob requests
Meta:
Background:
We are having a lot of traffic for single tree blob or single file blob:
https://fburl.com/scuba/mononoke_test_perf/0eainxd3https://fburl.com/scuba/mononoke_test_perf/0ye4eduq
Let's route them consistently!
Reviewed By: quark-zju
Differential Revision: D46860178
fbshipit-source-id: 50298c583d82585b3345ec4c4ca6b7ab4ab57ab1
Summary:
The casefolding pushrebase check is blocking the sync of www commits causing
problems. Let's exempt www/ dir from it.
We'll also have to modify commit hooks - but that's a separate thing.
Reviewed By: markbt
Differential Revision: D46860559
fbshipit-source-id: 87db959e0d025c0c1fc5c6cfecbdcf96af9e0f81
Summary:
This diff does the following refactoring:
* Remove redundent key checking when parsing blobstore key, and repo_id from it
* Replace all the `writeln!` with `println!`
* Print errors using `eprintln!`
* Remove misuse of verbose
Differential Revision: D46857681
fbshipit-source-id: 1b065756eb91d8b4b67422ef49ded2783660992f
Summary: This diffs allows the new admin tool to print out how long it takes to process each file during bulk blobstore key unlinking.
Differential Revision: D46838088
fbshipit-source-id: 92fa57edcce5b171a1d4f6be6d7a57571fdb618b
Summary:
This diff:
* Adds a new HashMap to store the blobstres, regarding a given repo_id, to avoid duplicated computation
* Removes some debugging output
Differential Revision: D46802649
fbshipit-source-id: 8ddf0243b192b5684af2d4a430cdb8873a78246d
Summary:
This diff allows the new admin tool to actually delete the content from the blobstore.
When the key is not present, there is nothing happening.
Differential Revision: D46802651
fbshipit-source-id: ac1d9aabcb4fd17455263c29b1348f5d1619868b
Summary:
[plan_deletion] Add subcommand to redact relevant blobs
To make it easy to use in the context of plan deletion, we take the provisional deletion plan issued from the `propose-deletion-plan` subcommand in and we output the redaction id to a file.
In the real world, one would use this redaction id to make a diff to the mononoke config in configerator and restart the mononoke servers to make sure that the redaction actually takes effect before moving on to the next step.
Reviewed By: markbt
Differential Revision: D46768075
fbshipit-source-id: 368dafd485a02d677f3a74c6ede59a80cbe931d4
Summary:
The current interface relies on the assumption that redacted content only relates to file contents.
`list`, which is the way to read what was redacted discovers the blobs for filecontents by walking the fsnodes manifests.
For a feature I'm working on, I need to be able to redact arbitrary blobs, so I need a way to identify what was redacted
without relying on some derived data blobs being available.
Reviewed By: markbt
Differential Revision: D46806525
fbshipit-source-id: 78b5470d4dd741538e3e85353c0b1634f3b83b1c
Summary: Instead of taking in the full context of a `MononokeApp`, only take the two blobstores that we need.
Reviewed By: markbt
Differential Revision: D46768087
fbshipit-source-id: 773f0f31e12c58ee0cc8341e638fcbcedde8bab9
Summary:
The crux of the redaction process is this `create_key_list` function which appends each key to the redaction blobstores and provides instruction to generate the correct redaction config.
I will need to re-use this mechanism, so extract it to an accessible crate.
Reviewed By: markbt
Differential Revision: D46763714
fbshipit-source-id: 8f06a12c6348bc49d0ca4fe65155818a216c5b88
Summary:
The blackbox was reporting a mode mismatch. The cause was the filesysytem walk yielding a mode with permission bits set (e.g. 0666), but os.lstat within dirstate.normal() yields a mode without permission bits set (e.g. 0000). Fix by only reporting a mode change when the executable bit changes (which seems like the intention of the code).
I don't think this is a recent change in behavior, so I'm not going to look further into it. Hopefully the Python filesystem code will go away soon with the Rust status.
Reviewed By: zzl0
Differential Revision: D46848722
fbshipit-source-id: cdd73137ae891bc8038d7d7ddcd09d8c95170051
Summary:
This diff allows the new admin tool to extract the blobstore key from a given key.
For example, `flat/repo2122.blame.fileunode.blake2.2e58d897760aa5927c7ca1b0755992c9109be6048a9dc1deab1c86a4619f5839` will give `repo2122.blame.fileunode.blake2.2e58d897760aa5927c7ca1b0755992c9109be6048a9dc1deab1c86a4619f5839`.
Differential Revision: D46802650
fbshipit-source-id: bec63353b1ca50eeeab73cd84b5e3d8fb967c694
Summary:
This diff adds
* a new function to extract repo id from a given key
* a new function to construct all the blobstores given a repo id
Differential Revision: D46763054
fbshipit-source-id: 64bb1ab4fd8bcae8ab414dd03e78f0a7c2dd5639
Summary: Some of the usages of the commit graph (e.g. in pushrebase) query very recent commits that might not have been fully replicated in xdb yet. This diff switches to using fetch_edges_required and fetch_many_edges_required everywhere except in exists, changeset_parents and changeset_generation, to avoid any errors caused by replication lag. This shouldn't cause any performance degredation as we always use the normal read connection first and only query master in the rare case that a commit is missing from replicas.
Reviewed By: mitrandir77
Differential Revision: D46842804
fbshipit-source-id: 3822d883b079447d0af90c9169cd3db7600f9b25
Summary: The SQL implementation of fetch_edges_required first tries fetching the changeset edges using the normal read connection, then tries to fetch any missing changesets using the master connection. This can sometimes be the desired behavior when working with very recent commits so let's expose it through changeset_generation_required and changeset_parents_required.
Reviewed By: markbt
Differential Revision: D46796193
fbshipit-source-id: 38b3d79dd88ca4b8b98f36705ca1a215379a87b3
Summary: Added the missing blobstore fetch encoding except RedactionKeyList as that require creating a separate config and already has its own newadmin command.
Differential Revision: D46491769
fbshipit-source-id: 82208ac9dcb1acb5370ff7904c285e843be4f18f
Summary:
This makes commits created by megarepo tools roundtrippable through bonsais (so far they were not).
D27852341 shows that it's not a problem for actual backup job but.. it's a
problem for example for mirror_hg_commits binary so let's make thos commits
nice and tidy.
Reviewed By: mzr
Differential Revision: D46489608
fbshipit-source-id: d15ba6f6622f51189a1ba9e76efd68faf1bf2b71
Summary:
Verification is used to verify if given megarepo mapping configuration can be cleanly applied to given pair of small repo and large repo commits. It's used when we want to change current config and we want to check if according to the new config the mapping is still sound.
The previous *fast path* verification worked by limiting the amount of visited entries to `O(small repo)`
The new fast path verification doesn't walk every file in the repository, instead it leverages FSNodes to compare hashes of entire directories. This was if the repository verifies OK the verification is very fast. The amount of entries visited is `O(differences to report + number of config entries)`. When the configuration is simple and the repository is intact (usual case) it's almost instant. In case of more complex configurations (like ovrsource one) it's still **much** faster than current one.
WARNING: The implementation is a bit hacky due to the path mover functions being orignally designed with moving file paths not, directory paths. The hack is mostly contained to wrap_mover_result functiton.
Differential Revision: D45864549
fbshipit-source-id: 2fad0ed29a5718e655fa4a69b28306ca3db31dda
Summary:
I want to test for some bug in megarepotooling and I'd like to use that
feature.
Differential Revision: D46440647
fbshipit-source-id: 1f4bb0d937f6e681a02f9843a7241fde4ae9b774
Summary:
Some of our commits have those fields set so let's allow setting them on test
commits.
Differential Revision: D46440648
fbshipit-source-id: 393d9fcb616ddee5481f239264ab87bf5c8315e9
Summary:
fbcode is migrating to LLVM-15 for safer and more up-to-date code and new compiler features. All contbuilds in your directory have passed our build test with LLVM-15, and your directory does not host any packages. This diff will migrate it to LLVM-15.
If you approve of this diff, please use the "Accept & Ship" button. If you have a reason for why it should not build with LLVM 15, please make a comment and send it back to author. Otherwise we will land this on Thursday 06/15/2023.
See the [FAQ post](https://fb.workplace.com/groups/llvm15platform010/posts/749154386769776/)! Please also direct any questions to [this group](https://fb.workplace.com/groups/llvm15platform010).
- If you approve of this diff, please use the "Accept & Ship" button :-)
Build directives:
Reviewed By: meyering, zzl0
Differential Revision: D46662743
fbshipit-source-id: 1af43eea7bb103d29f3b409634f1b7a60ae84c88
Summary:
We need that for current catchup to avoid confusing clients with backdated
commits.
Reviewed By: markbt
Differential Revision: D46743487
fbshipit-source-id: a71ee30eada63dab7c64cbe43a4495662fe04371
Summary: Walker previously specified a default for the deprecated `cachelib-only-blobstore` argument. Switch to `cache-mode`, allowing us to specify local caching for all types, not just the blobstore.
Differential Revision: D45089769
fbshipit-source-id: 978ac545590935be5a825669db9df58f0281c41a
