Summary:
This flag allows the repo import to progress through all the steps
but bail before actually merging in the repos.
Currently the repo_import tool is so slow that all the pre-merge steps
can take hours. This makes it really hard to control when the actual merge
commit will happen. This flag will allow us to prepare all those steps
ahead-of-time and then resume with just merge in mind.
I don't think it's a proper long-term fix but I found it useful when working
wath whatsapp/biz and would use it again util we properly optimize repo_import
tool.
Differential Revision: D46802952
fbshipit-source-id: 2e8185482c4ba9c04fed20013efcc80d75e80bad
Summary:
D45510982 enabled rust status for debugruntest, but this test is not
ready yet. This diff disable rust status explicitly in the test.
Reviewed By: quark-zju
Differential Revision: D46938285
fbshipit-source-id: b9cb514d8721328a65ac88229aeff8f85dbaf922
Summary: This diff allows the new admin tool to double check if the key is really we wanted to delete, before doing the actual deletion.
Reviewed By: mitrandir77
Differential Revision: D46901306
fbshipit-source-id: 58904c5272d22b696dd22b3c83a9caf33fa3a0b2
Summary:
Similarly named method in scmquery pushes commits onto some branch.
Let's make the description very clear.
Reviewed By: malmond77
Differential Revision: D46916234
fbshipit-source-id: 6137f42df97be80fff2775638f766f05c1113488
Summary:
Add "connection pool" logic that emulates a pool using a directory.
This can be useful on Windows where `fork()` is unavailable.
Reviewed By: zzl0
Differential Revision: D46885307
fbshipit-source-id: 26c85f6836fc7a9437ccaef0aaf4a7f0a6e8561f
Summary:
Provide APIs to obtain the `NodeIpc` instance instead of raw sockets.
This makes it possible to use `NodeIpc` features like sending the
file descriptors, node channel, etc.
Reviewed By: muirdm
Differential Revision: D46812533
fbshipit-source-id: 7b3d51a6027c9dbacffcf74476f9b8a4a3440313
Summary:
Add a crate to make it easier to use UDS (unix-domain-socket) communication.
For now, it only contains the low-level bind, and connect functions.
Reviewed By: muirdm
Differential Revision: D46812534
fbshipit-source-id: efbf31dbb1286ae64051b570c20ca6f41c736afa
Summary:
The `FileDescriptor` type will close the underlying fd on drop. In our
use-cases `r` and `w` refer to the same fd so we only need to close it
once. Note the reason we use separate `Mutex`es for r and w is that we
want `recv()` and `send()` running from different threads to not block
by each other.
Reviewed By: zzl0
Differential Revision: D46885308
fbshipit-source-id: b2a5644251ac4773581a995ae8c3d955b0b36861
Summary: The crate is no longer used.
Reviewed By: muirdm
Differential Revision: D46812535
fbshipit-source-id: 659b73a563bf14cc17cb1b09e9a57422d21a6377
Summary: `recv_stdio` seems a better place for the logic.
Reviewed By: muirdm
Differential Revision: D46811171
fbshipit-source-id: ddd91489610fcf6a148897bbd4f35b1e404cbd0b
Summary:
This makes it easier to use. In addition, the "singleton" IPC initially decided
by NODE_CHANNEL_FD is handled internally so the callsite is less error-prone,
since it does not need to care about maintaining the NODE_CHANNEL_FD env var,
or the initialization order of the singleton.
Reviewed By: muirdm
Differential Revision: D46811166
fbshipit-source-id: a2a8a7a36986422a8d7c967d6fb5b820bedce0c9
Summary:
Remove the side effect when constructing the NodeIpc. It does not seem
necessary. This allows us to construct NodeIpc without checking the underlying
fd until first send/recv.
Reviewed By: muirdm
Differential Revision: D46811162
fbshipit-source-id: 4fc7941a1e9d6b9a9c47daa699de4283bfcec5c3
Summary:
This allows us to update it dynamically, without going through the
`NODE_CHANNEL_FD` environment variable.
Reviewed By: muirdm
Differential Revision: D46811169
fbshipit-source-id: bf52b5597fcf69291da81675d2917dc0dc010904
Summary: This makes it more flexible so we can replace the singleton if we want.
Reviewed By: muirdm
Differential Revision: D46811160
fbshipit-source-id: 902b7ff05a4fb3f8a7d7218afd3d07e0f3a89a9f
Summary:
Make it possible to send fds on POSIX. This is similar to what chg
(hgclient.c:attachio and osutil.c:recvfds) does. Note the `sendfd`
crate seems to have a solid implementation but its public API is
on the `std::os::unix::net::UnixStream` type, which is incompatible
with our struct that uses the `FileDescriptor`.
Example:
% cargo run --example spawn_sendfd
Parent: spawning child
Parent: sending hello
Parent: sending stdio and a.txt file descriptors
Parent: waiting for child to exit
Child: started with IPC handle 4
Child: got message Some("hello")
Child: got sendfd payload SendFdPayload { raw_fds: [3, 5, 6, 7] }
Child: writing "something\n" to fd 3
something
Child: writing "something\n" to fd 5
something
Child: writing "something\n" to fd 6
something
Child: writing "something\n" to fd 7
% cat a.txt
something
Reviewed By: muirdm
Differential Revision: D46811163
fbshipit-source-id: 37686327f02a7fa0b0e8768d025094a7fcb0e2b5
Summary:
This avoids over reading which might drop special messages from sendmsg(),
on POSIX systems.
The behavior disables itself if we need libuv compatibility on POSIX.
Reviewed By: muirdm
Differential Revision: D46811164
fbshipit-source-id: 3aed4bb4d63ad10ecc6a7bf76007cc7263c1c580
Summary:
This avoids over reading which might drop special messages (like file descriptors)
from sendmsg(). Buffered reader might turn those special messages into regular
messages and will lose the file descriptors.
Reviewed By: muirdm
Differential Revision: D46811158
fbshipit-source-id: 2294c5c2b132c25c21950f26e9f0d44094743fe0
Summary:
We're going to use winapi to implement sending and receiving file descriptors
on Windows.
Reviewed By: muirdm
Differential Revision: D46811161
fbshipit-source-id: dca85722da9f96e15f1ec8bb4676058487c51872
Summary: They will be used by upcoming changes.
Reviewed By: muirdm
Differential Revision: D46811168
fbshipit-source-id: fdd60a2aa96b979dd2dc5bb2d016189435bbf695
Summary: This makes it a bit easier to use in upcoming changes.
Reviewed By: zzl0
Differential Revision: D46811157
fbshipit-source-id: 333e17415d4a67fcb58bec61153b15e4cad03d23
Summary:
Exposes the ability to construct `NodeIpc` from a Windows HANDLE, not a
msvcrt/ucrt's fd.
Reviewed By: zzl0
Differential Revision: D46811167
fbshipit-source-id: 94f48dc0d059bca109c1792dceb0b733e4737adf
Summary:
In the next change I'm adding "raw ..." concept. Let's rename `raw_fd` to
`libc_fd` to make it obvious that the raw something is different from the raw
"HANDLE" on Windows.
Reviewed By: zzl0
Differential Revision: D46811165
fbshipit-source-id: e0fe48633fd8d928ce7d7ae0652f041a90aec922
Summary:
While most PrjFS operations only communicate about the filesystem object they're modifying via the [`PRJ_CALLBACK_DATA->FilePathName`](https://learn.microsoft.com/en-us/windows/win32/api/projectedfslib/ns-projectedfslib-prj_callback_data) member, notifications via the [`PRJ_NOTIFICATION_CB`](https://learn.microsoft.com/en-us/windows/win32/api/projectedfslib/nc-projectedfslib-prj_notification_cb) have a `destinationFileName` argument as well.
This is documented as:
> If notification is PRJ_NOTIFICATION_PRE_RENAME or PRJ_NOTIFICATION_PRE_SET_HARDLINK, this points to a null-terminated Unicode string specifying the path, relative to the virtualization root, of the target of the rename or set-hardlink operation.
While the individual STRACE `NotificationArgRenderer` functions handled the extra path, `PrjfsLiveRequest::formatTraceEventString` does not.
This forwards the string to that object, allowing it to be rendered if needed, so that it shows up in the `eden trace fs` output.
Reviewed By: xavierd
Differential Revision: D46809287
fbshipit-source-id: 781db1d911a975902d0b27d9ba4f553cc3b85299
Summary: We were forbidding \n but not \r. A couple Python spots validate against "\r" and "\n" in filenames (scmutil.checkfilename and manifest._checkforbidden), so let's do it in Rust as well.
Reviewed By: zzl0
Differential Revision: D46646011
fbshipit-source-id: 41b5cbad057e2b966e3c00011d75cd6cd6e804fc
Summary:
This test needed significant changes to use remotefilelog (unrelated to Rust status).
In particular:
- "hg cat" doesn't report "no such file" for non-existing files. Previously that came from py manifest, but Rust manifest walk doesn't support.
- Kill tampered bundle check. I tried for a while to get this working, but everything fails really early with the Rust manifest since bad paths are immediately rejected by the RepoPath type.
- Various update-over-symlink tests now work since the Rust vfs will remove symlinks in order to write the new file (thus avoiding the "xxx traverses symbolic link yyy" audit error).
Also this test revealed the need to process directories in the Rust status to catch a file being replaced by a directory. I changed the (Watchman) code path to no longer skip directories returned from watchman.
Reviewed By: zzl0
Differential Revision: D46646010
fbshipit-source-id: a2f8a92e85a97611212d917b2a6118f0dd1f5eed
Summary:
The x2p auth advice header is handled/displayed by the network doctor in practice now.
I'm making tests support the Rust status, and opting to remove this one instead of update it.
Reviewed By: zzl0
Differential Revision: D46646012
fbshipit-source-id: 72bd074b0e8d76ecbc4206920f6c8867d6d794cd
Summary: I had to tweak the case collision auditor to exclude untracked files. The Rust status (using Watchman) seem so to add untracked files to the dirstate earlier than Python, which was causing extra "possible case-folding" warnings.
Reviewed By: quark-zju
Differential Revision: D46832704
fbshipit-source-id: fa0e30df757804804ca437df94a4fb4b56dcbeda
Summary:
Some Python commands depend on the "bad" matcher callback getting called by dirstate.status(). Tweak things so this happens when we use the Rust status as well. This also adds support for the "clean" option when using the Rust status since clean files and "bad" are somewhat coupled.
I moved invalid file type detection from filesystem.py into dirstate.py so it covers Rust status as well.
Reviewed By: quark-zju
Differential Revision: D46646006
fbshipit-source-id: 28278af15e02e0f794aefc7bde850ce58b319ac6
Summary: It aleady passes for these, so turn it on.
Reviewed By: quark-zju
Differential Revision: D45510980
fbshipit-source-id: f45a2dbe11186411adc259863b88c6ff5ee1914e
Summary: It's annoying to have to remember when this setting is and isn't enabled by default.
Reviewed By: quark-zju
Differential Revision: D45510982
fbshipit-source-id: 7d54c0c95d7edf61f358692b1a7c487e5889b5fa
Summary:
add option for consistent routing of single tree requests and single blob requests
Meta:
Background:
We are having a lot of traffic for single tree blob or single file blob:
https://fburl.com/scuba/mononoke_test_perf/0eainxd3https://fburl.com/scuba/mononoke_test_perf/0ye4eduq
Let's route them consistently!
Reviewed By: quark-zju
Differential Revision: D46860178
fbshipit-source-id: 50298c583d82585b3345ec4c4ca6b7ab4ab57ab1
Summary:
The casefolding pushrebase check is blocking the sync of www commits causing
problems. Let's exempt www/ dir from it.
We'll also have to modify commit hooks - but that's a separate thing.
Reviewed By: markbt
Differential Revision: D46860559
fbshipit-source-id: 87db959e0d025c0c1fc5c6cfecbdcf96af9e0f81
Summary:
This diff does the following refactoring:
* Remove redundent key checking when parsing blobstore key, and repo_id from it
* Replace all the `writeln!` with `println!`
* Print errors using `eprintln!`
* Remove misuse of verbose
Differential Revision: D46857681
fbshipit-source-id: 1b065756eb91d8b4b67422ef49ded2783660992f
Summary: This diffs allows the new admin tool to print out how long it takes to process each file during bulk blobstore key unlinking.
Differential Revision: D46838088
fbshipit-source-id: 92fa57edcce5b171a1d4f6be6d7a57571fdb618b
Summary:
This diff:
* Adds a new HashMap to store the blobstres, regarding a given repo_id, to avoid duplicated computation
* Removes some debugging output
Differential Revision: D46802649
fbshipit-source-id: 8ddf0243b192b5684af2d4a430cdb8873a78246d
Summary:
This diff allows the new admin tool to actually delete the content from the blobstore.
When the key is not present, there is nothing happening.
Differential Revision: D46802651
fbshipit-source-id: ac1d9aabcb4fd17455263c29b1348f5d1619868b
Summary:
[plan_deletion] Add subcommand to redact relevant blobs
To make it easy to use in the context of plan deletion, we take the provisional deletion plan issued from the `propose-deletion-plan` subcommand in and we output the redaction id to a file.
In the real world, one would use this redaction id to make a diff to the mononoke config in configerator and restart the mononoke servers to make sure that the redaction actually takes effect before moving on to the next step.
Reviewed By: markbt
Differential Revision: D46768075
fbshipit-source-id: 368dafd485a02d677f3a74c6ede59a80cbe931d4
Summary:
The current interface relies on the assumption that redacted content only relates to file contents.
`list`, which is the way to read what was redacted discovers the blobs for filecontents by walking the fsnodes manifests.
For a feature I'm working on, I need to be able to redact arbitrary blobs, so I need a way to identify what was redacted
without relying on some derived data blobs being available.
Reviewed By: markbt
Differential Revision: D46806525
fbshipit-source-id: 78b5470d4dd741538e3e85353c0b1634f3b83b1c
Summary: Instead of taking in the full context of a `MononokeApp`, only take the two blobstores that we need.
Reviewed By: markbt
Differential Revision: D46768087
fbshipit-source-id: 773f0f31e12c58ee0cc8341e638fcbcedde8bab9
Summary:
The crux of the redaction process is this `create_key_list` function which appends each key to the redaction blobstores and provides instruction to generate the correct redaction config.
I will need to re-use this mechanism, so extract it to an accessible crate.
Reviewed By: markbt
Differential Revision: D46763714
fbshipit-source-id: 8f06a12c6348bc49d0ca4fe65155818a216c5b88
Summary:
The blackbox was reporting a mode mismatch. The cause was the filesysytem walk yielding a mode with permission bits set (e.g. 0666), but os.lstat within dirstate.normal() yields a mode without permission bits set (e.g. 0000). Fix by only reporting a mode change when the executable bit changes (which seems like the intention of the code).
I don't think this is a recent change in behavior, so I'm not going to look further into it. Hopefully the Python filesystem code will go away soon with the Rust status.
Reviewed By: zzl0
Differential Revision: D46848722
fbshipit-source-id: cdd73137ae891bc8038d7d7ddcd09d8c95170051
Summary:
This diff allows the new admin tool to extract the blobstore key from a given key.
For example, `flat/repo2122.blame.fileunode.blake2.2e58d897760aa5927c7ca1b0755992c9109be6048a9dc1deab1c86a4619f5839` will give `repo2122.blame.fileunode.blake2.2e58d897760aa5927c7ca1b0755992c9109be6048a9dc1deab1c86a4619f5839`.
Differential Revision: D46802650
fbshipit-source-id: bec63353b1ca50eeeab73cd84b5e3d8fb967c694
Summary:
This diff adds
* a new function to extract repo id from a given key
* a new function to construct all the blobstores given a repo id
Differential Revision: D46763054
fbshipit-source-id: 64bb1ab4fd8bcae8ab414dd03e78f0a7c2dd5639
Summary: Some of the usages of the commit graph (e.g. in pushrebase) query very recent commits that might not have been fully replicated in xdb yet. This diff switches to using fetch_edges_required and fetch_many_edges_required everywhere except in exists, changeset_parents and changeset_generation, to avoid any errors caused by replication lag. This shouldn't cause any performance degredation as we always use the normal read connection first and only query master in the rare case that a commit is missing from replicas.
Reviewed By: mitrandir77
Differential Revision: D46842804
fbshipit-source-id: 3822d883b079447d0af90c9169cd3db7600f9b25
Summary: The SQL implementation of fetch_edges_required first tries fetching the changeset edges using the normal read connection, then tries to fetch any missing changesets using the master connection. This can sometimes be the desired behavior when working with very recent commits so let's expose it through changeset_generation_required and changeset_parents_required.
Reviewed By: markbt
Differential Revision: D46796193
fbshipit-source-id: 38b3d79dd88ca4b8b98f36705ca1a215379a87b3
Summary: Added the missing blobstore fetch encoding except RedactionKeyList as that require creating a separate config and already has its own newadmin command.
Differential Revision: D46491769
fbshipit-source-id: 82208ac9dcb1acb5370ff7904c285e843be4f18f
Summary:
This makes commits created by megarepo tools roundtrippable through bonsais (so far they were not).
D27852341 shows that it's not a problem for actual backup job but.. it's a
problem for example for mirror_hg_commits binary so let's make thos commits
nice and tidy.
Reviewed By: mzr
Differential Revision: D46489608
fbshipit-source-id: d15ba6f6622f51189a1ba9e76efd68faf1bf2b71
Summary:
Verification is used to verify if given megarepo mapping configuration can be cleanly applied to given pair of small repo and large repo commits. It's used when we want to change current config and we want to check if according to the new config the mapping is still sound.
The previous *fast path* verification worked by limiting the amount of visited entries to `O(small repo)`
The new fast path verification doesn't walk every file in the repository, instead it leverages FSNodes to compare hashes of entire directories. This was if the repository verifies OK the verification is very fast. The amount of entries visited is `O(differences to report + number of config entries)`. When the configuration is simple and the repository is intact (usual case) it's almost instant. In case of more complex configurations (like ovrsource one) it's still **much** faster than current one.
WARNING: The implementation is a bit hacky due to the path mover functions being orignally designed with moving file paths not, directory paths. The hack is mostly contained to wrap_mover_result functiton.
Differential Revision: D45864549
fbshipit-source-id: 2fad0ed29a5718e655fa4a69b28306ca3db31dda
Summary:
I want to test for some bug in megarepotooling and I'd like to use that
feature.
Differential Revision: D46440647
fbshipit-source-id: 1f4bb0d937f6e681a02f9843a7241fde4ae9b774
Summary:
Some of our commits have those fields set so let's allow setting them on test
commits.
Differential Revision: D46440648
fbshipit-source-id: 393d9fcb616ddee5481f239264ab87bf5c8315e9
Summary:
fbcode is migrating to LLVM-15 for safer and more up-to-date code and new compiler features. All contbuilds in your directory have passed our build test with LLVM-15, and your directory does not host any packages. This diff will migrate it to LLVM-15.
If you approve of this diff, please use the "Accept & Ship" button. If you have a reason for why it should not build with LLVM 15, please make a comment and send it back to author. Otherwise we will land this on Thursday 06/15/2023.
See the [FAQ post](https://fb.workplace.com/groups/llvm15platform010/posts/749154386769776/)! Please also direct any questions to [this group](https://fb.workplace.com/groups/llvm15platform010).
- If you approve of this diff, please use the "Accept & Ship" button :-)
Build directives:
Reviewed By: meyering, zzl0
Differential Revision: D46662743
fbshipit-source-id: 1af43eea7bb103d29f3b409634f1b7a60ae84c88
Summary:
We need that for current catchup to avoid confusing clients with backdated
commits.
Reviewed By: markbt
Differential Revision: D46743487
fbshipit-source-id: a71ee30eada63dab7c64cbe43a4495662fe04371
Summary: Walker previously specified a default for the deprecated `cachelib-only-blobstore` argument. Switch to `cache-mode`, allowing us to specify local caching for all types, not just the blobstore.
Differential Revision: D45089769
fbshipit-source-id: 978ac545590935be5a825669db9df58f0281c41a
Summary: To avoid showing a bunch of "?" files, we need to ignore things within submodules. For watchman, we do similar to Python and add submodules to the ignored dirs list. For the manual walker, we similarly skip directories in the ignore list. Note that the walker already skipped the submodules due to the presence of the "dot dir" (e.g. ".sl") in the submodules, but I piped through the ignored dirs skipping to be consistent.
Reviewed By: quark-zju
Differential Revision: D46547238
fbshipit-source-id: b001dc02a3c73f69d74fe36615a812f9ac983944
Summary: Python status for Git has submodule aware logic we don't have in Rust, so we need to defer to Python. However, Python will still be able to use the internal Rust status API.
Reviewed By: zzl0
Differential Revision: D46547239
fbshipit-source-id: 2caf1436819219cea007307f602627fd1a38ef51
Summary: It wasn't used from Rust (only MultiWalker(threads=8) was used), and Rust walker is not enabled from Python. It's annoying to have to make changes twice, so let's kill the non-parallel walker.
Reviewed By: zzl0
Differential Revision: D46547240
fbshipit-source-id: 4a57222ec705059acbc6865f8e1a9701ae584ccc
Summary: This wasn't enabled in production, and the parallel walker doesn't currently work from Python due to matcher GIL deadlock. Anyway, I'm working to make Python use the Rust status directly, so this intermediate optimization of using the Rust walker in the Python status won't buy us anything.
Reviewed By: zzl0
Differential Revision: D46646007
fbshipit-source-id: 645d88adb85700062c3edf2483fb5a5ffb76cca2
Summary: These make it more convenient to conditionalize logic based on storage format.
Reviewed By: zzl0
Differential Revision: D46547242
fbshipit-source-id: ef833c954090f5cb2b1c2c6e616d5d67a442886d
Summary: I'm going to use this to skip submodules in the Rust status implemenation.
Reviewed By: zzl0
Differential Revision: D46547241
fbshipit-source-id: e5cf5dc5ee77aa3e8927ab3cf5922f3e29a25ae5
Summary:
`std::io::IsTerminal` was stabilized in 1.70. It is intended to be a drop in replacement for `atty::is(Stdout)`.
This codemod removes all remaining instances of atty
`arc f` was ran twice, followed by two invocations of `arc lint -e extra --take RUSTFIXDEPS`. `arc autocargo` was also ran
This script was a little naive though, and as a result, I had to manually move the newly added imports to the correct line. I ran lints again afterwards.
Reviewed By: dbxfb
Differential Revision: D46736264
fbshipit-source-id: a686b96b1fa0aa4389f65487ed426f226c86e8e9
Summary: www has lots of empty commits that are being synced to all small repos. Until we fix that problem let's refrain from backsyncing any empty commits. I'm making it a tunable for easy on-off
Reviewed By: liubov-dmitrieva
Differential Revision: D46727657
fbshipit-source-id: 5f204cc8231fec3e6a4eaf25b6575c0d637d67a3
Summary:
[plan_deletion] Implement `commit-deletion-plan-to-deletion-log-db` subcommand
This is the last necessary step before enacting the deletion.
As the outcome of this step, the deletion_log db is populated and we can start taking the necessary steps to safely proceed with the actual deletion of changesets and blobs.
We still need to document and potentially write tools to help with the whole process, including blobs redaction.
Differential Revision: D46724273
fbshipit-source-id: bc60209a28e2ee8a50ac0371f78c0b3be234781b
Summary: This diff moves all the bulb unlinking functions into a struct.
Reviewed By: mitrandir77
Differential Revision: D46759318
fbshipit-source-id: 143171706485ec6a56152bdfea6202221d88fe03
Summary: This diff makes the `get_blobstores` function to be public, so that we can use it to get blobstores. These blobstores are going to be used to unlink keys in a bulk.
Reviewed By: mitrandir77
Differential Revision: D46729000
fbshipit-source-id: 11d7ddc5f90527ebdb79141f61f1671997108ac5
Summary:
This diff allows the new admin tool to do the following for the new bulk unlking keys subcommand:
* Read the key list from the files
* Print all the keys extracted when running in dry-run mode
* Report the progress
* Fix the argument to take a boolean value
Reviewed By: mitrandir77
Differential Revision: D46725312
fbshipit-source-id: c039b89e00a9ffbbbee41e20617f45543ef147ce
Summary: This diff adds a new subcommand for our new admin tool, so that we can delete keys bulkly. The input is the directory that contains a list of files, each file contains a list of *Manifold* keys to remove.
Reviewed By: mitrandir77
Differential Revision: D46689957
fbshipit-source-id: 077b6a4c76872ef01bc7f0eab7096abeb87d7047
Summary:
`std::io::IsTerminal` was stabilized in 1.70. It is intended to be a drop in replacement for `atty::is(Stdout)`.
This codemod replaces all usages of `atty::is(Stdout)` with `stdout().is_terminal()`, importing the necessary paths.
It used this script (generated with assistance from Metamate):
```
#!/bin/bash
files=$(fbgs -sl "atty::is(atty::Stream::Stdout)" | arc linttool debugfilterpaths --take RUSTFMT)
for file in $files; do
sed -i 's/atty::is(atty::Stream::Stdout)/stdout().is_terminal()/g' $file
done
for file in $files; do
sed -i '1i use std::io::{stdout, IsTerminal};' $file
done
```
`arc f` was ran twice, followed by two invocations of `arc lint -e extra --take RUSTFIXDEPS`.
This script was a little naive though, and as a result, I had to manually move the newly added imports to the correct line. I ran lints again afterwards.
Reviewed By: danielocfb
Differential Revision: D46601191
fbshipit-source-id: 3f2269c518e36241ab0e995b87976323d7da9cc6
Summary:
`std::io::IsTerminal` was stabilized in 1.70. It is intended to be a drop in replacement for `atty::is(Stderr)`.
This codemod replaces all usages of `atty::is(Stderr)` with `stderr().is_terminal()`, importing the necessary paths.
It used this script (generated with assistance from Metamate):
```
#!/bin/bash
files=$(fbgs -sl "atty::is(atty::Stream::Stderr)" | arc linttool debugfilterpaths --take RUSTFMT)
for file in $files; do
sed -i 's/atty::is(atty::Stream::Stderr)/stderr().is_terminal()/g' $file
done
for file in $files; do
sed -i '1i use std::io::{stderr, IsTerminal};' $file
done
```
`arc f` was ran twice, followed by two invocations of `arc lint -e extra --take RUSTFIXDEPS`.
This script was a little naive though, and as a result, I had to manually move the newly added imports to the correct line. I ran lints again afterwards.
Reviewed By: Imxset21
Differential Revision: D46601192
fbshipit-source-id: 18ef53a9fa25cbd733ca81f16b2c13ce8c5ddb98
Summary: `gitimport` is typically used for mirror repos which by definition are locked. Creating or moving bookmarks on locked repos is disallowed by default. This behaviour can be overriden by passing the bypass flag. This diff adds argument support for bypassing the read-only check.
Differential Revision: D46722005
fbshipit-source-id: 9defcbc159a55d0e7ff197d8d732e3d5a208a998
Summary: Use the EDEN_TRY macro to reduce error-handling line noise.
Reviewed By: genevievehelsel
Differential Revision: D46668011
fbshipit-source-id: bb54ff710928b3a5c394b6e79474c25ef4775895
Summary: Adds a macro to simplify bailing out of Try-returning functions, sort of like with Rust's question mark operator.
Reviewed By: chadaustin
Differential Revision: D46567326
fbshipit-source-id: f13eec678a101e423a99ba469a20d6de5513d73c
Summary: optimize the check against skip revset by reusing the unskipped result.
Reviewed By: muirdm
Differential Revision: D46687070
fbshipit-source-id: 01e5be4499daaa9e24437ddb16e2cc04f0436211
Summary:
Currently, the 'skip' option is too slow when users want to use it to bisect on
a file or folder like below:
sl bisect --skip "!( file('path:foo') & file('path:bar') )"
Since it tries to eval the skip revset upfront, which takes a long time for
current `file` implementation. This diff make it lazy evaluated.
Reviewed By: muirdm
Differential Revision: D46643574
fbshipit-source-id: c063fbea2bc0703772e07d5167adc8fbf989a68f
Summary:
move the bisect kind check earlier, so that it will not do unnecessary
node lookup
Reviewed By: sggutier
Differential Revision: D46643576
fbshipit-source-id: 624c6410aaf95f82a993093df4e24add6c5e48ea
Summary: this will be needed for making 'skip' lazy evaluated
Reviewed By: muirdm
Differential Revision: D46643575
fbshipit-source-id: 53cfe2395d320d85582fbf707f62d69a5088bea2
Summary: we will update this function in the following diffs, abstract it to a separate function will avoid making the main function too complex
Reviewed By: muirdm
Differential Revision: D46643577
fbshipit-source-id: 227e0b60fe2efa1d6d00fa409ea95a30cc972747
Summary: We handle the error gracefully, when deleting a key from the underlying blobstores.
Reviewed By: RajivTS
Differential Revision: D46684795
fbshipit-source-id: 45349aab01e56b11fd1ab455aec2bb69d72c497b
Summary:
If there's corruption in the cache, it should be considered a miss,
rather than failing.
Reviewed By: chadaustin
Differential Revision: D46577606
fbshipit-source-id: 4cc99e743e869422a149a073b726722896d971d4
Summary:
The dagcopytrace was enabled for source missing file in D46315431.
It's too slow if there are many files deleted in the source commit. I will debug the root cause tomorrow, let's disable it for hg repo first. We are not disabling it for git repo, because there is not alternative for git repos.
Reviewed By: muirdm
Differential Revision: D46711460
fbshipit-source-id: 4e20bbb34be126c13af081475953b83df61872df
Summary: Adding to track possible corruption issues we've been getting user reports about
Reviewed By: chadaustin
Differential Revision: D46706164
fbshipit-source-id: 6a868d9f9ffd3f72303123486bbb8312f2002072
Summary: This diff allows the new admin tool to delete keys from all the related blobs (either the single blobstore specified by the id, or all the underlying blobstore given a repo)
Differential Revision: D46564415
fbshipit-source-id: d0299262a82acb4e1a9f4cedd5510418f270d527
Summary:
This diff adds a new function to construct multiple blobstores from a given repo.
The finaly step is to refactoring the existing `get_blobstore` function.
Reviewed By: RajivTS
Differential Revision: D46559402
fbshipit-source-id: 9b9901d00566556614264831f8643f7c6d4ee20f
Summary: Running a job in sharded setting requires it to support pre-emption in case ShardManager decides to move the repo around. This diff adds support for that pre-emption.
Differential Revision: D46514772
fbshipit-source-id: fb086e74a9c56e0ced5921877de49401bc426170
Summary: As in title. This tool is currently run as 5-6 different TW jobs that compute and publish data for specific repos. Instead of having multiple jobs, if we manage to shard this job across set of available tasks, it becomes much more easier to manage.
Differential Revision: D46483039
fbshipit-source-id: d7d879dee088c8f431c22915bd7ef9602b83aa00
Summary:
In the win32 implementation of `readFile`, the ret string wasn't being resized
to the number of bytes actually read, which may be smaller than given in
`num_bytes`. This change makes it match the behavior of folly's readFile.
Reviewed By: xavierd
Differential Revision: D46532796
fbshipit-source-id: 1da3e4ff4190673796731610ae8a798b953d0c96
Summary:
This diff adds a new function of constructing a single blobstore from config and inner_id.
Next step is to add a new function to construct multiple blobstores from a given repo.
The finaly step is to refactoring the existing `get_blobstore` function.
Reviewed By: RajivTS
Differential Revision: D46559565
fbshipit-source-id: b59e77e13fbe76ba013caa62bfeb9f56cbcf31d2
Summary: This diff added a function to retrieve all the inner blobstore ids from a given multiplexed blobstore.
Reviewed By: RajivTS
Differential Revision: D46523044
fbshipit-source-id: e4c3b02e1672bc2eea3208bac4f23d879c004515
Summary: Now that the `blob_repo` dependency has been removed from `statistics_collector`, this diff migrates the tool from old Mononoke App to new Mononoke App getting rid of the old clap dependency.
Differential Revision: D46478775
fbshipit-source-id: dd713eef2175fa01934508445866d64ef01657f0
Summary: In order to better understand what happens in cases described in https://fb.workplace.com/groups/edenfswindows/permalink/1315557692693439/ adding better logging and some data integrity checks.
Reviewed By: chadaustin
Differential Revision: D46605682
fbshipit-source-id: 485c5c83834cdde9ccfea9cfa94323cf03df7002
Summary:
eden doctor can discover when an inode is missing for a file, but can't
remediate the issue. restart usually remediates the issue, but it would be
better to have doctor remediate since restart can be very slow.
We could do something similar to our remediation for phantom inodes (performing
a filesystem operation). However, messing with the filesystem leaves us open to
races with concurrent modifications. The point of the filesystem io is to make
eden see a notification about a certain path and match it's state to the
filesystem. So we can directly do that instead.
We can more directly do this by introducing a thrift call to make eden match
it's internal state to the filesystem.
We could replace the other remediation with this thrift call. I'll leave that
for a follow up.
Reviewed By: xavierd
Differential Revision: D46243633
fbshipit-source-id: a1df5929428dc4f6c8fd71d826fe1e7371ebf283
Summary:
We've got a couple sets of prjfs tests now and they have some utility functions.
Let's move this all into a base class so it can easily be reused.
Reviewed By: xavierd
Differential Revision: D46379827
fbshipit-source-id: c4b708a1effebe5f11c97a63519295be625c8018
Summary:
eliminate completely unnecessary traffic to commit graph endpoint for
main bookmark (for segmented changelog supported repos, they use pull_fast_forward_master edenapi endpoint)
After the fast path pull being completed, the filtration of what is already
present wasn't working correctly. It could easily skip (not to be able to
filter out) what was just fetched via the fast path.
That resulted in completely unnecessary traffic to commit graph (which is
slower, especially for large bits of the master branch).
During the SEV that traffic was super expensive due to the www merge commit, but even without SEV this traffic leads to unnecessary load on our services especially if the repo is significantly behind master.
The solution is to invalidate caches.
Differential Revision: D46590498
fbshipit-source-id: c8d920f4ffd3411297ca7cd01368b731f3c2236e
Summary:
fix incorrect logging to scuba number of slow path heads
it makes more sense to log after the filter I think because in this case it
would mean how many we need to pull via commit graph (slower) path
Reviewed By: zzl0
Differential Revision: D46559984
fbshipit-source-id: 4059c189afbb6ecc1dea69f0726af7b00e576f74
Summary:
we have added repo._dagcopytrace property in a previous diff, let's
use it in the debugcopytrace command
Reviewed By: quark-zju
Differential Revision: D46570173
fbshipit-source-id: 729c12160e8f0ce89ae07a5c94ca98852bbaf68f
Summary:
update master bookmark on pull
update of the master bookmark normally uses fast path for most of the repos
pulling of bookmarks is the first thing the repo.pull implementation does, if
after that the commit is present in the repo, slow path pull won't be called
otherwise, we are calling slow path pull for fetching a chunk of main branch
if the clone is old, it is a long chunk, if there are merges, it gets even
more complicated for the slow path pull
the current version seems to have the issue with phases as well, then you restore snapshot from old clone, you got enormous smartlog.
Differential Revision: D46556754
fbshipit-source-id: a771a856813d513cd34b207ea76f1a410dc0b1c0
Summary:
After previous commit D46473305, ctrl-c is no longer handled by streampager in "direct" mode since streampager isn't polling for user input. This leaves the terminal in a funny state since streampager doesn't clean up.
Fix by waiting up to 10ms for streampager to shut down after we send it the "quit" action. I had to make the "direct" mode in streampager actually handle the "quit" action.
Reviewed By: quark-zju
Differential Revision: D46474536
fbshipit-source-id: 11df5849ea13ac530e169bdba33d5061a77647e0
Summary: Tweak the "direct" mode run loop to use recv_timeout on the event channel instead of a try_recv+sleep. This allows it to respond faster to events (in particular, I'm going to make it support "quit" in following commit).
Reviewed By: zzl0
Differential Revision: D46508080
fbshipit-source-id: efb1261324b2720baaa1960fae60a3b09f4d82ce
Summary:
Command line users commonly type their next command before the current command has finished (or, they paste in a series of commands all at once). Previously if one of the commands invoked the streampager, all remaining pending input to the terminal would get lost. This was because streampager would always put the terminal into raw mode, which loses pending input AFAICT. Raw mode also disables terminal echo, which is also undesirable for streampager "direct" mode.
Fix by deferring enablement of raw mode until it is needed. I also disabled the startup_poll_input option since that requires raw mode in order to poll for user input. This means a user can't type "q" or "h" anymore in direct mode, but I think that is a worthwhile tradeoff since users probably don't even know they are in a pager. This makes streampager behave more like "less -F".
Reviewed By: zzl0
Differential Revision: D46473305
fbshipit-source-id: 4523050bc6c8b96d0f37e2b6078555aa23ae2b28
Summary: Log repo name as well as commit hash and other commit info.
Differential Revision: D46432233
fbshipit-source-id: dff22dd00297d3d5694f09a2b64ef7d6c4b7c87e
Summary: This is a refactor aimed to retain the same behavior while consolidating the code and improving it's resilience to future changes.
Reviewed By: mitrandir77
Differential Revision: D46190582
fbshipit-source-id: a956f5df1044c52346a712e271c75baf298b481d
Summary:
In the case where PrjfsChannel::start failed to initialized, EdenFS would crash
due to the stopPromise_ not being fulfilled. That promise is being set in the
PrjfsChannel::stop method, but unfortunately, when start throws, the caller
wouldn't catch the exception.
To fix this, we can simply move all the code in the initialize method, which if
it throws, the caller already properly tears down the mount thus solving the
crash mentioned above.
Reviewed By: genevievehelsel
Differential Revision: D46210460
fbshipit-source-id: 1524e8e0fc74ce3119b1b6dcd35b5419efcebdf3
Summary: See the previous commit for context.
Reviewed By: muirdm
Differential Revision: D45959185
fbshipit-source-id: ff60352111d175fa9dfd9609e0a1bb2a15dd776e
Summary: See the added test for context.
Reviewed By: muirdm
Differential Revision: D45959186
fbshipit-source-id: e437d8dc51b3730504528ee836632ed0b5ebe088
Summary:
The Rust `NameSet` tracks O(1) `min`, `max` in its `Hints` structure.
Use it to implement `fastmin`, `fastmax`. This is used by the next change.
Reviewed By: muirdm
Differential Revision: D45959181
fbshipit-source-id: a0a506243610914c879978715c59e1277863508d
Summary:
Similar to `fastlen`, these methods return `None` if the information cannot be
computed quickly.
Reviewed By: muirdm
Differential Revision: D45959183
fbshipit-source-id: 4c693eab2122c12bad40d8c9e9ab228f6d1f5057
Summary:
Optimize fastlog follow() revset to avoid iterating on the slow `subset` for
both ASC and DESC order.
Reviewed By: muirdm
Differential Revision: D45936985
fbshipit-source-id: 951af6d82376d0256c25b03af531a49d45a01559
Summary:
For commands like:
hg log -r SOMETHING PATH
Use `follow(PATH,heads(SOMETHING))` instead of `_matchfiles(...)` as an optimization.
Reviewed By: muirdm
Differential Revision: D45936986
fbshipit-source-id: 0922fff697b38278f4c33a11f8c9ec66246ae0d9
Summary:
The behavior was added by D2634918. It sets `--removed` with the intention of
forcing the "slow path", in other words, "do not use filelog() path". It does
not really care about the `--removed` behavior.
Note: tweakdefaults might add the `--follow` flag, depending on whether
tweakdefaults or remotefilelog runs first, the `--removed` flag might be added
undesirably if tweakdefaults is going to add `--follow` next.
Anyway, with D45936982, the `filelog()` code path knows to disable itself
for remotefilelog repos. So there is no need for remotefilelog to use
`--removed` to disable the `filelog()` code path. Let's remove it so we can
get the true `--removed` flag decided by the user.
Reviewed By: muirdm
Differential Revision: D45936984
fbshipit-source-id: e800c09d0489e84e01a7d0b499a6d31963c73bf6
Summary:
The `_makelogrevset` feels cryptic due to the intertwined deeply nested
conditions about what strategy it uses to generate the revset.
This diff makes the strategies more explicit by using dedicated functions to
make the code (and future changes) easier to reason about.
Reviewed By: muirdm
Differential Revision: D45936982
fbshipit-source-id: 531464a1dc16e2e0b4aa61eda49a6c87488ae23b
Summary:
The `_makelogrevset` feels cryptic, partially because its use of `opt2revset`
and related logic for adhoc revset expression calculation.
This diff removes `opt2revset` and related logic, so:
- `opts` is no longer both input and output. It's only input. The output is a
different variable.
- Use `revsetlang` APIs to format revset strings directly, without going
through an indirection via `opts`. Avoid `repr` or `a % b` formatting.
Note: `%d` is formatted as `_intlist(v)` instead of `rev(v)`.
The new code generates slightly simpler revset expressions with less `()`s
(or "group" in revsetlang AST).
Reviewed By: zzl0
Differential Revision: D45936983
fbshipit-source-id: 3155016a85fa9ea61f9334a2d1fc82eac5450fec
Summary:
It's already the same as `descendants` in our current implementation.
It's non-trivial to implement efficiently and is rarely used. So let's just
remove it for now.
Note the order of children of a commit is undefined, unlike parents. If we want
to provide a "meaningful" order and implement this in the future we might want
to use commit date etc to sort children.
Reviewed By: zzl0
Differential Revision: D45936979
fbshipit-source-id: bff72a130a5ffb7c573f06374da8f9c02791049f
Summary: Useful to chain a list of revset together.
Reviewed By: zzl0
Differential Revision: D45936980
fbshipit-source-id: 1775cd472e239f56971d0b441aae3e416852ea3f
Summary: Named branches are removed in our implementation. Clean them up in the log command.
Reviewed By: zzl0
Differential Revision: D45936987
fbshipit-source-id: 98d3df01b1f3db53861f886289cb720dccf0535a
Summary: It's not used anywhere. This removes a dependency on the legacy filelog.
Reviewed By: zzl0
Differential Revision: D45936981
fbshipit-source-id: 1c3af8dc792454d1bf893e922c1c37c80fa0a79b
Summary: This can simplify logic that checks a bunch of repo requirements.
Reviewed By: zzl0
Differential Revision: D45936978
fbshipit-source-id: eec8ca0035642bf2324782b7fb7e97cb740cc02d
Summary: We need that for optimizing our storage.
Reviewed By: RajivTS
Differential Revision: D46362446
fbshipit-source-id: d81973cd05cffdc78ad84da867fa4791efea877c
Summary: As in title. Once `BlobRepo` has been removed from this tool, I will also migrate it to the new `Mononoke Admin`
Reviewed By: mitrandir77
Differential Revision: D46444744
fbshipit-source-id: 059623d32d608f852254f16f811bcaf4e6bf3b4a
Summary: This includes PR 7 so drops the patch.
Reviewed By: zertosh
Differential Revision: D46451758
fbshipit-source-id: 5df8ccbfa610eab470744e28b5bffe411a983996
Summary: Removing the previous default ctor to ensure that we always compute and pass blake3.
Reviewed By: chadaustin
Differential Revision: D46268716
fbshipit-source-id: d9bfbc7bfd07b61dbb2e915c3fe72d7526d919e1
Summary: Adding a method to retrieve blake3 hash for a list of files.
Reviewed By: chadaustin
Differential Revision: D46268718
fbshipit-source-id: 59cb3d25a1d059a7e9b6a4da784a820945ffbd32
Summary: Expose blake3 via file attributes and changing some integration tests accordingly. Added blake3_sum so that it could be used to verify blake3 hashes and updated the tests to work with blake3.
Reviewed By: chadaustin
Differential Revision: D46307686
fbshipit-source-id: 6f2a4e8e25757862ef17d56f92b90a95c7f5a474
Summary: This utility binary could be used from the integration tests to compute blake3 hashes as it is easier to have it than adding the python pypi package with native dependencies.
Reviewed By: chadaustin
Differential Revision: D46307685
fbshipit-source-id: 1c48c689312cd9a04a19a62ad02ae3e9185041e5
Summary: If backing store doesn't provide a blake3 hash then we should be able to compute and cache it on demand.
Reviewed By: chadaustin
Differential Revision: D46316187
fbshipit-source-id: 9b77368ac5bb351d0edd80274b280692f8e1f774
Summary: Adding some util functions to ThriftUtil to support blake3
Reviewed By: chadaustin
Differential Revision: D46268709
fbshipit-source-id: 1bc7b7b02ecb121e10e4b15036aa645466e7f762
Summary:
Adding blake3 support into TreeEntry and backing stores.
Note: Http and RE CAS stores don't provide blake3 hash so far. While it would be pretty easy to add support for RE, not sure how hard it would be for the http store.
Reviewed By: chadaustin
Differential Revision: D46268715
fbshipit-source-id: db66e63fe0348eb582a8050f22cdc0ff720ccf85
Summary: Exposing blake3 from hg to eden
Reviewed By: chadaustin
Differential Revision: D46268710
fbshipit-source-id: 5f34b5f9586cc8c9507cc987378dc172959b4079
Summary:
Adding blake3 into the `Entry` and `Info` objects to make it possible to retrieve blake3 hashes.
The FileNode part will be integrated in the next diff
Reviewed By: chadaustin
Differential Revision: D46268719
fbshipit-source-id: b787b583e5abb3ed306fd1870f96bbdbc9fc0711
Summary: Adding blake3 support to the FileHash util that is used to compute hashes on Windows.
Reviewed By: chadaustin
Differential Revision: D46268714
fbshipit-source-id: cdca3dced494b1f37841121df09351acfdd6c745
Summary: Frequently we will be operating with the blake3 key that is represented as `std::optional` so making it is easier to initialize blake3 appropriately.
Reviewed By: chadaustin
Differential Revision: D46268711
fbshipit-source-id: 2b9882683e79de2325d23353a4ca7260f3853f47
Summary:
Adding blake3 support to SerializedBlobMetadata.
* The serialized data is stored as:
* - version (1 byte, big endian)
* - blob_size (up to 10 bytes, big endian)
* - used_hashes (1 byte, big-endian)
* - hashes stored in order of their type values e.g. from less significant
to more significant
* - hash (N bytes)
...
* - hash (M bytes)
Reviewed By: chadaustin
Differential Revision: D44500908
fbshipit-source-id: d64081db8591f85be5cdf3fd98ff9839c028d5fb
Summary: Use `-` for old working copy parent, so we can easily go back to the previous working copy parent with `hg go -`. This is similar to `cd -` to go back to the previous folder.
Reviewed By: quark-zju
Differential Revision: D46368217
fbshipit-source-id: ab51992f8dde66a7535ba673892bbb6c0f01ecec
Summary:
first in a set of diffs with the intent of removing `lazy_static!` from fbcode.
this diff changes `std::sync::Mutex` and `parking_lot::Mutex` to plain `static` initializations where possible.
Reviewed By: zertosh
Differential Revision: D46335784
fbshipit-source-id: 105848f38dcb8083be77a4ad4db267058081fe53
Summary: Show 9 characters for autocompleted revs, to match rev numbers shown in e.g. `hg sl`. This makes it a little easier to verify the rev that was autocompleted for you matches the one you were looking at in the smart log (i.e. if you're looking at the end of the rev rather than the beginning).
Reviewed By: zzl0
Differential Revision: D46398818
fbshipit-source-id: 70f19d60df7aa85371e1c1838b362b621da5b493
Summary: This library doesn't seem to be used anymore in that file, so seems to be safe to delete it
Reviewed By: zzl0
Differential Revision: D46433066
fbshipit-source-id: 7c522b99048130f2ecd996e6ba901b5e5cccad2b
Summary:
saintelmosfire
Will only land this after D45356044 is landed and tested.
Differential Revision: D45356043
fbshipit-source-id: 068aa45e1edd954b8ba5dc3355f9392ab06419b5
Summary:
Now we interpret merge tool specifications as "merge tool scripts" and evaluate them in the context of each conflicting file. This allows the merge tool script to use a different underlying tool depending on the type of conflict. For now, the only test the script can do is whether the file was deleted on one side.
In particular, we evaluate merge tools specified via "--tool", "HGMERGE", "ui:merge" config, and "merge-patterns" config.
Note that some existing valid inputs are invalid merge tool script (e.g. "--tool '/bin/sh do_something'"). It's invalid because of the space, among other potential reasons. I don't think this is common, but to work around it we let the input through un-intrepreted if it doesn't contain "if(" (i.e. doesn't look like a merge tool script that needs evaluating).
Reviewed By: quark-zju
Differential Revision: D46163268
fbshipit-source-id: 8a3d4b799e7f8d1d36bcc21035d7310de3d30ddb
Summary:
Add a wrapper that contains the three file contexts. This will be used in next commit to simplify integration of "merge tool script" tool selection.
I tweaked the debugpickmergetool to allow specifying whether the local or other fctx is absent rather than the less specific "changedelete".
Reviewed By: quark-zju
Differential Revision: D46163269
fbshipit-source-id: 184ca01ca9dc3bdf80d2ec23f2e0a3f1d745f581
Summary:
A user wants to use a different merge tool for delete+modified conflicts. The current merge tools are not flexible or composable enough to handle this. For example, even if we added a ":delete-keep-local" tool, there is no way to specify the fallback merge tool to use for other kinds of conflicts.
In this diff I introduce a simple DSL to solve the above problem. I called it "merge tool script". It only supports the "if" function, and "isabsent" to test of the "local" or "other" file context is absent/deleted.
Reviewed By: zzl0
Differential Revision: D46163270
fbshipit-source-id: 28f2579e665dc9dd66ab816b10e0b7a758e772f5
Summary: Move string parsing code from revsetlang.py into the parser so other langs can use it.
Reviewed By: zzl0
Differential Revision: D46194125
fbshipit-source-id: cb8c829ceb52a829a831a33b4265410ef704fef9
Summary: v1 and v2 have both been written out forever, so I think we can drop the v1 format.
Reviewed By: zzl0
Differential Revision: D46075828
fbshipit-source-id: 373dae9cf6a53534489022716986aa85b5de5552
Summary: upgrade parking_lot from version 0.11.2 to version 0.12.1
Reviewed By: zertosh, diliop, dtolnay
Differential Revision: D46398501
fbshipit-source-id: ffae0ea188abad5253f1524216dd18ce9a53a74c
Summary:
In D41769964, we decided not to stop `gitimport` simply because `create-bookmark` fails cause that could be due to the fact that the bookmark being created already exists. If that is indeed the case, we try to resolve the bookmark and move it to the new value. However, there could be other unrelated reasons for `create_bookmark` to fail in which case we would still try to resolve the bookmark and end up in unexpected situations (e.g. Luibov trying to import `paws` repo faced this problem). This diff does two things:
- Performs an early check to determine if the bookmark exists by resolving it. If it does exist, it executes the move bookmark branch of code
- If the bookmark doesn't exist, then it creates it.
If there are any errors during bookmark creation or movement, they get logged for the user.
Reviewed By: mitrandir77
Differential Revision: D46071703
fbshipit-source-id: 446b0bc720d13b9e9314feccdb147b2bab0b7cd9
Summary: Added `fill_from_function` which makes it much more useful
Reviewed By: fanzeyi
Differential Revision: D46342378
fbshipit-source-id: 9bc8c01dcb10806a03f1eab628f9763c4e5edc81
Summary:
Our telemetry was showing that starting mid-April the cost of
ObjectStore::getBlobmetadata went down significantly but investigating the root
cause of the improvement didn't seem to yield any results. It turns out the
root cause was that the code no longer timed the cost of the future.
Reviewed By: kmancini
Differential Revision: D46369667
fbshipit-source-id: 0c7d997c93d94498384e27b3447a9c736dcce3b2
Summary:
the previous change broke `test-fb-ext-copytrace.t` tests, but somehow the failure didn't show up in the CI, this is to fix the that.
The issue happened when rebasing 8f852f229 (mv a b) to 046961699 (del a), and file `a` doesn't in the dest commit. The old logic is to keep b in the final commit, and my previous abort with an error that a is not in dest. I think a better way is to report a conflict and ask users to resolve it. But for now, I will keep the old behevior.
```
@ 046961699 72 seconds ago zhaolong
│ del a
│
│ o 8f852f229 107 seconds ago zhaolong
├─╯ mv a b
│
o 7a737d7e6 2 minutes ago zhaolong
╭─╯ add a
```
Reviewed By: sggutier
Differential Revision: D46357844
fbshipit-source-id: f5860c0037b909e2dd1a62700775dba505b88849
Summary: The `lock_env` function from `scm/lib/config/loader/src/test_util.rs` was way too useful to leave in its own crate, since it could be useful for Rust unit tests that modified environment variables
Reviewed By: quark-zju
Differential Revision: D46133861
fbshipit-source-id: b32842c55bd7279490f69601fbb22201da7a0767
Summary: Adds a trait similar to `IOOutput`, which can be used for determining whether the input object held by `IO` is in a tty
Reviewed By: zzl0
Differential Revision: D46129268
fbshipit-source-id: 2fe2b19baf85d6ba361e515cd9e017bda628cc60
Summary: This function will be used in later diffs for determining the default user config file
Reviewed By: muirdm
Differential Revision: D46005612
fbshipit-source-id: 52a0fcba6fbb3d47d24c759d3ff946301ccaa922
Summary:
Currently, the `_forwardcopies` doesn't support git format repo, which
means copytrace can not find the renames on the source side. So let's use
`dagcopytrace` for it as well. It also benefits hg format repos as well, since
`dagcopytrace` support content similarity based rename finder.
Reviewed By: quark-zju
Differential Revision: D46315431
fbshipit-source-id: 5f9ba4a9346f1dbd7e96731b9559dfe686ff028c
Summary:
Currently, the `_forwardcopies` doesn't support git format repo, which
means copytrace can not find the renames on the source side. Add a test to
reproduce this issue.
Reviewed By: quark-zju
Differential Revision: D46315430
fbshipit-source-id: c25f8db6da121a7baa02045720b867c08845f169
Summary:
Both the getBlobMetadata and fetchBlobMetadata were publishing to the same counter, making them indistinguishable.
Created from CodeHub with https://fburl.com/edit-in-codehub
Reviewed By: genevievehelsel
Differential Revision: D46343472
fbshipit-source-id: 04e09372c21dd5cb09336259ff34a5394d10d3b4
Summary:
add new ods logging for bookmark fetching from phases crate
This would help with investigation sevs like S344849, also having this info we
can setup alerts.
Reviewed By: YousefSalama
Differential Revision: D46314640
fbshipit-source-id: 66e72da19ad4eb528c78d0783c297746f57cd4f1
Summary:
We've seen crashes on Windows when the mount is initialized but a later step at
mount time fails. In between these 2 events, if a write to the working copy
happens the ProjectedFS code assumes that the InodeMap is initialized and thus
tries to access the rootInode. This assumption is unfortunately broken by the
call to mountFinished and the code should instead follow the regular unmount
code path to make sure that no pending notifications are inflight.
I suspect NFS and FUSE also would exhibit the same issue.
Reviewed By: kmancini
Differential Revision: D46169999
fbshipit-source-id: ed1eb17dc2f084acb7e2d88c555150b3b8c6662e
Summary:
This offers a new interface for Shard Manager managed repos to be able to load redacted content.
The new API entry point: `open_managed_repos_unredacted` is analogous to `open_repo_unredacted`, but supports a Shard Manager based workflow.
Reviewed By: RajivTS
Differential Revision: D46280863
fbshipit-source-id: e2e4aebcb08d01f3be97d11d6fe8da2822ba6f29
Summary: Adding a way to configure the blake3 key for blake3 hashing
Reviewed By: chadaustin
Differential Revision: D44499469
fbshipit-source-id: 344edd91cfc867c8a49d0db9369dfe2132e86c77
Summary: I am planning to enable `copytrace.fallback-to-content-similarity=True` for hg repos, and will use this metric to monitor how useful the content similarity fallback is
Reviewed By: muirdm
Differential Revision: D46163311
fbshipit-source-id: d21b0288965f8ec4da7c6c80e37d1b557b357ea6
Summary: Our cache is in the dagcopytrace instance, so let's make it a singleton.
Reviewed By: quark-zju
Differential Revision: D46105882
fbshipit-source-id: 0984015e7b052482b27f950fc05563c432aa23b1
Summary: this function will be used in the following diff
Reviewed By: quark-zju
Differential Revision: D46105885
fbshipit-source-id: 9d02fb62a44b1e5bd18eba5236522e297fae0f64
Summary:
This diff modifies the behavior of `gitimport` and `remote-gitimport` when importing `annotated-tags`. In addition to:
- storing the raw tag
- converting the tag to a bookmark
- storing the metadata of the tag as changeset
this diff maintains a mapping between the tag-name and the changeset representing the metadata of the tag. This mapping would be useful to answer queries about the tag metadata when queried through Git and also enable going back to Git from Mononoke without losing data.
Differential Revision: D46067809
fbshipit-source-id: e2463160cdf647b50fa4ecfe4674d079016b4adc
Summary: Implement an alternative to ChangesetPathHistory::history using the new commit graph. This is almost identical to the skiplist implementation except that an (incorrect) optimization was removed that was trying to minimize the number of is_ancestor queries which should no longer be needed as they are a lot faster with the new commit graph.
Differential Revision: D46109911
fbshipit-source-id: 3e8f0c36a48084116673dda3741de87e0741f1a2
Summary: Implement an alternative to ChangesetContext::history using the new commit graph by using AncestorsStreamBuilder. Also made the tests not depend on the exact output history order but instead accept any topological order.
Differential Revision: D46109910
fbshipit-source-id: 31f3e219e9addf04b49fc500ad245d541f110887
Summary:
Factors out the logic of creating an ancestors stream to a builder to allow for more customizability (required for ChangesetContext::history) and reduce code duplication.
Additionally this optimizes range_stream to only traverse changesets that are descendants of start_id. Ancestry checks are kept to a minimum (worst case O(merges)) by only checking on parents of changesets that have a skip tree parent with generation lower than the generation of start_id.
Reviewed By: liubov-dmitrieva
Differential Revision: D46276429
fbshipit-source-id: 65c4f2892c92baaab0fef4f5041f06095ec49e05
Summary: allow using `SqliteOverlay` on non-Windows, since OverlayChecker has now been updated to work with it.
Reviewed By: chadaustin
Differential Revision: D44105360
fbshipit-source-id: 5f78e6c9b5323faeee394da4a51d7e84f8d44a2f
Summary:
In the rare case where fetching both the BlobMetadata and the Blob failed, the
LocalStoreCachedBackingStore would still try to dereference the nullptr Blob,
leading to a crash.
Reviewed By: chadaustin
Differential Revision: D46211469
fbshipit-source-id: f98dd215dbda432507f3823439d354d3d662dc6f
Summary: We didn't have a way to do this before so let's add it.
Reviewed By: mitrandir77
Differential Revision: D46190481
fbshipit-source-id: 84544f0226ef3a4ef4d8ac9b4bf66551e4671793
Summary: It's not ideal that we pull mononoke_app and even instantiate MononokeApp here. This is an attempt to decouple Mononoke from Metagit.
Reviewed By: mitrandir77
Differential Revision: D45862434
fbshipit-source-id: 6eab3577743810461f22535d89cf9a00dc1cbb65
Summary: Most of the useful ways to use these methods requires providing an async closure so let's asyncify them.
Reviewed By: RajivTS
Differential Revision: D45607981
fbshipit-source-id: ee7b471cc5d60a793a79928f6c15c48c0da1d08e
Summary: Currently, the code scans the repo when revision translation is failed, which is very slow.
Reviewed By: muirdm
Differential Revision: D46200075
fbshipit-source-id: 31eaca43582b480a0b3abd7af0b27ec8d6406195
Summary:
We get both `added` and `deleted` files in the function, this will be
needed for directory move detection.
Reviewed By: muirdm
Differential Revision: D46105884
fbshipit-source-id: d0081f8afaf27ef4ab105ab7c58b3f5cc694debc
Summary:
We've seen a case where a large `hg update` was taking an absurdly long time in
`ObjectStore::getTree` but the telemetry was showing us that most of the time
wasn't spent fetching trees from Mercurial! The suspicion is that most of the
time was spent in the LocalStore but with no evidence to prove it.
Let's thus add some timing telemetry to various LocalStore read requests to
fill this gap.
Reviewed By: mshroyer
Differential Revision: D46154456
fbshipit-source-id: b439ac48889ed3db71db136ff6c1cc91f48c926a
Summary:
Extends `FsckTest` to test `SqliteOverlay`. Two major changes were made to non-test files:
1) `loadInodeInfo` was moved from `OverlayChecker` into `InodeCatalog`
2) `getAllParentInodeNumbers` was added to facilitate `OverlayChecker::readInodes`
Reviewed By: chadaustin
Differential Revision: D45447172
fbshipit-source-id: 9ad291a0c4c34f1e44e1f05c1fdbf2ac7ea7d309
Summary: In the case of a multiplexed blobstore, the put operation can exit after it succeeds in one inner blobstore before finishing in all, and leave the rest running in the background. This sleep tries to prevent exiting early before they all finish.
Reviewed By: mitrandir77
Differential Revision: D46190037
fbshipit-source-id: ac2ae0d1453ebdc923170b9da7ca98df7519bf52
Summary: This will no longer be tied to the fscatalog, so it makes more sense to live in a more general location
Reviewed By: chadaustin
Differential Revision: D44980396
fbshipit-source-id: 3cabf00d87df7eb00671c2b82b04c8f12283e394
Summary: Implements a public method changeset_children in CommitGraph that fetches the changeset ids of the children of a given changeset.
Differential Revision: D46185434
fbshipit-source-id: f0b2e2ee53c7a34fe94ee5ef1f74f6e31429d056
Summary: used cached version of API
Reviewed By: YousefSalama
Differential Revision: D46184380
fbshipit-source-id: 5deb8c455ad682afcf1f0cad822d34fdd632b88b
Summary:
Telemetry exist for where Blob and BlobMetadata are coming from, but not for
Tree. Let's bring the Tree telemetry up to par.
Reviewed By: genevievehelsel
Differential Revision: D46154455
fbshipit-source-id: ab3c31d55b6a91009289b3b07853fa574bbaa137
Summary:
I migrated the last caller of debugGetScmBlobMetadata to debugGetBlobMetadata
in the last diff. Let's delete the old method.
I am not really concerned about waiting for that lower diff to rollout and all
because this is a debugging command that we only sparingly ask users to run
(As far as I know like maybe 3 times).
Reviewed By: chadaustin
Differential Revision: D45972617
fbshipit-source-id: 6d4689a225fee972f8e3a802115852043f0bade8
Summary:
I created debugGetBlobMetadata a while ago and this is the last caller of
debugGetScmBlobMetadata. Let's migrate it to the new endpoint so we can delete
the old one.
Reviewed By: chadaustin
Differential Revision: D45972618
fbshipit-source-id: cbb0977260f9a1543d3bf0a31bf7b948c2f35143
Summary:
I created debugGetBlob a while ago and there are no more callers of
debugGetScmBlob from code search, so let's delete this.
Reviewed By: chadaustin
Differential Revision: D45972619
fbshipit-source-id: 97276ec2182fbe7d9022a06e9a0df34ae4e80235
Summary: The admin tool currently doesn't register the `fb303` extension with `MononokeApp` during its creation. This makes the tool incapable of registering for health check and responding to thrift health check requests by TW. In absence of ALIVE response from the job, TW ends up killing it. This diff adds the extension during `MononokeApp` construction. This will not impact admin tool's usage through CLI but the error messages in the integration tests will need to be updated.
Differential Revision: D46147128
fbshipit-source-id: f06b8bcdce938346726b9ff520a3d809d9fda6fc
Summary: Makes it easier to rollback
Reviewed By: mitrandir77
Differential Revision: D46156526
fbshipit-source-id: eae70801069d09ab03b3b683712d9353fb1e5a43
Summary:
EdenFS contains several layers of caching for BlobMetadata:
- In the hgcache,
- In the in-memory BlobMetadata cache,
- In the LocalStore (configurable),
- In the FileInode,
- In the TreeEntry,
All of the above must be filled prior to reading a blob, therefore computing
the sha1+size when reading a blob is likely not needed and superfluous.
Alternatively, we could move the code to be done in the background in the
LocalStoreCachedBackingStore to avoid blocking reading the blob. Reviewers
opinions are welcome!
Reviewed By: chadaustin
Differential Revision: D45625432
fbshipit-source-id: ad7d897c32cbfd590dce5a3a9c8073349e9b70c8
Summary: The bookmark `get` command simply displays the changeset that is pointed to by the bookmark. This diff modifies the command to also handle git annotated tags converted into bookmarks. If the bookmark is a tag bookmark, then the command also outputs the metadata changeset associated with the bookmark. This will be useful for validating that imported annotated tags retain their metadata through metadata changesets.
Differential Revision: D46022692
fbshipit-source-id: b3980f4799066913268ef7acc1f4956d3feb5b23
Summary: `BonsaiTagMapping` exists as a facet so let's add the appropriate builder methods in `RepoFactory` and `TestRepoFactory`. Follow-up diffs will add this new facet in the `Repo` facet container for `SCS` (for `remote-gitimport`) and in `gitimport`.
Differential Revision: D46020528
fbshipit-source-id: 88e65a498355bf21edc8c8d8aa3755a5507fe991
Summary:
The `metadata-diff` endpoint in SCS is used by Phabricator to get information about the difference between files of two commits. Currently, this endpoint works using raw file content. This diff makes the following optimizations:
- Avoids loading file content in memory if the file is `binary` or `non-utf8`
- Avoids parsing `generated` and `partially-generated` tags if the file is known to be neither
- Avoid parsing raw file bytes into UTF-8 string if the file is non-generated
These optimizations will be most helpful for the case of `binary` or `non-utf8` files but should improve performance of the endpoint in general.
Reviewed By: YousefSalama
Differential Revision: D46109336
fbshipit-source-id: b3c915005686f77defceeb53f1587049bc8b6bd8
Summary:
For tools that want to take advantage of the same fast-path logic when
directories don't change across updates, expose a semistable ID they
can use to cache derived data or get a rough understand when a
directory has changed its contents.
Reviewed By: kmancini, xavierd
Differential Revision: D45974142
fbshipit-source-id: 7b2b482876b07e73514a936e198de2dc31ed1597
Summary:
thrift_test is pretty general. For something like getFileAttributes,
getFileAttributesV2, and readdir, it's nice to have them in one file.
Reviewed By: kmancini
Differential Revision: D45974095
fbshipit-source-id: ca5ecd4795f1d278f4de5b83cb8c9af94c111902
Summary: Implement an alternative to slice_repository using the new commit graph that doesn't depend on the skiplist. Useful when doing any kind of processing that requires ancestors to be processed first (backfilling derived data, mirroring commits, ...)
Reviewed By: mitrandir77
Differential Revision: D45607347
fbshipit-source-id: b2012972472e718522e4f33ff01fc9b253b76fff
Summary: Makes slightly more sense
Reviewed By: mitrandir77
Differential Revision: D45607346
fbshipit-source-id: 6b505796da96ff24cd7d555290dcf341f93c797f
Summary: My fingers are tired from scrolling
Reviewed By: mitrandir77
Differential Revision: D45607349
fbshipit-source-id: fe366797a37fbd1682ca917d6bdd39c88195c8e9
Summary: common_base returns all of the highest generation changesets that are common ancestors of two changesets. Implementation is optimized for the most common case were there are no merges between the two changesets and their lowest common ancestor (taking O(log(height) iterations in that case), but is sound for all cases.
Reviewed By: mitrandir77
Differential Revision: D45480232
fbshipit-source-id: 147b6feaa57cdbeb5efb4b77335728f38105fb87
Summary:
The changed something in 0.5 version, and cargo compilation of buck2 broke.
Let's just use latest version.
Reviewed By: zertosh
Differential Revision: D46106140
fbshipit-source-id: 67f4f63eff8e80be8408089003f1171e8b0e1865
Summary:
The bookmark update log is sequential and we can't write two things to it at
once (for a single repo). The aosp megarepo infra sometimes moves two bookmarks
concurrently and hits this ugly race: P743467889. Let's add retries to failed
bookmark moves.
Reviewed By: YousefSalama
Differential Revision: D46105621
fbshipit-source-id: 5c822301fc3201b070c1fe29eda6c395d265c33f
Summary: This diff adds unit tests to validate that the `BonsaiTagMapping` implementation works as expected.
Reviewed By: mitrandir77
Differential Revision: D46020108
fbshipit-source-id: 12e8268363f0917645a5eadd37eff321b510b755
Summary: Constructs a preloaded commit graph when preloaded_commit_graph_blobstore_key is set in repo config.
Reviewed By: RajivTS
Differential Revision: D45993556
fbshipit-source-id: 3dd22617cbabcc007d02965e7392eb2f5283b1b8
Summary: We need a way to specify which repos will use a preloaded commit graph and the blobstore key of the commit graph blob in that case.
Reviewed By: RajivTS
Differential Revision: D45993558
fbshipit-source-id: 4cf2ea84b72e3a50187fab3ece86e623a754d571
Summary: Adds a new command update-preloaded that tries to load a blob of the preloaded commit graph, update it with any newly added changeset edges, and re-upload it to the blobstore.
Reviewed By: mitrandir77
Differential Revision: D45993560
fbshipit-source-id: 4d14cbf091d86f165ed58fbba9305f0fa6021632
Summary: Implements a commit graph storage for preloading commit graph edges in memory. Useful for repos with complicated commit graph structures but relatively small overall size. It uses Reloader to periodically update the preloaded edges. CompactChangesetEdges are introduced to lower the memory usage by referencing other changesets using interned ids.
Reviewed By: mitrandir77
Differential Revision: D45992604
fbshipit-source-id: 5852079d10a0ba8c491e7b239d8cf4c58813dd45
Summary: These are not specific to caching so let's move them
Reviewed By: RajivTS
Differential Revision: D45993559
fbshipit-source-id: b5f127bf2997c4bf5381013df10af28f1c42c1c9
