Summary: Untested code breaks silently. Tested code also breaks, but we get to know exactly when it happens.
Reviewed By: yancouto
Differential Revision: D34899023
fbshipit-source-id: b50fd4ff1c75853bdd3dae443160d8f630fdb5be
Summary: For mutable renames tests, I'll need copy-from commits. Make drawdag generate them, because it's awesome to not have to run a sequence of Mercurial commands
Reviewed By: Croohand
Differential Revision: D34898156
fbshipit-source-id: f5e73fca63ae391d934b73da6cd3680179b4cd34
Summary: When you add a new mutable renames entry to a commit, we should copy over immutable renames automatically, then override with the newly added one.
Reviewed By: Croohand
Differential Revision: D34893586
fbshipit-source-id: cde687a1e84979f2a57ced4d9e57f0de918d30cc
Summary:
For commits using mutable renames, we don't inspect immutable renames.
Make it possible to copy immutable renames to mutable renames, overwriting any overlapping mutable renames
Reviewed By: Croohand
Differential Revision: D34893588
fbshipit-source-id: 8fde522852b9390be7593ec8f29c36357bcf34de
Summary: Make it easy to add mutable renames by admin intervention. This does recursion into subdirectories automatically, so that one command can be used to add mutable renames markers for an entire directory tree.
Reviewed By: Croohand
Differential Revision: D34893587
fbshipit-source-id: e126bbcc507ec0bdb366f6b9caa90fdff64c602d
Summary:
EdenFS doesn't want to use retries while it has the hg import
fallback. Let's allow disabling them. Once there is a config that controls the hg
import fallback, we can enable retries when the fallback is disabled.
Reviewed By: xavierd
Differential Revision: D35020586
fbshipit-source-id: d51d7fb70d2627e55c1170f130c548d7b14a8e6e
Summary:
The diff includes the following key changes:
- Bounded the range of keys for enumeration in `PrefixBlobstore` by prepending the `prefix` for both empty and non-empty keys.
- In case of empty `end` keys, added `prefix + ascii_end_character` for limiting the search results to the prefixed namespace.
- Fixed enumeration in `ManifoldBlobstore` by expanding the struct definition to include a user-specified `directory`. In case of `None`, the `flat` directory (i.e. no directory) is used.
- The input keys for `get`, `put`, `is_present`, `copy` and `unlink` are now first qualified with the directory path before being used for the relevant operations in Manifold and ThriftManifold blobstores.
- Updated `ManifoldBlobstore` and `ThriftManifoldBlobstore` creation sites to explicitly pass the directory or `None`.
Reviewed By: farnz
Differential Revision: D34996443
fbshipit-source-id: d7b7a7d034b3cce15f64472a08f415494fda2746
Summary:
I got a task about a flaky LFS server test being disabled. The failure
was easy to reproduce when running 10 concurrent tests in opt mode.
The test was verifying that the LFS server will fetch content from its own
internal store if the upstream server (in this case netcat that sleeps for a
second) is slow to respond.
The flakiness was because sometimes the upstream server was quicker to respond
than checking our own internal store.
After some debugging, I found that this was because the server was failing to
lookup the name of the upstream. This meant that the test was comparing whether
it was faster to fail a DNS lookup compared to checking internal storage for
content.
```
+ V0322 12:39:45.061522 2262733 [tk] eden/mononoke/lfs_server/src/batch.rs:151] upstream_objects, request_id: 5afbf
+ V0322 12:39:45.061622 2262733 [tk] eden/mononoke/lfs_server/src/batch.rs:161] upstream_objects pre res, request_id: 5afbf
+ V0322 12:39:45.061681 2262733 [tk] eden/mononoke/lfs_server/src/lfs_server_context.rs:378] upstream: http://:48155/objects/batch, request_id: 5afbf
+ V0322 12:39:45.061839 2262733 [tk] eden/mononoke/lfs_server/src/batch.rs:301] internal_objects, request_id: 5afbf
+ V0322 12:39:45.062321 2262733 [tk] eden/mononoke/lfs_server/src/batch.rs:571] batch: internal ready, request_id: 5afbf
+ V0322 12:39:45.062728 2262733 [tk] eden/mononoke/lfs_server/src/batch.rs:163] upstream_objects: res: Err(UpstreamBatchNoResponse(An error occurred forwarding the request to upstream
+
+ Caused by:
+ 0: error trying to connect: dns error: failed to lookup address information: Name or service not known
+ 1: dns error: failed to lookup address information: Name or service not known
+ 2: failed to lookup address information: Name or service not known)), request_id: 5afbf
+ V0322 12:39:45.062896 2262733 [tk] eden/mononoke/lfs_server/src/batch.rs:589] batch: upstream ready, request_id: 5afbf
```
This change was introduced in D34923381 (acb8e3725a), which updated `${LOCALIP}` to
`${mononoke_host}`. However, `LOCALIP` is a variable name whilst
`mononoke_host` is a function name. Update the syntax to reflect this.
Reviewed By: mzr
Differential Revision: D35046184
fbshipit-source-id: 5ec9c29e2e9e8658ddabe29c755debdfb3a49938
Summary:
Our use of RCU is fundamentally incompatible with N4895 since it allows a read
section to be entered in one thread and closed in another one. We use this
feature in Prjfs to prevent a mount to be unmounted while requests are ongoing
on it.
Let's instead make use of a folly::AtomicReadMostlyMainPtr which is equivalent
to a folly::Synchronized<std::shared_ptr<>> without requiring as many atomics
in the read path.
Reviewed By: Decave
Differential Revision: D34982722
fbshipit-source-id: 299f25fc8c8aa72fb89cc9fc880860780ea0517d
Summary: Migrate blobstore_healer tool (which we use to continuously verify if all our underlying storages agree about contents of blobs and fix any discrepancies) to new framework.
Reviewed By: mitrandir77
Differential Revision: D34859855
fbshipit-source-id: 5814fd275f22113689afe25b55c28a9d5b062551
Summary:
On Windows, some users have experienced hangs during checkout, and their
backtraces points to a directory invalidation being run in the server state
thread pool. Most of these have been moved to a background thread in D33773892 (67e6d48b0d)
but one was missed: when removing an empty directory, a directory invalidation
is done in a blocking manner.
To fix this, this diff moves the location of where this removal is happening,
from the post checkout step, to the parent inode. This avoids having to make
the removal method async.
Reviewed By: fanzeyi
Differential Revision: D34930808
fbshipit-source-id: 76762dab5a8143ef37ca49cb6386e6df73e47d9f
Summary:
In order to add solana-client = "1.9.5" to third-party crates in fbcode, we need to update the versions of following crates.
- sha2
- zstd related including zstd, zstd-sys, zstd-safe
Next step is to add solana-client.
Reviewed By: wqfish
Differential Revision: D34648217
fbshipit-source-id: 719e499d9fbb4fd35624f3b9c46ffc6dfc94576d
Summary:
Sometimes we need to make a loud announcement to all users of a given repo.
(For example to tell them that their repo will go away soon). This allows us to
do it without further code changes.
Reviewed By: farnz
Differential Revision: D34934883
fbshipit-source-id: 771ee0c32d3435650a872efb7191350fdb40d12d
Summary:
Key changes in this diff include:
- Introduced the `ephemeral_store` new admin command for working with data stored within bubbles (snapshots for now).
- Implemented the outline for `list`, `cleanup` and `info` subcommands within `ephemeral_store`.
- Added deletion checks based on `BubbleDeletionMode` configuration flag within `ephemeral_store`.
Reviewed By: yancouto
Differential Revision: D34936192
fbshipit-source-id: 4f0b1f759112585ad165332e141687be0c39cedd
Summary: Add the requested URL to the HttpError details. This makes it clear what was being requested when we got an unexpected HTTP response.
Reviewed By: DurhamG
Differential Revision: D34907302
fbshipit-source-id: 18a7da77038b6c8acd1933465a31f5744ed4d8fc
Summary: Enums in rust are one of the ways the Rust compiler checks for edge cases, so I'd love to use them in my code.
Reviewed By: dalyadickstein
Differential Revision: D34932071
fbshipit-source-id: 4e5fd23d34e462f9e5a533c917704ace1afe99e1
Summary:
A EdenFS messenger dogfooder on macOS is hitting our check here that file
handles be a certain size.
This likely indicates that we have a big somewhere in parsing that is causing
us to parse something that is not a file handle as a file handle. Its also
possible this request could be malformed.
Eventually we should not trust clients enough to allow them to crash eden with
a malformed request, but the first step to fixing this bug is understanding
wether the request that crashes us is malformed or not.
So first I am adding some extra logging info to this xcheck so that we can
determine the cause. I will follow up with a solution to prevent clients
from crashing EdenFS.
Reviewed By: xavierd
Differential Revision: D34969740
fbshipit-source-id: 783ca1147f6c9d9c86996836d14263a0a79d6b7d
Summary:
Tuple structs as part of public interfaces are hard to modify, and result in
`.0` appearing all over the code. Make the field private and add well-defined
methods with appropriate visibility.
Reviewed By: farnz
Differential Revision: D34971449
fbshipit-source-id: 3078469527e4021dceef5dcb6a5f5f2f980d836a
Summary:
`mononoke_admin create-bonsai` is a dangerous command as it does not check the
bonsai is valid. It is used in tests to test Mononoke correctly handles
invalid bonsai changesets.
Move it to the test tool so it is clear that it is only for use in tests.
Reviewed By: farnz
Differential Revision: D34892793
fbshipit-source-id: dd11b9e33920d06883258c1051eadc98e9ea3d81
Summary: Port the split-commit command to the new admin tool (as `newadmin commit split`).
Reviewed By: yancouto
Differential Revision: D34864663
fbshipit-source-id: 303b1ef568a903f38d8eff33924c5d8a750ee48a
Summary:
To allow tests that use the `tests_utils` crate to work with repos other than
`BlobRepo`, migrate this crate to using facet reference definitions.
In a lot of tests, this was the only thing that required `BlobRepo`, which
means type inferencing now does not know which concrete type to use. In these
cases, we specify `BlobRepo` for now. In the future these can be changed to a
simpler repo type for that test.
Reviewed By: yancouto
Differential Revision: D34864664
fbshipit-source-id: e4142e8e47c3e942a5d9ceac4ba4270fe597f097
Summary:
Separate the code to save new changesets to a repo out from the blobrepo crate.
In the spirit of `bookmarks_movement`, this is named `changesets_creation`.
Reviewed By: yancouto
Differential Revision: D34864665
fbshipit-source-id: 29931f6e4f619c52fce71f8277fa8925d67c66f4
Summary:
Port the `redaction` subcommand to the new admin tool.
There are a couple of changes vs the old tool:
* Old-style SQL redactions are not supported. We can continue to use the
old admin tool until they are removed.
* We use fsnodes, rather than hg manifests to find the content ids. This avoids
extra lookups between hg and bonsai hashes, and means we no longer depend on
Mercurial for redaction.
Reviewed By: yancouto
Differential Revision: D34790119
fbshipit-source-id: 575581675abf42b6048f61e4a449fa3326ba838b
Summary:
- Using the newly added `ints_64` and `int_64_lists` configs (see D34962606 (86bb486a78)).
- Integration tests + unit tests were also updated to use these versions for commitsize.
Reviewed By: mitrandir77
Differential Revision: D35007018
fbshipit-source-id: 572cec88e67f3cb9184ff40e5998df2752a82837
Summary:
The `Eq`, `PartialOrd`, `Ord`, and `Hash` traits would be useful when working
with the `CommitId` type in Rust. This diff adds those derives through the
`rust.ord` Thrift annotation.
Reviewed By: singhsrb
Differential Revision: D34911454
fbshipit-source-id: bb893df3a2ee468ffa8958df2442e14ffe461cc3
Summary:
- udpated thrift definition to support i64 ints;
- updated metaconfig loader type to parse and load the new 64bit properties;
Reviewed By: mzr
Differential Revision: D34962606
fbshipit-source-id: 95653139b2149bb9df2899b8b703fa58ee4473c3
Summary:
It fails sometimes:
{F711107980}
```
Start and wait for verify_integrity_service
$ verify_integrity_service
+ Verify Integrity service did not start in 30 seconds, took 30
```
Reviewed By: farnz
Differential Revision: D34896341
fbshipit-source-id: 9a39c30236f43140507d5abe17866d86a778b24f
Summary:
Add support for the `--reuse-message` option to `hg fold`, allowing re-use of a
particular commit message for the folded commit.
The fold code uses common infrastructure for creating commits, so we just need
to add it as a permitted flag.
Reviewed By: DurhamG
Differential Revision: D34966420
fbshipit-source-id: 32e7f06ddb9e20dc55665e0924dead943dafdc9b
Summary:
This diff finishes up the main functionality in `workingcopy`'s `pendingchanges`. Files that may or may not have changes relative to p1 (based on a purely metadata check) now have their contents checked against p1.
In another version of this diff, I used `scmstore` to query the SHA-256 hash of the contents. However, I was told that I should prefer using the traits from `storemodel`. So, for this diff, I'm using `ReadFileContents`. This means that we do end up having to read the entire contents from the repo (not just the hash). However, maybe it's not a big deal because this is only for a physical filesystem anyway.
`ReadFileContents` is an async trait, which doesn't fit in that well with the surroundings (e.g. looking up a key for a path in the manifest isn't async). In the future, when more of these models are available with async, we may consider reworking the logic to use async for all of the I/O.
Future work: update stored mtime of files that haven't changed so we don't have to keep re-checking them.
Reviewed By: DurhamG
Differential Revision: D34569176
fbshipit-source-id: 5ca324cde30cafda981605c747376e55459a4f70
Summary: This diff improves the core functionality of the ExactMatcher in the previous diff to intelligently return whether or not a directory needs to be traversed by checking an internal tree structure to see if any of the matcher's files are in that directory. This should improve the performance of Matcher-based tree traversal.
Reviewed By: DurhamG
Differential Revision: D34912969
fbshipit-source-id: 51349938b3a58de5462a77f0519c021014f3b228
Summary:
The pathmatcher crate has a `Matcher` trait that represents something that can match or not match file paths. It also allows for an efficient traversal of a tree, by exposing the ability to query directories and see if 1) all files match 2) some files may match (and thus should be traversed) or 3) no files match.
The Matcher trait is used in a few places. We're currently missing an efficient "exact matcher", that takes a list of files it should match and only matches those. This will be used to look up keys for file contents from a TreeManifest.
This initial version is dumb: it stores all of the paths in a HashSet and checks against it. It always returns "ShouldTraverse" -- that is, it doesn't prune the tree when traversing. The next diff will change this to only require traversing directories that contain some files.
Reviewed By: DurhamG
Differential Revision: D34912970
fbshipit-source-id: a4f69e5f538ad0cd4b6940368d946922b82e2535
Summary: Change command to take OptionalRepo and set a default reponame if not present (this is required for constructing valid edenapi URLs).
Reviewed By: DurhamG
Differential Revision: D34839331
fbshipit-source-id: fbbea1437909b6c3efa0561eea108cf81da3bf68
Summary:
This way repo-optional commands run outside a repo will have a more complete config to work with (e.g. debugnetworkdoctor).
Some stuff already supported the no-repo case, but it wasn't consistent. These were the tweaks I made:
- If we don't have a repo path we use the "config dir" as our dynamic config cache directory. The config dir is "dirs::cache_dir()/edenscm". We now also create the directory if it doesn't exist.
- We don't try to figure out the repo name in load_dynamic if we have no repo path.
I updated a few dynamic config related debug commands to respect the new optional arg.
Reviewed By: DurhamG
Differential Revision: D34839333
fbshipit-source-id: 3687d71467d9fe96b26d89c356cdb51c8f73de62
Summary:
Add utility function to create specified directory and any ancestors as necessary.
A similar method exists in indexedlog utils, but has different semantics than the dir stuff in "util", so I opted to keep it separate.
Reviewed By: DurhamG
Differential Revision: D34874922
fbshipit-source-id: cd39a7ad2a26af586f61a39b9fb63c6d4c72ec09
Summary: Give optional repo commands OptionalRepo instead of Option<Repo>. This way they have access to the global/non-repo config object when not inside a repo.
Reviewed By: DurhamG
Differential Revision: D34839334
fbshipit-source-id: 2891d85a96a38c772929257cce37ef574aaad8d5
Summary: I hit a curl timeout error on the shuttle when testing. Now we handle that and give a generic "check your network" message. It is possible it is a server problem, but I think much more likely a local network issue.
Reviewed By: DurhamG
Differential Revision: D34803930
fbshipit-source-id: 0936f6a0b96a26fb7afdc89f093a867198650efc
Summary: For testing, sometimes plain http is simpler. To suppor this, now the network doctor only assumes there should be an auth section if the edenapi url scheme is "https".
Reviewed By: DurhamG
Differential Revision: D34803929
fbshipit-source-id: 3d780b591834b2a958efc725b4cc40f66e3b7eef
Summary: Now the network doctor invokes the existing auth::check_certs function which catches unparseable and expired certs. I also included the tlsauthhelp/tlshelp messages as appropriate.
Reviewed By: DurhamG
Differential Revision: D34803931
fbshipit-source-id: 07915788105709a626320627c91e2d64f329dee0
Summary: Include a brief version of the diagnosis in dev_command_timers so we can see how often the network doctor makes diagnoses. I set the column to "ok" if the doctor finds no problem so that there is a positive indication the doctor ran and found no problem.
Reviewed By: DurhamG
Differential Revision: D34803932
fbshipit-source-id: 7ef833b325283eb8a299c57ad23fbdb107b5aeba
Summary: Make response bodies print out in text based on response content-type (currently only for text/html and application/json).
Reviewed By: DurhamG
Differential Revision: D34770197
fbshipit-source-id: 52d239e26edf4ac892173622c5c35787546c475a
Summary:
Start with a few cases:
- suggest the user check their svnuser membership on 403
- extract x2pagentd and proxygen hints from the response headers
Reviewed By: DurhamG
Differential Revision: D34770198
fbshipit-source-id: b7eada4a33debd3e6dc8593c9ef8adcf7b358021
Summary: I'm expanding HttpError to mean something like "we had a problem when making an http request". It isn't used very much in Python, and I'm co-opting it to be one of the key triggers for running the network doctor automatically.
Reviewed By: DurhamG
Differential Revision: D34739986
fbshipit-source-id: daea571b06bf030257c6acf534af45abbbf2048c
Summary:
Now the network doctor also attempts to make http requests to suss out what is wrong. We request /edenapi/:repo/capabilities since this will trigger permission errors (as opposed to /health_check). It tries with and without x2pagentd (if configured) to further triangulate the source of the problem.
This commit only contains high level categorization of the problem. I will follow up with the "diagnosis" which will start to map HTTP responses into informational messages for users.
Reviewed By: DurhamG
Differential Revision: D34734186
fbshipit-source-id: 794c5dd8a89d6d0cfc2068b8e3f5cc98e6b7959b
Summary: Update debugnetworkdoctor cli command to produce the user facing "treatment" option along with error details.
Reviewed By: DurhamG
Differential Revision: D34649740
fbshipit-source-id: 5bafa00904d938e0676aa6a00ce89520470b48bb
