Summary:
Key changes in this diff include:
- Introduced the `ephemeral_store` new admin command for working with data stored within bubbles (snapshots for now).
- Implemented the outline for `list`, `cleanup` and `info` subcommands within `ephemeral_store`.
- Added deletion checks based on `BubbleDeletionMode` configuration flag within `ephemeral_store`.
Reviewed By: yancouto
Differential Revision: D34936192
fbshipit-source-id: 4f0b1f759112585ad165332e141687be0c39cedd
Summary: Add the requested URL to the HttpError details. This makes it clear what was being requested when we got an unexpected HTTP response.
Reviewed By: DurhamG
Differential Revision: D34907302
fbshipit-source-id: 18a7da77038b6c8acd1933465a31f5744ed4d8fc
Summary: Enums in rust are one of the ways the Rust compiler checks for edge cases, so I'd love to use them in my code.
Reviewed By: dalyadickstein
Differential Revision: D34932071
fbshipit-source-id: 4e5fd23d34e462f9e5a533c917704ace1afe99e1
Summary:
A EdenFS messenger dogfooder on macOS is hitting our check here that file
handles be a certain size.
This likely indicates that we have a big somewhere in parsing that is causing
us to parse something that is not a file handle as a file handle. Its also
possible this request could be malformed.
Eventually we should not trust clients enough to allow them to crash eden with
a malformed request, but the first step to fixing this bug is understanding
wether the request that crashes us is malformed or not.
So first I am adding some extra logging info to this xcheck so that we can
determine the cause. I will follow up with a solution to prevent clients
from crashing EdenFS.
Reviewed By: xavierd
Differential Revision: D34969740
fbshipit-source-id: 783ca1147f6c9d9c86996836d14263a0a79d6b7d
Summary:
Tuple structs as part of public interfaces are hard to modify, and result in
`.0` appearing all over the code. Make the field private and add well-defined
methods with appropriate visibility.
Reviewed By: farnz
Differential Revision: D34971449
fbshipit-source-id: 3078469527e4021dceef5dcb6a5f5f2f980d836a
Summary:
`mononoke_admin create-bonsai` is a dangerous command as it does not check the
bonsai is valid. It is used in tests to test Mononoke correctly handles
invalid bonsai changesets.
Move it to the test tool so it is clear that it is only for use in tests.
Reviewed By: farnz
Differential Revision: D34892793
fbshipit-source-id: dd11b9e33920d06883258c1051eadc98e9ea3d81
Summary: Port the split-commit command to the new admin tool (as `newadmin commit split`).
Reviewed By: yancouto
Differential Revision: D34864663
fbshipit-source-id: 303b1ef568a903f38d8eff33924c5d8a750ee48a
Summary:
To allow tests that use the `tests_utils` crate to work with repos other than
`BlobRepo`, migrate this crate to using facet reference definitions.
In a lot of tests, this was the only thing that required `BlobRepo`, which
means type inferencing now does not know which concrete type to use. In these
cases, we specify `BlobRepo` for now. In the future these can be changed to a
simpler repo type for that test.
Reviewed By: yancouto
Differential Revision: D34864664
fbshipit-source-id: e4142e8e47c3e942a5d9ceac4ba4270fe597f097
Summary:
Separate the code to save new changesets to a repo out from the blobrepo crate.
In the spirit of `bookmarks_movement`, this is named `changesets_creation`.
Reviewed By: yancouto
Differential Revision: D34864665
fbshipit-source-id: 29931f6e4f619c52fce71f8277fa8925d67c66f4
Summary:
Port the `redaction` subcommand to the new admin tool.
There are a couple of changes vs the old tool:
* Old-style SQL redactions are not supported. We can continue to use the
old admin tool until they are removed.
* We use fsnodes, rather than hg manifests to find the content ids. This avoids
extra lookups between hg and bonsai hashes, and means we no longer depend on
Mercurial for redaction.
Reviewed By: yancouto
Differential Revision: D34790119
fbshipit-source-id: 575581675abf42b6048f61e4a449fa3326ba838b
Summary:
- Using the newly added `ints_64` and `int_64_lists` configs (see D34962606 (86bb486a78)).
- Integration tests + unit tests were also updated to use these versions for commitsize.
Reviewed By: mitrandir77
Differential Revision: D35007018
fbshipit-source-id: 572cec88e67f3cb9184ff40e5998df2752a82837
Summary:
The `Eq`, `PartialOrd`, `Ord`, and `Hash` traits would be useful when working
with the `CommitId` type in Rust. This diff adds those derives through the
`rust.ord` Thrift annotation.
Reviewed By: singhsrb
Differential Revision: D34911454
fbshipit-source-id: bb893df3a2ee468ffa8958df2442e14ffe461cc3
Summary:
- udpated thrift definition to support i64 ints;
- updated metaconfig loader type to parse and load the new 64bit properties;
Reviewed By: mzr
Differential Revision: D34962606
fbshipit-source-id: 95653139b2149bb9df2899b8b703fa58ee4473c3
Summary:
It fails sometimes:
{F711107980}
```
Start and wait for verify_integrity_service
$ verify_integrity_service
+ Verify Integrity service did not start in 30 seconds, took 30
```
Reviewed By: farnz
Differential Revision: D34896341
fbshipit-source-id: 9a39c30236f43140507d5abe17866d86a778b24f
Summary:
Add support for the `--reuse-message` option to `hg fold`, allowing re-use of a
particular commit message for the folded commit.
The fold code uses common infrastructure for creating commits, so we just need
to add it as a permitted flag.
Reviewed By: DurhamG
Differential Revision: D34966420
fbshipit-source-id: 32e7f06ddb9e20dc55665e0924dead943dafdc9b
Summary:
This diff finishes up the main functionality in `workingcopy`'s `pendingchanges`. Files that may or may not have changes relative to p1 (based on a purely metadata check) now have their contents checked against p1.
In another version of this diff, I used `scmstore` to query the SHA-256 hash of the contents. However, I was told that I should prefer using the traits from `storemodel`. So, for this diff, I'm using `ReadFileContents`. This means that we do end up having to read the entire contents from the repo (not just the hash). However, maybe it's not a big deal because this is only for a physical filesystem anyway.
`ReadFileContents` is an async trait, which doesn't fit in that well with the surroundings (e.g. looking up a key for a path in the manifest isn't async). In the future, when more of these models are available with async, we may consider reworking the logic to use async for all of the I/O.
Future work: update stored mtime of files that haven't changed so we don't have to keep re-checking them.
Reviewed By: DurhamG
Differential Revision: D34569176
fbshipit-source-id: 5ca324cde30cafda981605c747376e55459a4f70
Summary: This diff improves the core functionality of the ExactMatcher in the previous diff to intelligently return whether or not a directory needs to be traversed by checking an internal tree structure to see if any of the matcher's files are in that directory. This should improve the performance of Matcher-based tree traversal.
Reviewed By: DurhamG
Differential Revision: D34912969
fbshipit-source-id: 51349938b3a58de5462a77f0519c021014f3b228
Summary:
The pathmatcher crate has a `Matcher` trait that represents something that can match or not match file paths. It also allows for an efficient traversal of a tree, by exposing the ability to query directories and see if 1) all files match 2) some files may match (and thus should be traversed) or 3) no files match.
The Matcher trait is used in a few places. We're currently missing an efficient "exact matcher", that takes a list of files it should match and only matches those. This will be used to look up keys for file contents from a TreeManifest.
This initial version is dumb: it stores all of the paths in a HashSet and checks against it. It always returns "ShouldTraverse" -- that is, it doesn't prune the tree when traversing. The next diff will change this to only require traversing directories that contain some files.
Reviewed By: DurhamG
Differential Revision: D34912970
fbshipit-source-id: a4f69e5f538ad0cd4b6940368d946922b82e2535
Summary: Change command to take OptionalRepo and set a default reponame if not present (this is required for constructing valid edenapi URLs).
Reviewed By: DurhamG
Differential Revision: D34839331
fbshipit-source-id: fbbea1437909b6c3efa0561eea108cf81da3bf68
Summary:
This way repo-optional commands run outside a repo will have a more complete config to work with (e.g. debugnetworkdoctor).
Some stuff already supported the no-repo case, but it wasn't consistent. These were the tweaks I made:
- If we don't have a repo path we use the "config dir" as our dynamic config cache directory. The config dir is "dirs::cache_dir()/edenscm". We now also create the directory if it doesn't exist.
- We don't try to figure out the repo name in load_dynamic if we have no repo path.
I updated a few dynamic config related debug commands to respect the new optional arg.
Reviewed By: DurhamG
Differential Revision: D34839333
fbshipit-source-id: 3687d71467d9fe96b26d89c356cdb51c8f73de62
Summary:
Add utility function to create specified directory and any ancestors as necessary.
A similar method exists in indexedlog utils, but has different semantics than the dir stuff in "util", so I opted to keep it separate.
Reviewed By: DurhamG
Differential Revision: D34874922
fbshipit-source-id: cd39a7ad2a26af586f61a39b9fb63c6d4c72ec09
Summary: Give optional repo commands OptionalRepo instead of Option<Repo>. This way they have access to the global/non-repo config object when not inside a repo.
Reviewed By: DurhamG
Differential Revision: D34839334
fbshipit-source-id: 2891d85a96a38c772929257cce37ef574aaad8d5
Summary: I hit a curl timeout error on the shuttle when testing. Now we handle that and give a generic "check your network" message. It is possible it is a server problem, but I think much more likely a local network issue.
Reviewed By: DurhamG
Differential Revision: D34803930
fbshipit-source-id: 0936f6a0b96a26fb7afdc89f093a867198650efc
Summary: For testing, sometimes plain http is simpler. To suppor this, now the network doctor only assumes there should be an auth section if the edenapi url scheme is "https".
Reviewed By: DurhamG
Differential Revision: D34803929
fbshipit-source-id: 3d780b591834b2a958efc725b4cc40f66e3b7eef
Summary: Now the network doctor invokes the existing auth::check_certs function which catches unparseable and expired certs. I also included the tlsauthhelp/tlshelp messages as appropriate.
Reviewed By: DurhamG
Differential Revision: D34803931
fbshipit-source-id: 07915788105709a626320627c91e2d64f329dee0
Summary: Include a brief version of the diagnosis in dev_command_timers so we can see how often the network doctor makes diagnoses. I set the column to "ok" if the doctor finds no problem so that there is a positive indication the doctor ran and found no problem.
Reviewed By: DurhamG
Differential Revision: D34803932
fbshipit-source-id: 7ef833b325283eb8a299c57ad23fbdb107b5aeba
Summary: Make response bodies print out in text based on response content-type (currently only for text/html and application/json).
Reviewed By: DurhamG
Differential Revision: D34770197
fbshipit-source-id: 52d239e26edf4ac892173622c5c35787546c475a
Summary:
Start with a few cases:
- suggest the user check their svnuser membership on 403
- extract x2pagentd and proxygen hints from the response headers
Reviewed By: DurhamG
Differential Revision: D34770198
fbshipit-source-id: b7eada4a33debd3e6dc8593c9ef8adcf7b358021
Summary: I'm expanding HttpError to mean something like "we had a problem when making an http request". It isn't used very much in Python, and I'm co-opting it to be one of the key triggers for running the network doctor automatically.
Reviewed By: DurhamG
Differential Revision: D34739986
fbshipit-source-id: daea571b06bf030257c6acf534af45abbbf2048c
Summary:
Now the network doctor also attempts to make http requests to suss out what is wrong. We request /edenapi/:repo/capabilities since this will trigger permission errors (as opposed to /health_check). It tries with and without x2pagentd (if configured) to further triangulate the source of the problem.
This commit only contains high level categorization of the problem. I will follow up with the "diagnosis" which will start to map HTTP responses into informational messages for users.
Reviewed By: DurhamG
Differential Revision: D34734186
fbshipit-source-id: 794c5dd8a89d6d0cfc2068b8e3f5cc98e6b7959b
Summary: Update debugnetworkdoctor cli command to produce the user facing "treatment" option along with error details.
Reviewed By: DurhamG
Differential Revision: D34649740
fbshipit-source-id: 5bafa00904d938e0676aa6a00ce89520470b48bb
Summary:
Exception handling knows about CertificateError and inserts the appropriate help text for you.
I tweaked the exception handling code to use the repo's ui object, if available. Things like ui.config("help", "tlsauthhelp") weren't working on the pre-repo barebones ui object since it doesn't include the repo's config.
Reviewed By: DurhamG
Differential Revision: D34649746
fbshipit-source-id: 36aee47120fa7cb1cf3862a172f98e9d05d6c2ca
Summary:
Now the top level python exception handling invokes the network doctor on certain exceptions that can correlate to network problems. This behavior is behind the experimental.network-doctor config flag.
I added another level of try/except so the network doctor gets first dibs at exceptions, and if no problem is found (or the doctor is disabled), the previous exception handlers will still run for the exception.
If there is a network problem detected by networkdoctor, the user will see the doctor's "treatment" suggestion, e.g. "Please check your VPN", etc. With --verbose, the user will also see the detailed network doctor error. With --debug, the user will also see the original exception that was caught.
Reviewed By: DurhamG
Differential Revision: D34649748
fbshipit-source-id: 91c419660cfa8fbcf53a289071b18229f64061cf
Summary: Look for special "test_fail" and "test_succeed" URL hosts which will shortcut the network doctor to produce fail/succeed results. This will be used in integration tests to avoid real network access.
Reviewed By: DurhamG
Differential Revision: D34649751
fbshipit-source-id: 058f5d5f5ac455a1335a018fb37e21dcf46b5c49
Summary: Add better error formatting and add user visible "treatment" (i.e. tips/instructions on how to resolve the problem).
Reviewed By: DurhamG
Differential Revision: D34649752
fbshipit-source-id: 26a3ab77084ceeaac2eb6cf5ae31c01760645ad8
Summary:
taggederror wasn't used and doesn't seem like the direction we want to go in, so excise it.
taggederror allowed for error tagging and error metadata, and was wired around in various places. It tries to solve real problems, but it was too quick to prescribe how things should work without applying the design to see if it works in practice.
Reviewed By: DurhamG
Differential Revision: D34649753
fbshipit-source-id: bc60896f4e1dbe3feb7d5804561392d1f1ebda72
Summary:
Remove for a few reason:
1. It wasn't enabled anywhere.
2. It would eventually be obsolete anyway once there are no certs on disk anymore.
3. It worked by reading and checking the certs before every request, which has performance implications.
The real reason I want to remove it, though, is to clean up the corresponding edenapi error codes as I look to beef up error handling.
I will reintroduce cert checking as part of an explicit network diagnostic check that happens in response to errors.
Also kill unused check_cert py binding.
Reviewed By: DurhamG
Differential Revision: D34649747
fbshipit-source-id: 5a263e9b66712ee9ee90a4d4ffe848d05d9cb5ca
Summary: Implement std::error::Error manually so the KeyedError variant can get a source() implementation. I made it pick the first error.
Reviewed By: DurhamG
Differential Revision: D34649745
fbshipit-source-id: 4ae0c995638b6eb591d1526e6c95477f5b7e312d
Summary: "diagnose" will first check for network connectivity to corp, and if that fails will check an external URL to check for general internet connectivity.
Reviewed By: DurhamG
Differential Revision: D34649741
fbshipit-source-id: 6f914c33006118842e6fcf4aa4cc4200d65079ac
Summary: Doesn't do anything yet, but will by my entrypoint for functionally testing the network doctor logic.
Reviewed By: DurhamG
Differential Revision: D34649742
fbshipit-source-id: 0b33e5ecf68291c2d8696a5a3a9c08afb7531cfa
Summary: Start things off with a basic DNS and TCP connectivity test. This will be one of the building blocks to diagnosing network problems.
Reviewed By: DurhamG
Differential Revision: D34649750
fbshipit-source-id: 87945ec5622869eb8cb0295de14181d95a00ed8e
Summary:
This diff fixes the tests in that broke in D34801437 (85b8783db2), as tracing changed how `tracing_core::Metadata` is represented. This change in representation changed how `fmt::Debug` renders `tracing_core::Metadata` , thereby breaking the unit tests in `//eden/scm/lib/tracing-runtime-callsite` and `//eden/hg-server/lib/tracing-runtime-callsite`. Note that while we don't consider the output of `fmt::Debug` to be stable, there isn't a good mocking/unit test library for recording tracing spans/event that I'm aware of (outside of tracing's internal test framework, which we haven't gotten around to releasing—once we do, I'll happily send a diff with the updates).
Since D34801437 (85b8783db2) is 85,000 lines, I thought it'd be easier to review the fixes to broken builds in a smaller diff, such as this one.
Reviewed By: quark-zju
Differential Revision: D34941763
fbshipit-source-id: fe050d5898f5ed67d024837478d64b897e0d589e
Summary: I missed those when I worked on this for the first time
Reviewed By: farnz
Differential Revision: D34923381
fbshipit-source-id: 0e8423fda17d94c58495f83e1ce1345c69da1af7
Summary:
Lfs servers was never running listening on raw ipv6 address (rather than
hostname). This is needed for integration tests now
Reviewed By: farnz
Differential Revision: D34823679
fbshipit-source-id: 922aaca52693d8af7d719155d869ab34133bc585
