Summary: It was only useful server-side and will be incompatible with upcoming changes.
Reviewed By: DurhamG
Differential Revision: D28974549
fbshipit-source-id: 70a715ce170baa78adb8b1fcf7d29e3d1479c05e
Summary:
fncache and store have been default on for years. Enable them unconditionally.
This also makes sure that metalog is always available.
Practically, the only place that does not use fncache is hgsql server repos and
they are irrelevant now.
Reviewed By: DurhamG
Differential Revision: D28595289
fbshipit-source-id: 32b9906c179518acdb17a206b54f98a3dc994921
Summary: This makes it possible to use non-debugshell to compact the metalog.
Reviewed By: DurhamG
Differential Revision: D28550902
fbshipit-source-id: 789830ba35243d248397e6a52ee343584c1e01a9
Summary:
This makes it easier to get rid of revlog stores.
`debugindexdot` is no longer supported since it reads revlogs.
Two tests use flat manifest bundles. They are no longer supported
due to remotefilelog today has some assumptions that treemanifest
extension is also being used.
Reviewed By: DurhamG
Differential Revision: D27971126
fbshipit-source-id: fdb992a8d72bbcf562b5cb95b3f29051dd1c9464
Summary:
This command executes native checkout dry run code for updating between specificed revisions
Command tests network and storage perf, without actually affecting working copy
Examples of usages:
1) User reports that when updating between revisions multiple times, they see unexpected fetches, even though they expected that all data should already be in cache. Running this command shows that update between those revs fetches more data then a cache size, so repating fetches are not a surprise
2) People working on storage/network layer optimizations can utilize this command to test performance without messing with their working copy
This command understands nativecheckout.usescmstore config, same way it is used in actual native checkout
Differential Revision: D28040641
fbshipit-source-id: e5454f3110ade11f3227d6adc804a22176f530b9
Summary:
The zstore for commit messages are now handled by Rust entirely. There is no
need to keep the Python zstore around, except for migration and doctor
use-cases.
Reviewed By: andll
Differential Revision: D27897893
fbshipit-source-id: 21b10596af28c45425f6f102fd13f0421d1e8373
Summary:
Extend debugscmstore command to fetch arbitrary files / trees by key.
Replace debugpyscmstore with a python fallback for debugscmstore (allowing you to test with the store as it is constructed for Python, with legacy fallback).
Refactor some functionality so it is shared between the rust and python versions of debugscmstore.
Currently the output is pretty ugly. It uses the `{:#?}` format for everything. In the next change, I propose modifying the `Debug` implementation for `minibytes::Bytes` to use ascii-escaped bytestrings rather than the default slice formatter to make things much nicer.
This new `debugscmstore` functionality should be useful in integration tests for testing scmstore under different repo configurations, and for test harnesses and performance testing (fetch a specific set of things easily, simulate delays in the key stream by delaying the input pipe, etc).
Reviewed By: andll
Differential Revision: D27351321
fbshipit-source-id: 8650480e3f5b045b279472643570309c48d7fe6b
Summary:
Recently we saw some progress rendering issues. Add a command to attempt to
reproduce them.
Reviewed By: DurhamG
Differential Revision: D27669184
fbshipit-source-id: 62fcf82d8261fd27e91ba5a116c61f4df1919007
Summary: Add the `--insecure` flag to the list of global hg flags. This flag is already supported by Python, and adding it here will make it available to Rust code as well. This is used later in the stack.
Reviewed By: quark-zju
Differential Revision: D27243029
fbshipit-source-id: 150d42ee96e1e3194ff1be6a33d9b36887d86f2c
Summary:
Rename "newstore" to "scmstore"
The three main options I'm considering are "edenstore", "scmstore", and "hgstore". They all describe the project sufficiently well, in my opinion. "edenstore" might be a stretch, as I have no reals plans for Mononoke to use it, while "hgstore" might be a bit limited, as integration with EdenFS is a core goal, and it'll be used by EdenFS to fetch remote data via EdenApi, not just Mercurial's local storage. I feel like "scmstore" rolls off the tongue the easiest, particularly vs. "hgstore" (both "H.G. Store" and "Mercurial Store"), and is a bit easier to type than "edenstore" (which has "ede" all with the same finger). I don't have a strong opinion on this matter, so If you'd like a different name I'm open to suggestions.
Speak now or forever hold your peace.
Reviewed By: andll
Differential Revision: D27180332
fbshipit-source-id: 19e6972ea0f6527e671792845dcfd339cf1ab767
Summary:
Introduce newfilestore class in pyrevisionstore, which constructs a newfilestore (BoxedReadStore for files) and a corresponding `ContentStore` which share the underlying IndexedLog object and EdenApi client.
Modify remotefilelog to construct ContentStore via this new class.
Currently, no methods are provided for the newfilestore - it is meant to be passed back into Rust code, where you may call it's Rust methods as shown in the `test_newstore` method (which will be removed in the future).
Currently the `util` module is made public for access from pyrevisionstore. In the future, this will be replaced in favor of a `NewFileStoreBuilder` which handles these concerns internally.
Reviewed By: DurhamG
Differential Revision: D26526331
fbshipit-source-id: c0f439fbee4c303db4a82171c866a3f3a5fc2324
Summary:
Written by Martin von Zweigbergk at https://phab.mercurial-scm.org/D9551,
or https://www.mercurial-scm.org/repo/hg/rev/bdc2bf68f19e.
He suggested it and it's a useful feature. I did minor compatibility changes
(encoding, avoid rev numbers, foo_bar -> foobar).
Original commit message:
I use 3-way conflict markers. Often when I resolve them, I manually
compare one the base with one side and apply the differences to the
other side. That can be hard when the conflict marker is large. This
patch introduces a new type of conflict marker, which I'm hoping will
make it easier to resolve conflicts.
The new format uses `<<<<<<<` and `>>>>>>>` to open and close the
markers, just like our existing 2-way and 3-way conflict
markers. Instead of having 2 or 3 snapshots (left+right or
left+base+right), it has a sequence of diffs. A diff looks like this:
```
------- base
+++++++ left
a
-b
+c
d
```
A diff that adds one side ("diff from nothing") has a `=======` header
instead and does not have have `+` prefixed on its lines. A regular
3-way merge can be viewed as adding one side plus a diff between the
base and the other side. It thus has two ways of being represented,
depending on which side is being diffed:
```
<<<<<<<
======= left
contents
on
left
------- base
+++++++ right
contents
on
-left
+right
>>>>>>>
```
or
```
<<<<<<<
------- base
+++++++ left
contents
on
-right
+left
======= right
contents
on
right
>>>>>>>
```
I've made it so the new merge tool tries to pick a version that has
the most common lines (no difference in the example above).
I've called the new tool "mergediff" to stick to the convention of
starting with "merge" if the tool tries a regular 3-way merge.
The idea came from my pet VCS (placeholder name `jj`), which has
support for octopus merges and other ways of ending up with merges of
more than 3 versions. I wanted to be able to represent such conflicts
in the working copy and therefore thought of this format (although I
have not yet implemented it in my VCS). I then attended a meeting with
Larry McVoy, who said BitKeeper has an option (`bk smerge -g`) for
showing a similar format, which reminded me to actually attempt this
in Mercurial.
Reviewed By: DurhamG
Differential Revision: D26947920
fbshipit-source-id: 8b4446862897ff9a6dfdf5a2e35617d4db09e883
Summary: This is intended to be used in Sandcastle / OnDemand use-cases to provide a cleaner state.
Reviewed By: DurhamG
Differential Revision: D26924711
fbshipit-source-id: a37d24a10c99d953e2af842f729ef634cbb2e2f9
Summary: Introduce a minimal version of an async, typed `ReadStore` trait and corresponding `EdenApi` implementation, along with a debug command, `debugnewstore` to exercise it.
Reviewed By: DurhamG
Differential Revision: D26050641
fbshipit-source-id: 2c14c3715e7067f9ecd1e649e6ca146a1ce249bf
Summary: This would allow us to ignore bogus data and still get something useful for logging.
Reviewed By: DurhamG
Differential Revision: D20343844
fbshipit-source-id: 763d294bc44bb203c1f206ca80e0839396e8de6e
Summary:
We expect `.hg/store` to exist (ex. `storerequirements`). Drop support for repo
layout without a store.
Reviewed By: DurhamG
Differential Revision: D24340585
fbshipit-source-id: 757dc8633db9549a6bea082281326ebdefcbfcba
Summary:
This is unused code. It broke a test while I was doing python 3 work,
so let's just delete this.
Reviewed By: quark-zju
Differential Revision: D24063953
fbshipit-source-id: 323b323ff2b9717c9c16eed54adac467a321e2ae
Summary:
Add a command to rebuild the changelog without recloning other parts of the
repo. This can be used as a way to recover from corrupted changelog. It
currently uses revlog because revlog is still the only supported format during
streamclone.
In the future this can be used for defragmentation.
Reviewed By: DurhamG
Differential Revision: D23720215
fbshipit-source-id: 6db0453d18dbf553660d55d528f990a4029d9da4
Summary: It is about legacy revlog formats that are no longer relevant.
Reviewed By: DurhamG
Differential Revision: D23390436
fbshipit-source-id: 58c2c432804181bcc6517d6c988777b843fc9ba4
Summary:
extdiff uses shutil.rmtree which calls os.rmdir with new python 3
options. Since we pathc os.rmdir, we need to support those options.
Reviewed By: quark-zju
Differential Revision: D23350968
fbshipit-source-id: 081d179dcd67b51ffdeb6b85899adf4e574a8d0f
Summary:
The `debugfsync` command calls fsync on newly modified files in svfs.
Right now it only includes locations that we know have constant number
of files.
The fsync logic is put in a separate crate to avoid slow compiles.
Reviewed By: DurhamG
Differential Revision: D23124169
fbshipit-source-id: 438296002eed14db599d6ec225183bf824096940
Summary:
The `--trace` flag enables tracing Python modules.
For compatibility reasons, it also enables `--traceback`.
It can be used with debugshell to make `%trace` more useful.
Reviewed By: sfilipco
Differential Revision: D23278600
fbshipit-source-id: d6d0b34bd5c48111f8cd33d7df115f349b0e95b6
Summary:
The immediate goal is to run benchmarks on a commit graph provided by a git
repo without converting a whole (large) repo from git to hg. Note git repos can
be cloned in a shallow way so it only contains the commit graph. For example:
git clone https://github.com/torvalds/linux --filter=tree:0 -n
Note: The above command writes repositoryformat=1 in `.git/config`
which is not supported by libgit2. Manually editing it to repositoryformat=0
would enable libgit2 to read it for this crate's use-case.
In the longer term we might want to extend the support so refs/trees/files can
be read/written directly via the git repo based on this work. However that's
currently beyond scope.
Reviewed By: DurhamG
Differential Revision: D23095467
fbshipit-source-id: 868beb0c7de60453b47962639863eb8f7e3f5753
Summary: This allows migrating between a few changelog formats we have.
Reviewed By: DurhamG
Differential Revision: D22970571
fbshipit-source-id: d6b577ae5beb72a43fff999c26c35fcdc33e8f83
Summary: For now it just prints some details about the changelog backend.
Reviewed By: DurhamG, sfilipco
Differential Revision: D22970573
fbshipit-source-id: 719a5e5bb6f3856df3c9357e47daa9e7c8584952
Summary:
The `debugfsync` command calls fsync on newly modified files in svfs.
Right now it only includes locations that we know have constant number
of files.
The fsync logic is put in a separate crate to avoid slow compiles.
Reviewed By: DurhamG
Differential Revision: D22992103
fbshipit-source-id: b5503e498d5216d4ba19701ecd5582387e4f45f5
Summary:
This makes the main command table cleaner.
I dropped the `indexedlogrepair` command as it cannot rebuild indexes. `hg
doctor` is a better replacement. Some debug commands are renamed so they
no longer have `-` in the command name.
Reviewed By: DurhamG
Differential Revision: D22992107
fbshipit-source-id: f65d74e36fb971e592ad0cc8be9a94e245c39662
Summary: This change introduces a bail macro that allows tagging errors using the syntax `bail!(fault=Fault::Request, "my normal {}", bail_args)` or `bail!(Fault::Request, "my normal {}", bail_args)`.
Reviewed By: DurhamG
Differential Revision: D22646428
fbshipit-source-id: a6ec2940001b26db8ddc3a6d3620a1e17406c867
Summary:
Extend `_log_exception` to inspect exceptions for metadata, and log any that is found to Scuba. Eventually I'd like to merge at least some of this information from the `perfpipe_hg_errors` table into the `perfpipe_dev_command_timers`.
Introduce `Tagged` mixin and `Fault` enum Python types for tagging Python exceptions.
Introduce `debugthrowexception` for testing Python error handling flow.
Reviewed By: DurhamG
Differential Revision: D22585764
fbshipit-source-id: f6300ae25db25d5e75fab9c2a03fb4f9beb06c1f
Summary:
Implements based Rust-Python binding layer for error metadata propagation.
We introduce a new type, `TaggedExceptionData`, which carries CommonMetadata and the original (without metadata) error message for a Rust Anyhow error. This class is passed to RustError and can be accessed in Python (somewhat awkwardly) via indexing:
```
except error.RustError as e:
fault = e.args[0].fault()
typename = e.args[0].typename()
message = e.args[0].message()
```
As far as I can tell, due to limitations in cpython-rs, this can't be made more ergonomic without introducing a Python shim around the Rust binding layer, which could adapt the cpython-rs classes to use whatever API we'd like.
Currently, anyhow errors that are not otherwise special-cased will be converted into RustError, with both the original error message and any attached metadata printed as shown below
```
abort: intentional error for debugging with message 'intentional_error'
error has type name taggederror::IntentionalError and fault None
```
We can of course re-raise the error if desired to maintain the previous behavior for handling a RustError.
If we'd like other, specialized Rust Python Exception types to carry metadata (such as `IndexedLogError`), we'll need to modify them to accept a `TaggedExceptionData` like `RustError`.
Renamed the "cause an error in pure rust command" function to `debugcauserusterror`, and instead used the name `debugthrowrustexception` for a command which causes an error in rust which is converted to a Python exception across the binding layer.
Introduced a simple integration test which exercises `debugthrowrustexception`.
Added a basic handler for RustError to scmutil.py
Reviewed By: DurhamG
Differential Revision: D22517796
fbshipit-source-id: 0409489243fe739a26958aad48f608890eb93aa0
Summary:
Introduce new rust library, taggederror, which contains utilities for attaching metadata to errors. The library provides two main methods for attaching metadata to an error, the TaggedError wrapper type, and the AnyhowExt trait methods. Provides a struct, CommonMetadata, which contains all the metadata types introduced by taggederror (fault, transience, category, and typename), which can also be attached individually (and the same pattern can be used to attach other metadata).
Introduce a new native rust command, debugthrowrustexception, which causes the command to return an error, with some attached metadata.
Modify hg rust native command dispatch error handling to use debug formatter to print anyhow::Error errors. This will print out the source chain, contexts, and backtrace if available, which will cause the metadata we attach as a wrapper error or context to be printed.
Reviewed By: DurhamG
Differential Revision: D22420941
fbshipit-source-id: d38c5a10b686d86b69a2c0a19f5bcbf4ca24dff6
Summary: Add a new `EdenApiBlocking` trait that exposes blocking versions of the `EdenApi` trait's methods, for use in non-async code.
Reviewed By: quark-zju
Differential Revision: D22305396
fbshipit-source-id: d0f3a73cad1a23a4f0892a17f18267374e63108e
Summary:
With narrow-heads (enabled in production and most tests) directaccess is a
no-op. Remove the extension.
Reviewed By: DurhamG
Differential Revision: D22174990
fbshipit-source-id: 662f800cebc3cc7614fa921e4367c4d1f8806eb4
Summary:
Many tests want to change commit phase to public. We don't want to leak this
feature to end-users. Therefore make a debug command.
Reviewed By: kulshrax
Differential Revision: D22052481
fbshipit-source-id: 4ce6a61120da0d74374465e97ac0ec0ff9638088
Summary:
With narrow-heads (enabled in production and most tests) hiddenerror will
never happen. Remove the extension.
Reviewed By: kulshrax
Differential Revision: D22174987
fbshipit-source-id: e3e3823d3da89dfd51b60f65f8e138ea723cc00f
Summary: Not that useful and does not align with the direction we are headed.
Reviewed By: quark-zju
Differential Revision: D22213796
fbshipit-source-id: ffd86fc1a9207c134448836d0e54e48510a11135
Summary:
If we're going to iterate through the whole manifest, we should probably
prefetch it. Otherwise, we might end up doing a whole lot of sequential
fetching. We saw this this week when a change landed in sparse profiles that
caused requests to Mononoke to increase 100-fold.
Unfortunately, I don't think we can selectively only fetch the things we are
missing, so this just goes ahead and fetches everything unconditionally. If
there is a better way to do this, I'm all ears.
Reviewed By: StanislavGlebik, xavierd
Differential Revision: D22118926
fbshipit-source-id: f809fa48a7ff7b449866b42b247bf1da30097caa
Summary: We're getting rid of obsstore. The extension is no longer needed.
Reviewed By: sfilipco
Differential Revision: D21899142
fbshipit-source-id: 6e4dd2b2ddfa9325c19d6d1d9951099748260904
Summary:
Introduced a new local commit metadata storage system, smallcommitmetadata, which stores a simple local JSON file for a mapping of (node, category) -> value.
This data can be manipulated with the debugsmallcommitmetadata command, and can be consumed in the smartlog using the smallcommitmeta template func.
Total number of entries can be limited with smallcommitmetadata.entrylimit configuration option, which defaults to 100.
Reviewed By: DurhamG
Differential Revision: D21673611
fbshipit-source-id: 2239a47867118dd86b15944058505ddf67548549
Summary:
`fixcorrupt` accesses changelog in a way that is going to cause trouble with
the next change. Since `hg doctor` can do what `fixcorrupt` does in production
setup. Let's sunset fixcorrupt.
Reviewed By: singhsrb
Differential Revision: D21780575
fbshipit-source-id: 5efec3f066a2929018ccc68b1f52d10a76e59637
Summary:
Implements a "debugdetectissues" command, which runs a series of checks on the repository meant to detect potential issues, logging the data to Scuba so that we can determine how common certain issues are, and under what conditions they appear.
Future extensions of this change may involve merging the functionality into hg doctor, and setting the command to run automatically in the background on some interval.
Reviewed By: DurhamG
Differential Revision: D21558170
fbshipit-source-id: a878ae1804d5f11c83a574e0dc3c802b564d2ced
Summary:
This is helpful, when we have raw unbundle bytes and a server path and just
want to send these bytes server's way.
Very similar to `sendunbundlereplay`, but does not do anything additional,
and reads from stdin.
Reviewed By: markbt
Differential Revision: D21527243
fbshipit-source-id: 97726cb40a32c7e44f47e0f56d8c8eabc4faf209
Summary:
We want Mercurial to become more responsible for it's own
configuration, instead of relying on chef and other means. To do so, let's
introduce a new `hg debugdynamicconfig` that can generate dynamic configs for
a given repository based on various states, like what tier it's in or what shard
that machine is in. By default it generates to '.hg/hgrc.dynamic' for the given
repository.
Currently it just sets the hostgroup config.
Future diffs will make Mercurial consume this config, and possibly have Mercurial
call this command asynchronously when it notices the file is out-of-date.
Reviewed By: quark-zju
Differential Revision: D20828132
fbshipit-source-id: 6f5bf749f5b04e0a5989d6dc19ee788c2e47f88f
