The Mercurial wire protocol is under-documented. This includes a lack
of source docstrings and comments as well as pages on the official
wiki.
This patch adds the beginnings of "internals" documentation on the
wire protocol.
The documentation should have nearly complete coverage on the
lower-level parts of the protocol, such as the different transport
mechanims, how commands and arguments are sent, capabilities, and,
of course, the commands themselves.
As part of writing this documentation, I discovered a number of
deficiencies in the protocol and bugs in the implementation. I've
started sending patches for some of the issues. I hope to send a lot
more.
This patch starts with the scaffolding for a new internals page.
This patch subtly changes the behavior of the parsing of "X.Y" values
to not set the "section" variable when rendering a known sub-topic.
Previously, "section" would be the same as the sub-topic name. This
required the sub-topic RST to have a section named the same as the
sub-topic name.
When I made this change, the descriptions from help.internalstable
started being rendered in command line output. This didn't look correct
to me, as it didn't match the formatting of main help pages. I
corrected this by moving the top section to help.internalstable and
changing the section levels of all the "internals" topics.
The end result is that "internals" topics now match the rendering of
main topics on both the CLI and HTML. And, "internals" topics no longer
require a main section matching the name of the topic.
Now that we have a mechanism for declaring path sub-options, we can
start to pile on features!
Many power users have expressed frustration that bare `hg push`
attempts to push all local revisions to the remote. This patch
introduces the "pushrev" path sub-option to control which revisions
are pushed when no "-r" argument is specified.
The value of this sub-option is a revset, naturally.
A future feature addition could potentially introduce a "pushnames"
sub-options that declares the list of names (branches, bookmarks,
topics, etc) to push by default. The entire "what to push by default"
feature should probably be considered before this patch lands.
Before this patch, "hg help topic.section" might show unexpected
section of help topic in some encoding.
It applies str.lower() instead of encoding.lower(str) on translated
message to search section case-insensitively, but some encoding uses
0x41(A) - 0x5a(Z) as the second or later byte of multi-byte character
(for example, ja_JP.cp932), and str.lower() causes unexpected result.
To search section of help topic by translated section name correctly,
this patch replaces str.lower() by encoding.lower(str) for both query
string (in commands.help()) and translated help text (in
minirst.getsections()).
Before this patch, when printing help text using `hg help`, or `hg log -h`,
the output will wrap at 78 chars even if the user has a bigger terminal width
and there is no config option to change it, making the experience different
from the commonly used `man` tool.
This patch introduces a new config option `ui.textwidth`, which replaces the
hardcoded number. It's set to 78 by default to maintain compatibility. When
set to 0, `hg help` will behave more like `man`.
Before this commit --force option help description stated
that file was removed and deleted even if file was added
or modified which is not true. Force option removes added
file only from dirstate, it doesn't delete it from the
filesystem.
The clone bundles feature was introduced in Mercurial 3.6 behind an
experimental and disabled by default flag. The feature has been enabled
on hg.mozilla.org for a few months and has served many terabytes of
clones. Users have been encouraged to use the feature and reception
has been very positive (mainly due to faster clones as a result of
connecting to a CDN). I have heard no feedback about changing the
feature other than inquiries about when it will be enabled by default.
So, I think the feature is ready to be enabled by default.
This patch renames experimental.clonebundles to ui.clonebundles,
documents the option, and enables it by default. References to the
experimental state of clone bundles have been removed. The remaining
config option docs in clonebundles.py have been removed because they
are redudant with `hg help config`.
There are some oddities with behavior of clone bundles. Because clones
with clone bundles are effectively 2 `hg pull` operations, there may be
2 transactions. This could result in hooks running twice. If the
subsequent pull is aborted, it could result in partial rollback and an
incomplete clone. This behavior is a bit wonky and should probably
be documented. If this patch is accepted, I'll send a follow-up to
document it. I don't think this behavior should prevent the feature
being enabled by default. Reworking the clone mechanism to support
interrupted or multi-part clones feels like a major new feature and
something that when implemented can change the hook and rollback
semantics of clone bundles. Besides, partial clone is better than
full rollback and hooks running on initial clone are likely rare, so I
think the impact is minimal.
Before this patch debugignore was just displaying the list of ignore patterns.
This patch makes it support a list of filename as argument and tells the user
if those given files are ignored or not.
It seems like a good idea to document the revlog format.
There is a lot more that could be added to this documentation.
But you have to start somewhere.
We now have sub-topics in the help system. The "helptopics" template
serves as a mechanism for displaying an index of help topics.
Previously, it was only used to show the top-level list of help topics,
which includes special groupings of topics.
In the near future, we'll adapt "helptopics" for showing the index
of sub-topics. In this patch, we optionally render {earlycommands} and
{othercommands} since they aren't present on sub-topics.
If a sub-topic/section is requested and the main topic corresponds to
a topic with sub-topics, we now look for and return content for a
sub-topic if found.
With this patch, `hg help internals.X` now works. hgweb does not yet
render sub-topics, however.
We introduce the "internals" help topic, which renders an index of
available sub-topics. The sub-topics themselves are still not
reachable via the help system.
Before, hg help -c was the same as hg help, now it only shows commands.
Before, hg help -e was the same as hg help, now it only shows extensions.
Before, hg help -k crashed, now it shows all topics.
Power users often want to apply per-path configuration options. For
example, they may want to declare an alternate URL for push operations
or declare a revset of revisions to push when `hg push` is used
(as opposed to attempting to push all revisions by default).
This patch establishes the use of sub-options (config options with
":" in the name) to declare additional behavior for paths.
New sub-options are declared by using the new ``@ui.pathsuboption``
decorator. This decorator serves multiple purposes:
* Declaring which sub-options are registered
* Declaring how a sub-option maps to an attribute on ``path``
instances (this is needed to `hg paths` can render sub-options
and values properly)
* Validation and normalization of config options to attribute
values
* Allows extensions to declare new sub-options without monkeypatching
* Allows extensions to overwrite built-in behavior for sub-option
handling
As convenient as the new option registration decorator is, extensions
(and even core functionality) may still need an additional hook point
to perform finalization of path instances. For example, they may wish
to validate that multiple options/attributes aren't conflicting with
each other. This hook point could be added later, if needed.
To prove this new functionality works, we implement the "pushurl"
path sub-option. This option declares the URL that `hg push` should
use by default.
We require that "pushurl" is an actual URL. This requirement might be
controversial and could be dropped if there is opposition. However,
objectors should read the complicated code in ui.path.__init__ and
commands.push for resolving non-URL values before making a judgement.
We also don't allow #fragment in the URLs. I intend to introduce a
":pushrev" (or similar) option to define a revset to control which
revisions are pushed when "-r <rev>" isn't passed into `hg push`.
This is much more powerful than #fragment and I don't think #fragment
is useful enough to continue supporting.
The [paths] section of the "config" help page has been updated
significantly. `hg paths` has been taught to display path sub-options.
The docs mention that "default-push" is now deprecated. However, there
are several references to it that need to be cleaned up. A large part
of this is converting more consumers to the new paths API. This will
happen naturally as more path sub-options are added and more and more
components need to access them.
We have debug commands for displaying overall revlog statistics
(debugrevlog) and for dumping a revlog index (debugindex). As part
of investigating various aspects of revlog behavior and performance,
I found it important to have an understanding of how revlog
delta chains behave in practice.
This patch implements a "debugdeltachain" command. For each revision
in a revlog, it dumps information about the delta chain. Which delta
chain it is part of, length of the delta chain, distance since base
revision, info about base revision, size of the delta chain, etc. The
generic formatting facility is used, which means we can templatize
output and get machine readable output like JSON.
This command has already uncovered some weird history in
mozilla-central I didn't know about. So I think it's valuable.
The next patch will remove the progress extension completely, so we have
to pick another extension. The schemes is picked arbitrary.
This test was introduced at 57703c45ed60.
There are a lot of considerations server operators need to know before
deploying clone bundles. They should be documented. So I rewrote the
extension docs to contain this information.
For the same reasons that we don't produce stream clone bundles with `hg
bundle`, we don't support consuming stream clone bundles with `hg
unbundle`. We introduce a complementary debug command for applying
stream clone bundles. This command is mostly to facilitate testing.
Although it may be used to manually apply stream clone bundles until a
more formal mechanism is (possibly) adopted.
Now that we have support for recognizing the streaming clone bundle
type, add a debug command for creating them.
I decided to create a new debug command instead of adding support to `hg
bundle` because stream clone bundles are not exactly used the same way
as normal bundle files and I don't want to commit to supporting them
through the official `hg bundle` command forever. A debug command,
however, can be changed without as much concern for backwards
compatibility.
As part of this, `hg bundle` will explicitly reject requests to produce
stream bundles.
This command will be required by server operators using stream clone
bundles with the clone bundles feature.
Cloning can be an expensive operation for servers because the server
generates a bundle from existing repository data at request time. For
a large repository like mozilla-central, this consumes 4+ minutes
of CPU time on the server. It also results in significant network
utilization. Multiplied by hundreds or even thousands of clients and
the ensuing load can result in difficulties scaling the Mercurial server.
Despite generation of bundles being deterministic until the next
changeset is added, the generation of bundles to service a clone request
is not cached. Each clone thus performs redundant work. This is
wasteful.
This patch introduces the "clonebundles" extension and related
client-side functionality to help alleviate this deficiency. The
client-side feature is behind an experimental flag and is not enabled by
default.
It works as follows:
1) Server operator generates a bundle and makes it available on a
server (likely HTTP).
2) Server operator defines the URL of a bundle file in a
.hg/clonebundles.manifest file.
3) Client `hg clone`ing sees the server is advertising bundle URLs.
4) Client fetches and applies the advertised bundle.
5) Client performs equivalent of `hg pull` to fetch changes made since
the bundle was created.
Essentially, the server performs the expensive work of generating a
bundle once and all subsequent clones fetch a static file from
somewhere. Scaling static file serving is a much more manageable
problem than scaling a Python application like Mercurial. Assuming your
repository grows less than 1% per day, the end result is 99+% of CPU
and network load from clones is eliminated, allowing Mercurial servers
to scale more easily. Serving static files also means data can be
transferred to clients as fast as they can consume it, rather than as
fast as servers can generate it. This makes clones faster.
Mozilla has implemented similar functionality of this patch on
hg.mozilla.org using a custom extension. We are hosting bundle files in
Amazon S3 and CloudFront (a CDN) and have successfully offloaded
>1 TB/day in data transfer from hg.mozilla.org, freeing up significant
bandwidth and CPU resources. The positive impact has been stellar and
I believe it has proved its value to be included in Mercurial core. I
feel it is important for the client-side support to be enabled in core
by default because it means that clients will get faster, more reliable
clones and will enable server operators to reduce load without
requiring any client-side configuration changes (assuming clients are
up to date, of course).
The scope of this feature is narrowly and specifically tailored to
cloning, despite "serve pulls from pre-generated bundles" being a valid
and useful feature. I would eventually like for Mercurial servers to
support transferring *all* repository data via statically hosted files.
You could imagine a server that siphons all pushed data to bundle files
and instructs clients to apply a stream of bundles to reconstruct all
repository data. This feature, while useful and powerful, is
significantly more work to implement because it requires the server
component have awareness of discovery and a mapping of which changesets
are in which files. Full, clone bundles, by contrast, are much simpler.
The wire protocol command is named "clonebundles" instead of something
more generic like "staticbundles" to leave the door open for a new, more
powerful and more generic server-side component with minimal backwards
compatibility implications. The name "bundleclone" is used by Mozilla's
extension and would cause problems since there are subtle differences
in Mozilla's extension.
Mozilla's experience with this idea has taught us that some form of
"content negotiation" is required. Not all clients will support all
bundle formats or even URLs (advanced TLS requirements, etc). To ensure
the highest uptake possible, a server needs to advertise multiple
versions of bundles and clients need to be able to choose the most
appropriate from that list one. The "attributes" in each
server-advertised entry facilitate this filtering and sorting. Their
use will become apparent in subsequent patches.
Initial inspiration and credit for the idea of cloning from static files
belongs to Augie Fackler and his "lookaside clone" extension proof of
concept.
