Clone bundles require a well-defined string to specify the type of
bundle that is listed so clients can filter compatible file types. The
`hg bundle` command and cmdutil.parsebundletype() already establish the
beginnings of a bundle specification format.
As part of formalizing this format specification so it can be used by
clone bundles, we move the specification parsing bits verbatim to
exchange.py, which is a more suitable place than cmdutil.py. A
subsequent patch will refactor this code to make it more appropriate as
a general API.
Cloning can be an expensive operation for servers because the server
generates a bundle from existing repository data at request time. For
a large repository like mozilla-central, this consumes 4+ minutes
of CPU time on the server. It also results in significant network
utilization. Multiplied by hundreds or even thousands of clients and
the ensuing load can result in difficulties scaling the Mercurial server.
Despite generation of bundles being deterministic until the next
changeset is added, the generation of bundles to service a clone request
is not cached. Each clone thus performs redundant work. This is
wasteful.
This patch introduces the "clonebundles" extension and related
client-side functionality to help alleviate this deficiency. The
client-side feature is behind an experimental flag and is not enabled by
default.
It works as follows:
1) Server operator generates a bundle and makes it available on a
server (likely HTTP).
2) Server operator defines the URL of a bundle file in a
.hg/clonebundles.manifest file.
3) Client `hg clone`ing sees the server is advertising bundle URLs.
4) Client fetches and applies the advertised bundle.
5) Client performs equivalent of `hg pull` to fetch changes made since
the bundle was created.
Essentially, the server performs the expensive work of generating a
bundle once and all subsequent clones fetch a static file from
somewhere. Scaling static file serving is a much more manageable
problem than scaling a Python application like Mercurial. Assuming your
repository grows less than 1% per day, the end result is 99+% of CPU
and network load from clones is eliminated, allowing Mercurial servers
to scale more easily. Serving static files also means data can be
transferred to clients as fast as they can consume it, rather than as
fast as servers can generate it. This makes clones faster.
Mozilla has implemented similar functionality of this patch on
hg.mozilla.org using a custom extension. We are hosting bundle files in
Amazon S3 and CloudFront (a CDN) and have successfully offloaded
>1 TB/day in data transfer from hg.mozilla.org, freeing up significant
bandwidth and CPU resources. The positive impact has been stellar and
I believe it has proved its value to be included in Mercurial core. I
feel it is important for the client-side support to be enabled in core
by default because it means that clients will get faster, more reliable
clones and will enable server operators to reduce load without
requiring any client-side configuration changes (assuming clients are
up to date, of course).
The scope of this feature is narrowly and specifically tailored to
cloning, despite "serve pulls from pre-generated bundles" being a valid
and useful feature. I would eventually like for Mercurial servers to
support transferring *all* repository data via statically hosted files.
You could imagine a server that siphons all pushed data to bundle files
and instructs clients to apply a stream of bundles to reconstruct all
repository data. This feature, while useful and powerful, is
significantly more work to implement because it requires the server
component have awareness of discovery and a mapping of which changesets
are in which files. Full, clone bundles, by contrast, are much simpler.
The wire protocol command is named "clonebundles" instead of something
more generic like "staticbundles" to leave the door open for a new, more
powerful and more generic server-side component with minimal backwards
compatibility implications. The name "bundleclone" is used by Mozilla's
extension and would cause problems since there are subtle differences
in Mozilla's extension.
Mozilla's experience with this idea has taught us that some form of
"content negotiation" is required. Not all clients will support all
bundle formats or even URLs (advanced TLS requirements, etc). To ensure
the highest uptake possible, a server needs to advertise multiple
versions of bundles and clients need to be able to choose the most
appropriate from that list one. The "attributes" in each
server-advertised entry facilitate this filtering and sorting. Their
use will become apparent in subsequent patches.
Initial inspiration and credit for the idea of cloning from static files
belongs to Augie Fackler and his "lookaside clone" extension proof of
concept.
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
In the external pushrebase extension, it is valuable to be able to do some work
without taking the lock (like running expensive hooks). This enables
significantly higher commit throughput.
This patch adds an option to lazily acquire the lock. It means that all bundle2
part handlers that require writing to the repo must first call
op.gettransction(), when in this mode.
This is the beginning of client-side support for performing a stream
clone using bundle2. The main bundle2 pull function checks whether to
perform a streaming clone and outputs a message if so.
While we have a duplicate message, it seems easier to have all the
bundle2 console writing in one location and in an easy-to-read
conditional block.
This adds a cache and makes accessing the capabilities slightly simpler,
as you don't need to directly go through the bundle2 module. This will
also help prevent a function-level import in streamclone.py.
This patch arguably isn't necessary. But I think it makes things
slightly nicer.
Upcoming patches will introduce bundle2 based streaming clones. Add
"legacy" to the function name and add a docstring clarifying the intent of
the function.
Just like all the other pull steps. Consistency is good.
This seems a little excessive right now since maybeperformstreamclone is
such a short function. This will be addressed in a subsequent patch.
Stream clones are a special case of clones. Clones are a special case of
pull. Most of the logic for deciding what to do at pull time is in
exchange.py. It makes sense for the stream clone determination to live
there as well.
This patch moves the calling of the stream clone code into pull(). The
checks in streamclone.canperformstreamclone() ensure that we don't
perform a stream clone unless it is possible.
A future patch will convert maybeperformstreamclone() to accept a
pullop to make it consistent with everything else in pull(). It will
also grow some functionality (in case you doubted the necessity of a 4
line function).
An upcoming patch will move the invocation of stream cloning logic to
the normal pull code path (from localrepository.clone). In preparation
for this, we teach pull() and pulloperation about whether a streaming
clone is requested.
The return logic in localrepository.clone() has been reformatted
slightly because of line length issues.
We bulk move functions from exchange.py related to streaming clones.
Function names were renamed slightly to drop a component redundant with
the module name. Docstrings and comments referencing old names and
locations were updated accordingly.
The common ancestor set implementation was made lazy a couple years ago, but
this piece of code still required processing the entire repo by putting set()
around the lazy set. The code was introduced in 984b6b21bf13, a year before the
lazy ancestor set was added.
Dropping the set() shaves 3.5 seconds off of 'push -r' in repos with hundreds of
thousands of commits.
The assignment of the value from bundle2.processbundle() to 'r' is
unused. It is currently the same as its third argument (if given), and
since that argument may eventually go away (according to the method's
docstring), let's reassign the return value to 'op' instead to better
prepare for that.
Python 2.6 introduced the "except type as instance" syntax, replacing
the "except type, instance" syntax that came before. Python 3 dropped
support for the latter syntax. Since we no longer support Python 2.4 or
2.5, we have no need to continue supporting the "except type, instance".
This patch mass rewrites the exception syntax to be Python 2.6+ and
Python 3 compatible.
This patch was produced by running `2to3 -f except -w -n .`.
The 'getchangegroupraw' is very simple (two lines) so we inline it in its only
caller. This exposes the 'outgoing' object of the part generator function, allowing
us to add information on the number of changesets contained in the part in a
later changeset. Such information is useful for progress bar.
When using bundle2, the phase pushkey parts are now made mandatory. As a
result, failure to update the bookmark server side will result in the transaction
being aborted.
When using bundle2, the bookmark's pushkey parts are now made mandatory. As a
result failure to update the bookmark server side will result in the transaction
being aborted.
We add a way to register "pushkey failure callback" that will be used if the
push is aborted by a pushkey failure. A part generator adding mandatory pushkey
parts should register a failure callback for all of them. The callback will be
in charge of generating a meaningful abort if this part fails.
If no callback is registered, the error is propagated.
Catch PushkeyFailed error in exchange.
The current behavior (with bundle1) is to let the rest of the push succeed if
the pushkey call (phases, bookmarks) failed (this comes from the fact that each
item is sent in its own command).
We kept this behavior with bundle2, which is highly debatable, but let us keep
thing as they are now as a start. We are about to enforce 'mandatory' pushkey
part as 'mandatory' successful, so we need to marks parts as advisory to
preserve the current (debatable) behavior.
All known server implementations have listkeys support with bundle2, but people
in the process of implementing new servers may not. Let's be nice with them.
We are already fetching remote bookmarks to honor the -B option, we
now pass that data to the pull process so it can reuse it. This
prevents a race condition between the initial looking and the actual
pulling of changesets and bookmarks. Tests are updated to handle this
fact.
We have been feeling the need for this in extensions for quite some time. This
will be used to pass remote bookmark information around in the next changesets.
For efficiency and consistency purpose, remote bookmarks, retrieved at the time
the pull command code is doing lookup, will be reused during the core pull
operation.
A second step toward this is to avoid requesting bookmark information in
the bundle 2 if we already have them locally.
For efficiency and consistency purpose, remote bookmarks, retrieved at the time
the pull command code is doing lookup, will be reused during the core pull
operation.
A first step toward this is to setup the logic avoiding pulling the data again
during the discovery phase if some have already been provided.
All the test change have been isolated and validated. We have free to turn on
bundle2 as the default exchange protocol.
"To reach a port we must set sail –
Sail, not tie at anchor
Sail, not drift."
On Mozilla's mozilla-beta repository .hgtags fnodes resolution takes
~18s from a clean cache on my machine. This means that the first time
a user runs `hg tags`, `hg log`, or any other command that displays or
accesses tags data, a ~18s pause will occur. There is no output during
this pause. This results in a poor user experience and perception
that Mercurial is slow.
The .hgtags changeset to filenode mapping is deterministic. This
patch takes advantage of that property by implementing support
for transferring .hgtags filenodes mappings in a dedicated bundle2
part. When a client advertising support for the "hgtagsfnodes"
capability requests a bundle, a mapping of changesets to .hgtags
filenodes will be sent to the client.
Only mappings of head changesets included in the bundle will be sent. The
transfer of this mapping effectively eliminates one time tags cache related
pauses after initial clone.
The mappings are sent as binary data. So, 40 bytes per pair of
SHA-1s. On the aforementioned mozilla-beta repository,
659 * 40 = 26,360 raw bytes of mappings are sent over the wire
(in addition to the bundle part headers). Assuming 18s to populate
the cache, we only need to transfer this extra data faster than
1.5 KB/s for overall clone + tags cache population time to be shorter.
Put into perspective, the mozilla-beta repository is ~1 GB in size.
So, this additional data constitutes <0.01% of the cloned data.
The marginal overhead for a multi-second performance win on clones
in my opinion justifies an on-by-default behavior.
All bundle2 servers now support the 'listkeys' part(1), so we'll
always be able to fetch bookmarks data at the same time as the
changeset. This should be enough to avoid the one race condition that
this bookmark prefetching is trying to work around. It even allows
future server to make sure everything is generated from the same
"transaction" if they become capable of such. The current code was
already overwriting the prefetched value with the one in bundle2
anyway. Note that this is not preventing all race conditions in
related to bookmark in 'hg pull' it makes nothing better and nothing
worse.
Reducing the number of listkeys calls will reduce the latency on pull.
The pre-fetch is also moved into a discovery step because it seems to belong
there.
(1) Because all servers not speaking 'pushkey' parts are compatible with the
'HG2X' protocol only.
We are doing some strange special casing of phase push when:
- the source is a subrepo
- the destination is publishing
- some changeset are still draft on the destination
In that case we do not push phases information (to publish the draft changesets)
because it could break simple cycle of 'clone/pull/push' of subrepos. We have to
detect this case earlier to have bundle2 respecting it.
We change the test to check the behavior for both bundle1 and bundle2.
For reasons outlined in the previous commit, we want to make the code
for consuming "stream bundles" reusable. This patch extracts the code
into a standalone function.
Streaming clones are fast because they are essentially tar files.
On mozilla-central, a streaming clone only consumes ~55s CPU time
on clients as opposed to ~340s CPU time for a regular clone or gzip
bundle unbundle.
Mozilla is deploying static file "lookaside" support to our Mercurial
server. Static bundles are pre-generated and uploaded to S3. When a
clone is performed, the static file is fetched, applied, and then an
incremental pull is performed. Unfortunately, on an ideal network
connection this still takes as much wall and CPU time as a regular
clone (although it does save significant server resources).
We like the client-side wall time wins of streaming clones. But we want
to leverage S3-based pre-generated files for serving the bulk of clone
data.
This patch moves the code for producing a "stream bundle" into its
own standalone function, away from the wire protocol. This will enable
stream bundle files to be produced outside the context of the wire
protocol.
A bikeshed on whether exchange is the best module for this function
might be warranted. I selected exchange instead of changegroup because
"stream bundles" aren't changegroups (yet).
I just discovered that we are not displaying ssh server output in real time
anymore. So we can just fall back to the bundle2 output capture for now. This
fix the race condition issue we where seeing in tests. Re-instating real time
output for ssh would fix the issue too but lets get the test to pass first.
The current bundle2 processing was capturing all output. This is nice as it
provide better meta data about what output what, but this was changing two
things:
1) adding a prefix "remote: " to "other" output during local push (issue4613)
2) local and ssh push does not provide real time output anymore (issue4615)
As we are unsure about what form should be used in (1) and how to solve (2) we
disable output capture in this two cases. Output capture can be forced using an
experimental option.
Because bundle2 allows a more precise exchange of obsmarkers during pull, it
sends them in a different order (previously unstable because of sets.) As
a result, they are added to the repository in a different order. To stabilize
the order and ensure tests are unchanged when moving from bundle1 to bundle2 we
sort markers when exchanging them.
In the long run, the obsstore will probably not use a linear storage.
Until this changeset, we were only able to save output if an error happened
during the 'transaction.close()' phase. If the 'processbundle' call raised an
exception, the 'bundleoperation' object was never returned, so the reply bundle
was never accessible and no output could be salvaged. We introduce a quick (but
not very elegant) fix to gain access to any reply created during the processing.
This conclude this output related series. We should hopefully be able client-side to see the
whole server output, in a proper order.
The code is now complex enough that a refactoring of it would make sense on
default.
We were capturing all output issue during bundle2 processing, and all output
issue during transaction rollback in case of failure. However, the output issue
during transaction commit was still roaming the land freely. It is now put back
in line.
This let the user see output from 'pretxnclose' and 'txnclose' (and related) in
the right order.
External hook used to directly write on stdout and stderr. As a result their
output was not captured by the bundle2 processing. This resulted in confusing
out of order output on the client side. We are now capturing hooks output in
this context.
