We now take full advantage of the 'getunbundler' function by using a
'{version -> unbundler-class}' mapping. This map currently contains a single
entry but will make it easy to support more versions from an extension/the
future.
At some point, this map will probably contain bundler-class information too,
in the same fashion the packer map does. However, this is not critically required
right now so it will happen by itself when needed.
The main target is to allow HG2Y support in an extension to ease transition of
companies using the experimental protocol in production (yeah...) But I've no
doubt this will be useful when playing with a future HG21.
To support multiple bundle2 formats, we will need a function returning
the proper unbundler according to the header. We introduce such aa
function and change the usage in the code base. The function will get
smarter in later changesets.
This is somewhat similar to the dispatching we do for 'HG10' and 'HG11'.
The main target is to allow HG2Y support in an extension to ease transition of
companies using the experimental protocol in production (yeah...) But I've no
doubt this will be useful when playing with a future HG21.
This makes it easy to create a new bundler class that inherits from
the core one. This matches the way 'changegroup' packers work.
The main target is to allow HG2Y support in an extension to ease transition of
companies using the experimental protocol in production (yeah...) But I've no
doubt this will be useful when playing with a future HG21.
Bundlerepo uses the compressed() method to determine whether it should write
an uncompressed temporary file. Since we don't support compressed bundle2 files
at the moment, make this method return true.
Replace bare part.read() calls with part.seek(0, 2) since the return value is
being ignored. As this doesn't necessarily require building a string that
contains the rest of the part, the potential exists to reduce the memory
footprint of these operations.
This implements a seek() method for unbundlepart. This allows on-disk bundle2
parts to behave enough like files for bundlerepo to handle them. A future
patch will add support for bundlerepo to read the bundle2 files that are
written when the experimental.strip-bundle2-version config option is used.
This patch adds seek(), tell(), and close() implementations for unpackermixin
which forward to the file descriptor's implementation if possible. A future
patch will use this to make bundle2.unbundlepart seekable, which will in turn
make it usable as a file descriptor for bundlerepo.
The binary format description has always stated that the parttype should be simple,
but it was never really enforced. Recent discussions have convinced me we want to
keep the part type simple and easy to debug. There is enough extensibility in
the rest of the format.
Encoding whether or not a part is mandatory in the capitalization of the
parttype is unintuitive and error-prone. This sequence of patches separates
these concerns in the API to reduce programmer error and pave the way for
a potential change in how this information is transmitted over the wire.
Since the parttype and mandatory bit are separated in bundle2.unbundlepart
(see previous patch), there is no longer a need to remove the mandatory bit
before working with the parttype.
Encoding whether or not a part is mandatory in the capitalization of the
parttype is unintuitive and error-prone. This sequence of patches separates
these concerns in the API to reduce programmer error and pave the way for
a potential change in how this information is transmitted over the wire.
This patch separates the two pieces of information when reading the part header
so that it's unnecessary to know how they were combined during transmission.
This patch series is intended to allow bundle2 push reply part handlers to
make changes to the local repository; it has been developed in parallel with
an extension that allows the server to rebase incoming changesets while applying
them.
This diff adds an experimental config option "bundle2.pushback" which provides
a transaction to the reply unbundler during a push operation. This behavior is
opt-in because of potential security issues: the response can contain any part
type that has a handler defined, allowing the server to make arbitrary changes
to the local repository.
This patch series is intended to allow bundle2 push reply part handlers to
make changes to the local repository; it has been developed in parallel with
an extension that allows the server to rebase incoming changesets while applying
them.
The default transaction getter for processbundle is a private function that
raises an exception; this diff lets calling code pass None as the transaction
getter to explicitly request this default behavior.
The next diff will check a config option to determine whether to provide a
transaction to the reply bundle processor. If one shouldn't be provided, the
code needs a way to specify that the default behavior should be used.
This will let the bundle2 client and server detect what packer they should be using.
This detection part is not done. I expect it to be done with the addition of the
second packer (with generaldelta support).
If an exception is raised during a bundle2 part payload generation it is now
recorded in the bundle. If such exception occurs, we capture it, transmit an
abort exception through the bundle, cleanly close the current part payload and
raise it again. This allow to generate valid bundle even in case of exception so
that the consumer does not wait forever for a dead producer. This also allow to
raise the exception during unbundling at the exact point it happened during
bundling make debugging easier.
It is now possible to emit a single part in the middle of a payload production.
This part will be processed with limitation (only access to a `ui` object). The
goal is to let the server raise exception and output while a part is being
processed. The source motivation is to transmit exception that occurs while
generating a part.
This change is was the motivation to bump the bundle2 format from HG2X to HG2Y.
Somehow, the format bump made it into 3.2 without it. So this change go on
stable. It is low risk as bundle2 is still disabled by default.
Bundle2 opens doors to advanced features allowing to reduce load on
mercurial servers, and improve clone experience for users on unstable or
slow networks.
For instance, it could be possible to pre-generate a bundle of a
repository, and give a pointer to it to clients cloning the repository,
followed by another changegroup with the remainder. For significantly
big repositories, this could come as several base bundles with e.g. 10k
changesets, which, combined with checkpoints (not part of this change),
would prevent users with flaky networks from starting over any time
their connection fails.
While the server-side support for those features doesn't exist yet, it
is preferable to have client-side support for this early-on, allowing
experiments on servers only requiring a vanilla client with bundle2
enabled.
We are changing all integers that denote the size of a chunk to read to int32.
There are two main motivations for that.
First, we change everything to the same width (32 bits) to make it possible for
a reasonably agnostic actor to forward a bundle2 without any extra processing.
With this change, this could be achieved by just reading int32s and forwarding
chunks of the size read. A bit a smartness would be logic to detect the end of
stream but nothing too complicated.
Second, we need some capacity to transmit special information during the bundle
processing. For example we would like to be able to raise an exception while a
part is being read if this exception happend while this part was generated.
Having signed integer let us use negative numbers to trigger special events
during the parsing of the bundle.
The format is renamed for B2X to B2Y because this breaks binary
compatibility. The B2X format support is dropped. It was experimental to
allow this kind of things. All elements not directly related to the binary
format remain flagged "b2x" because they are still compatible.
This is code movement only. This will be useful to have it separated for reuse
purposes. We plan to introduce a new feature to the bundle format that allow
inserting a part in the middle of another part payload. This will be useful to
transmit a exception raised during a part generation.
The final writing of the empty part was done explicitly. We now using proper
pack call using symbolic constant. This open simple change in the bundle2
format.
Functions like getbundle and classes like unbundle10 really manipulate
changegroups and not bundles. A HG10 bundle is the same as a changegroup
plus a small header, but this is no longer the case for a HG2X bundle,
so it's better to separate the names a bit.
Right next to the function that encodes the supported versions in
capabilities we add a function that decodes the versions out of capabilities.
This is going to be useful to know what formats can be used for exchange.
This property can be used to know how much parts have been added to the bundle2.
This will be useful to check if any part have been generated for a push.
After ``listkeys`` we can now include ``pushkey`` request in a bundle2. The part
uses a very simple scheme closest as possible to the current wireproto command
for ``pushkey``. We may eventually decide for a more sophisticated part format
before the protocol becomes final.
The process of decoding remote bundle2caps blob into a dictionary is cumbersome.
We move it into a small helper function. This will clarify code that reads
bundle2 capabilities of peers and helps using it in new places.
