All currently core parts are moved to a `bx2` namespace (for "bundle 2
experimental"). This should avoid conflicts between the final stable
format and the one about to be released.
The current implementation of bundle2 is still very experimental and the 3.0
freeze is yesterday. The current bundle2 format has never been field-tested, so
we rename the header to HG2X. This leaves the HG20 header available for real
usage as a stable format in Mercurial 3.1.
We won't guarantee that future mercurial versions will keep supporting this
`HG2X` format.
This attribute conveys the capabilities supported by the destination of the
bundle. It is used to decide which parts to include in the bundle.
This is currently a set but will probably be turned into a dictionary to allow
capabilities with values.
When a reply is built, the bundle processing will capture the output of each
handler and sends it to the client in a dedicated part.
As a side effect, this add a "remote: " prefix to destination output on local
push. This is considered okay for now as:
1. bundle2 is still experimental,
2. Matt said he could be okay to change output for bundle2,
3. This keeps the implementation simple.
This changeset does it for stdout only. stderr will be done in a future changeset.
The bundle2 processing does not create a bundle2 reply by default anymore. It
is only done if the client requests it with a `replycaps` part. This part is
called `replycaps` as it will eventually contain data about which bundle2
capabilities are supported by the client.
We have to add a flag to the test command to control whether a reply is
generated or not.
The `readbundle` function will consume the 4 first bytes to dispatch between
various unbundler. We introduce a way to inform `unbundle20` that the header
has been read and it can be trusted.
Using `readbundle` in the part handlers creates a circular import hell. We are
now using a simple `HG10UN` stream with no header. Some parameters may
later be introduced on the part to change parameter.
Producers are updated as well.
This part is intended to hold the same role as the `heads` argument of the
unbundle function. The client fill it with the known heads at bundle time and
the server will abort if its heads changed.
The `unbundle` part gains a `read` method to retrieve payload content.
This method behaves as a python file-like read method.
The bundle-processing code is updated to make sure a part is fully consumed before
another one is extracted.
Test output changes because the debug output is even more interleaved now.
We have a new unbundle class and it is now responsible from extracting its own
data. The top level bundler only extracts the header (to detect an end of stream
marker) then leaves everything else to the `unbundlepart` class. The ultimate
goal is to have `unbundlepart` responsible for lazily extracting its payload.
This is mostly code movement.
The coming `unbundlepart` will need the same kind of method than `unbundle20`
for unpacking data from the stream. We extract them into a mixin class before
the creation of `unbundlepart`.
We are going to introduce an `unbundlepart` dedicated to reading bundle. So we
need to rename the one used to create bundle. Even if dedicated to creation, this
is still used for unbundling until we get the new class.
When the `part.data` attribute is an iterator, we assume it is an iterator of
chunks and use it.
We use a chunkbuffer to yield chunks of 4096 bytes.
The tests are updated to use this feature.
We are preparing streaming capability for part. So the generation of payload
chunk will becomes more complex. We extract this part in its own function before
any changes.
We now have an official way to return the result of addchangegroup. The tests are
updated to check that the return bundle is properly created. It will be used
when push is bundle2 enabled.
We do not know yet what kind of data future features and extensions will need to
exchange. To handle that, bundle2 allows to send arbitrary content to the
server. As a consequence, we need to be able to reply arbitrary content to the
client. And, we can use bundle2 to transmit those arbitrary data.
When a client will push a bundle2 to the server, the server will reply with a
bundle2 itself.
This changeset installs the first stone of this logic and test it.
For sending response to a pushed bundle, we need to link reply parts to request
part. We introduce a part id for this purpose. This is a 32 bit unique
integer stored in the header.
We use the `gettransaction` method approach already used for pull. We
need this because we do not know beforehand if the bundle needs a
transaction to be created. And (1) we do not want to create a
transaction for nothing. (2) Some bundle2 bundles may be read-only and
do not require any lock or transaction to be held.
The current changegroup format is put in a "changegroup" part and processed by
an appropriate handlers.
This is not production ready code, but let us start smoke testing.
Part handlers can now add records to the `bundleoperation` object. This can be
used to help other parts or to let the caller of the unbundling process react
to the results.
This object will hold all data and state gathered through the processing of a
bundle. This will allow:
- each handler to be aware of the things unbundled so far
- the caller to retrieve data about the execution
- bear useful object and logic (like repo, transaction)
- bear possible bundle2 reply triggered by the unbundling.
For now the object is very simple but it will grow at the same time as the
bundle2 implementation.
The unbundle can comes from multiple sources. (on disk file, peer, etc) and
(ultimately) of multiple type (bundle10, bundle20). The `processbundle` is no
longer in charge of creating the bundle.
When the bundle processing abort on unknown mandatory parts, we now makes sure
all the bundle content is read. This avoid leaving the communication channel in
an unrecoverable state.
Mandatory part cannot be ignored when unknown. We raise a simple KeyError
exception when this happen.
This is very early version of this logic, see inline comment for future
improvement lead.
We now have a function that interpret part content.
This is a version early version of this function. It'll see major changes in
scope and API in future development. As for previous I'm just focussing on
getting minimal logic setup. Refining will happen with real world usage.
We use the same approach that the other unpack, as function is given the struct
format and his both responsible for reading the right amount of data from the
header and unpack the struct.
This give use flexibility if we decide to change the size of something in the
format before the release.
We add the ability to bundle and unbundle a payload in parts. The payload is the
actual binary data of the part. It is used to convey all the applicative data.
For now we stick to very simple implementation with all the data fit in single
chunk. This open the door to some bundle2 testing usage. This will be improved before
bundle2 get used for real. We need to be able to stream the payload in multiple
part to exchange any changegroup efficiently. This simple version will do for
now.
Bundling and unbundling are done in the same changeset because the test for
parts is less modular. However, the result is not too complex.
The bundle2 now raise value error when seeing invalid parameter names. The first
introduced rules is: no empty parameter.
The test extension is improve to properly abort when ValueError are encountered.
This introduces support for arbitrary characters in stream parameters name and
value. The urlquote format has been chosen because it is:
- simple,
- standard,
- no-op on simple alphanumerical entry.
Parameter can now have a value. We use a `<name>=<value>` form inspired from
capabilities.
There is still no kind of escaping in the name or value, yet.
Stream level parameter have very restricted use case. Clarify why we chosen a
textual format and point that applicative data goes in applicative parts.
This changeset introduce an unbundler class to match the bundle2 bundler. It is
currently able to unbundle an empty bundle2 only and will gain more feature at
the same pace than the bundler.
It also comes with its special extension command in test.
This changeset is the very first of a long series. It create a new bundle2
module and add a simple class that generate and empty bundle2 container.
The module is documented with the current state of the implementation. For
information about the final goal you may want to consult the mercurial wiki
page:
http://mercurial.selenic.com/wiki/BundleFormat2
The documentation of the module will be updated with later patches adding more and
more feature to the format.
This patches also introduce a test case. This test case build and use its own
small extension that use the new bundle2 module. Since the new format is unable
to do anything right now, we could not use real mercurial code to test it.
Moreover, some advanced feature of the bundle2 spec will not be used by core
mercurial at all. So we need to have them tested.
