I just discovered that we are not displaying ssh server output in real time
anymore. So we can just fall back to the bundle2 output capture for now. This
fix the race condition issue we where seeing in tests. Re-instating real time
output for ssh would fix the issue too but lets get the test to pass first.
The current bundle2 processing was capturing all output. This is nice as it
provide better meta data about what output what, but this was changing two
things:
1) adding a prefix "remote: " to "other" output during local push (issue4613)
2) local and ssh push does not provide real time output anymore (issue4615)
As we are unsure about what form should be used in (1) and how to solve (2) we
disable output capture in this two cases. Output capture can be forced using an
experimental option.
Until this changeset, we were only able to save output if an error happened
during the 'transaction.close()' phase. If the 'processbundle' call raised an
exception, the 'bundleoperation' object was never returned, so the reply bundle
was never accessible and no output could be salvaged. We introduce a quick (but
not very elegant) fix to gain access to any reply created during the processing.
This conclude this output related series. We should hopefully be able client-side to see the
whole server output, in a proper order.
The code is now complex enough that a refactoring of it would make sense on
default.
We were capturing all output issue during bundle2 processing, and all output
issue during transaction rollback in case of failure. However, the output issue
during transaction commit was still roaming the land freely. It is now put back
in line.
This let the user see output from 'pretxnclose' and 'txnclose' (and related) in
the right order.
External hook used to directly write on stdout and stderr. As a result their
output was not captured by the bundle2 processing. This resulted in confusing
out of order output on the client side. We are now capturing hooks output in
this context.
The output from the transaction rollback was not included into the reply bundle.
It was eventually caught by the usual 'unbundle' output capture and sent to the
client but the result was out of order on the client side. We now capture the
output for the transaction release and transmit it the same way as all other
output.
We should probably rethink the whole output capture things but this would not be
appropriate for stable.
The is still multiple cases were output failed to be properly capture, they will
be fixed in later changesets.
Since transaction are used for more than just changesets, it is possible
to have a transaction without new changesets at all. In this case no
''00changelog.i.a' are written. In all cases the 'changelog.readpending'
method is called if the repository has any pending data. The 'revlog' logic
provides empty content if the file is missing, so the whole operation
resulted in an empty changelog.
We now skip reading the pending file if it is missing.
In case of errors, output parts salvaged from the reply bundle need to be
processed for outputting their content. This concludes our quest for fixing
issue4594.
In case of errors, output parts salvaged from the reply bundle are re-injected
into the bundle carrying the exception.
We still need to fix the situation for non-wireprotocol push.
A bundle2 may contain bookmark updates (or other extension content) that
requires the 'wlock' to be written. As 'wlock' must be acquired before 'lock',
we must stay on the side of caution and use both in all case to ensure their
ordering.
This should help us to catch new locking order issues as soon as possible.
There are two harmless test updates (from the config change). Moreover, some
bundle2 tests are displaying warning for a legitimate reason. The use of pushkey
during the unbundle process may requires the 'wlock' after 'lock' (around the
whole unbundle process was taken). This is non-trivial to fix, so I better have
the check on, with the warning in the test than the check off. See issue4596 for
details.
It is finally time to freeze the bundle2 format! To do so we:
- rename HG2Y to HG20,
- drop "b2x:" prefix from all part names,
- rename capability to "bundle2-exp" to "bundle2"
- rename the hook flag from 'bundle2-exp' to 'bundle2'
The pushkey operation used to be in its own wireprotocol command and (in
practice) always be lock free when running the hook. With bundle2, it happen in
a greater scheme and a hook running locking command would get stuck. We now run
such hooks after the lock release as similar hook do.
Bundle2 test are altered to ensure we are lockfree at hook running time.
This patch series is intended to allow bundle2 push reply part handlers to
make changes to the local repository; it has been developed in parallel with
an extension that allows the server to rebase incoming changesets while applying
them.
Most pushes already open a transaction in order to sync phase information.
This diff replaces that transaction with one that spans the entire push
operation.
This transaction will be used in a later patch to guard repository changes
made during the reply handler.
The double quotes are necessary, otherwise it tries to pipe into a program named
'short'. An '&' could serve as a command separator on Windows instead of ';',
but I don't see any obvious way to swap these depending on the platform. In
this case though, there really wasn't a need for multiple statements.
We are about to make bookmarks and phases available for hooks.
Therefore we need a witness for this new availability. We introduce
the new hooks in a distinct changeset to reduce the noise in the ones
with actual changes.
Such file are generated with a .pending prefix. It is up to the reader to
implement the necessary logic for reading pending files.
We add a test to ensure pending files are properly cleaned-up in both success and
error cases.
The post-transaction hooks run after the lock release (because hooks may want to
touch the repository), but they must only run if the transaction is successfully
closed.
We use the new 'addpostclose' method on transaction to register a callback
installing this post-lock-release call.
Hooks that run after the transaction need to be able to touch the
repository. So we need to run them after the lock release. This is
similar to what the "changegroup" hook is doing in the
`addchangegroup` function.
We need a wider set of hooks to process all the changes that happened during the
pull transaction. We reuse the experimental `b2x-transactionclose` hook set
from server's unbundle for consistency. This hook is experimental and will not
remains as-is forever, but this will open the door for experimentation in 3.2.
The source information can, should be applied once when opening the transaction
for the pull. This will lets element processed within a bundle2 be aware of them
and open the door to running a set of hooks when closing this pull transaction.
This is similar to what is done in server's unbundle call.
We store the source and url of the current data into `transaction.hookargs` this
let us inherit it from upper layers that may have created a much wider
transaction. We have to modify bundle2 at the same time to register the source
and url in the transaction. We have to do it in the same patch otherwise, the
`addchangegroup` call would fill these values and the hook calling will crash
because of the duplicated 'source' and 'url' arguments passed to the hook call.
addchangegroup creates a runhook function that is used to invoke the
changegroup and incoming hooks, but at the time the function is called,
the contents of hookargs associated with the transaction may have been
modified externally. For instance, bundle2 code affects it with
obsolescence markers and bookmarks info.
It also creates problems when a single transaction is used with multiple
changegroups added (as per an upcoming change), whereby the contents
of hookargs are that of after adding a latter changegroup when invoking
the hook for the first changegroup.
The obsolete._enabled flag has become a config option. This updates all but one
of the tests to use the minimal number of flags necessary for them to pass. For
most tests this is just 'createmarkers', for a couple tests it's
'allowunstable', and for even fewer it's 'exchange'.
We also track execution of the changegroup hook. The important information here
is to make sure the information that the transaction was processing a bundle2 is passed to
hook. This will let most hooks disable themselves while waiting for the hook
concluding bundle2 processing (the one we discovered to be not called for
pull in the previous changesets).
We can notice that this transaction wide hook is only happening during push and
it is missing changegroup-related information. We'll want to fix this but this
is not what this patch is about.
