This parameter is slightly confusingly named in wireproto, so it got
mis-specified from the start as 'push' instead of the URL to which we
are pushing. Sigh. I've got a patch for that which I'll mail
separately since it's not really appropriate for stable.
Fixes a regression in bundle2 from bundle1.
Bundle 2 is enable by default since 3.6 so we don't need this anymore.
The remaining use of 'experimental.bundle2-exp' are there to test legacy behavior
with bundle 1. Future patches will introduce a way outside of experimental to
test that.
chg currently does not support hg serve -d. It has a quick path testing if the
command is hg serve -d and fallbacks to hg if so. But the test only works if
"serve" is the first argument since the test wants to avoid false positives
(for example, "-r serve" is different).
This patch reorders "hg server" commands in tests, making them chg friendly.
Sometimes a txnclose or changegroup hook wants to iterate through all
the changesets in transaction: in that situation usually the revset
`$HG_NODE:` is used to select the revisions. Unfortunately this revset
sometimes may contain too many changesets because we don't have the
write lock while the hook runs newer changes may be added to
repository in the meantime.
That's why there is a need for extra variable carrying the information about
the last change in the transaction.
I recently implemented the server.bundle1* options to control whether
bundle1 exchange is allowed.
After thinking about Mozilla's strategy for handling generaldelta
rollout a bit more, I think server operators need an additional
lever: disable bundle1 if and only if the repo is generaldelta.
bundle1 exchange for non-generaldelta repos will not have the potential
for CPU explosion that generaldelta repos do. Therefore, it makes sense
for server operators to continue to allow bundle1 exchange for
non-generaldelta repos without having to set a per-repo hgrc option
to change the policy depending on whether the repo is generaldelta.
This patch introduces a new set of options to control bundle1 behavior
for generaldelta repos. These options enable server operators to limit
bundle1 restrictions to the class of repos that can be performance
issues. It also allows server operators to tie bundle1 access to store
format. In many server environments (including Mozilla's), legacy repos
will not be generaldelta and new repos will or might be. New repos often
aren't bound by legacy access requirements, so setting a global policy
that disallows access to new/generaldelta repos via bundle1 could be a
reasonable policy in many server environments. This patch makes this
policy very easy to implement (modify global hgrc, add options to
existing generaldelta repos to grandfather them in).
bundle2 is the new and preferred wire protocol format. For various
reasons, server operators may wish to force clients to use it.
One reason is performance. If a repository is stored in generaldelta,
the server must recompute deltas in order to produce the bundle1
changegroup. This can be extremely expensive. For mozilla-central,
bundle generation typically takes a few minutes. However, generating
a non-gd bundle from a generaldelta encoded mozilla-central requires
over 30 minutes of CPU! If a large repository like mozilla-central
were encoded in generaldelta and non-gd clients connected, they could
easily flood a server by cloning.
This patch gives server operators config knobs to control whether
bundle1 is allowed for push and pull operations. The default is to
support legacy bundle1 clients, making this patch backwards compatible.
Before bundle2, hook output from hook failures was prefixed with
"remote: ". Up to this point with bundle2, the output was converted to
the message to print in an Abort exception. This had 2 implications:
1) It was unclear whether an error message came from the local repo
or the remote
2) The exit code changed from 1 to 255
This patch changes the handling of error:abort bundle2 parts during push
to prefix the error message with "remote: ". This restores the old
behavior.
We still preserve the behavior of raising an Abort during bundle2
application failure. This is a regression from pre-bundle2 because the
exit code changed.
Because we no longer raise an Abort with the remote's message, we needed
to insert a message for the new Abort. So, I invented a new error
message for that. This is another change from pre-bundle2. However, I
like the new error message because it states unambiguously who aborted
the push failed, which I think is important for users so they can decide
what's next.
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
In the external pushrebase extension, it is valuable to be able to do some work
without taking the lock (like running expensive hooks). This enables
significantly higher commit throughput.
This patch adds an option to lazily acquire the lock. It means that all bundle2
part handlers that require writing to the repo must first call
op.gettransction(), when in this mode.
For some time, bookmark can and should be moved in the transaction. This
changeset migrates the 'hg bookmarks' commands to use a transaction.
Tests regarding rollback and transaction hooks are impacted for
obvious reasons. Some have to be slightly updated to keep testing the
same things. Some can just be dropped because they do not make sense
anymore.
This is a backout of 93589179c542, and a partial backout of 9b1628b91e74.
Windows won't execute 'dummyssh' directly, presumably because CreateProcess()
doesn't know how to execute a bash script:
$ hg clone -e "dummyssh" ssh://user@dummy/cloned sshclone
remote: 'dummyssh' is not recognized as an internal or external command,
remote: operable program or batch file.
abort: no suitable response from remote hg!
[255]
With the restoration of python as the executable, $TESTDIR needs to be restored
for these invocations, because python won't search $PATH for 'dummyssh':
$ hg clone -e "python dummyssh" ssh://user@dummy/cloned sshclone
remote: python: can't open file 'dummyssh': [Errno 2] No such file or directory
abort: no suitable response from remote hg!
[255]
We add a way to register "pushkey failure callback" that will be used if the
push is aborted by a pushkey failure. A part generator adding mandatory pushkey
parts should register a failure callback for all of them. The callback will be
in charge of generating a meaningful abort if this part fails.
If no callback is registered, the error is propagated.
Catch PushkeyFailed error in exchange.
So far, result of a pushkey operation had no consequence on the transaction
(beside the change). We makes it respect the 'mandatory' flag of part so that
failed pushkey call abort the whole transaction. This will allow rejecting
changes (primary target: changesets) regarding phases or bookmark criteria in
the future (when we will push such data in a mandatory part).
We currently raise an abort error because all clients support it. We'll
introduce a more precise error in the next changesets.
$TESTDIR is added to the path, so this is superfluous. Also,
inconsistent use of quotes means we might have broken on tests with
paths containing spaces.
I just discovered that we are not displaying ssh server output in real time
anymore. So we can just fall back to the bundle2 output capture for now. This
fix the race condition issue we where seeing in tests. Re-instating real time
output for ssh would fix the issue too but lets get the test to pass first.
The current bundle2 processing was capturing all output. This is nice as it
provide better meta data about what output what, but this was changing two
things:
1) adding a prefix "remote: " to "other" output during local push (issue4613)
2) local and ssh push does not provide real time output anymore (issue4615)
As we are unsure about what form should be used in (1) and how to solve (2) we
disable output capture in this two cases. Output capture can be forced using an
experimental option.
Until this changeset, we were only able to save output if an error happened
during the 'transaction.close()' phase. If the 'processbundle' call raised an
exception, the 'bundleoperation' object was never returned, so the reply bundle
was never accessible and no output could be salvaged. We introduce a quick (but
not very elegant) fix to gain access to any reply created during the processing.
This conclude this output related series. We should hopefully be able client-side to see the
whole server output, in a proper order.
The code is now complex enough that a refactoring of it would make sense on
default.
We were capturing all output issue during bundle2 processing, and all output
issue during transaction rollback in case of failure. However, the output issue
during transaction commit was still roaming the land freely. It is now put back
in line.
This let the user see output from 'pretxnclose' and 'txnclose' (and related) in
the right order.
External hook used to directly write on stdout and stderr. As a result their
output was not captured by the bundle2 processing. This resulted in confusing
out of order output on the client side. We are now capturing hooks output in
this context.
The output from the transaction rollback was not included into the reply bundle.
It was eventually caught by the usual 'unbundle' output capture and sent to the
client but the result was out of order on the client side. We now capture the
output for the transaction release and transmit it the same way as all other
output.
We should probably rethink the whole output capture things but this would not be
appropriate for stable.
The is still multiple cases were output failed to be properly capture, they will
be fixed in later changesets.
Since transaction are used for more than just changesets, it is possible
to have a transaction without new changesets at all. In this case no
''00changelog.i.a' are written. In all cases the 'changelog.readpending'
method is called if the repository has any pending data. The 'revlog' logic
provides empty content if the file is missing, so the whole operation
resulted in an empty changelog.
We now skip reading the pending file if it is missing.
In case of errors, output parts salvaged from the reply bundle need to be
processed for outputting their content. This concludes our quest for fixing
issue4594.
In case of errors, output parts salvaged from the reply bundle are re-injected
into the bundle carrying the exception.
We still need to fix the situation for non-wireprotocol push.
A bundle2 may contain bookmark updates (or other extension content) that
requires the 'wlock' to be written. As 'wlock' must be acquired before 'lock',
we must stay on the side of caution and use both in all case to ensure their
ordering.
This should help us to catch new locking order issues as soon as possible.
There are two harmless test updates (from the config change). Moreover, some
bundle2 tests are displaying warning for a legitimate reason. The use of pushkey
during the unbundle process may requires the 'wlock' after 'lock' (around the
whole unbundle process was taken). This is non-trivial to fix, so I better have
the check on, with the warning in the test than the check off. See issue4596 for
details.
It is finally time to freeze the bundle2 format! To do so we:
- rename HG2Y to HG20,
- drop "b2x:" prefix from all part names,
- rename capability to "bundle2-exp" to "bundle2"
- rename the hook flag from 'bundle2-exp' to 'bundle2'
The pushkey operation used to be in its own wireprotocol command and (in
practice) always be lock free when running the hook. With bundle2, it happen in
a greater scheme and a hook running locking command would get stuck. We now run
such hooks after the lock release as similar hook do.
Bundle2 test are altered to ensure we are lockfree at hook running time.
This patch series is intended to allow bundle2 push reply part handlers to
make changes to the local repository; it has been developed in parallel with
an extension that allows the server to rebase incoming changesets while applying
them.
Most pushes already open a transaction in order to sync phase information.
This diff replaces that transaction with one that spans the entire push
operation.
This transaction will be used in a later patch to guard repository changes
made during the reply handler.
The double quotes are necessary, otherwise it tries to pipe into a program named
'short'. An '&' could serve as a command separator on Windows instead of ';',
but I don't see any obvious way to swap these depending on the platform. In
this case though, there really wasn't a need for multiple statements.
We are about to make bookmarks and phases available for hooks.
Therefore we need a witness for this new availability. We introduce
the new hooks in a distinct changeset to reduce the noise in the ones
with actual changes.
Such file are generated with a .pending prefix. It is up to the reader to
implement the necessary logic for reading pending files.
We add a test to ensure pending files are properly cleaned-up in both success and
error cases.
The post-transaction hooks run after the lock release (because hooks may want to
touch the repository), but they must only run if the transaction is successfully
closed.
We use the new 'addpostclose' method on transaction to register a callback
installing this post-lock-release call.
Hooks that run after the transaction need to be able to touch the
repository. So we need to run them after the lock release. This is
similar to what the "changegroup" hook is doing in the
`addchangegroup` function.
We need a wider set of hooks to process all the changes that happened during the
pull transaction. We reuse the experimental `b2x-transactionclose` hook set
from server's unbundle for consistency. This hook is experimental and will not
remains as-is forever, but this will open the door for experimentation in 3.2.
