21c2df59a regressed bundle2 by catching all exceptions and trying to handle
them. The old behavior was to allow KeyboardInterrupts to throw and not have
graceful cleanup, which allowed it to exit immediately. Let's go back to that
behavior.
Differential Revision: https://phab.mercurial-scm.org/D960
The 'stepdone' set is design to be a client side mechanism. If the client used
some advanced capabilities to request necessary information (changeset,
obsmarkers, phases, etc). It marks the steps as done to avoid having a less
advanced mechanism issue a duplicated request.
So, the "stepdone.add('phases')" should be the result of a client choice,
because only the client can know it has requested all it needed to request. In
4a08cf1a2cfe this principle was broken because any phase-heads part sent by
the server to the client would declare the phases retrieval complete.
Now that there is an official phases related capability and code associated to
it. We do not need the change in 4a08cf1a2cfe anymore and we can back it out.
This brings back 'stepdone' management for 'phases' in line with the rest of
the code (including other phases handing).
Here is an example of potential misbehavior that 4a08cf1a2cfe introduced:
Imagine a server that pre-computes bundles. The bundles contains a changegroup
part and an (advisory) 'phase-heads' part. When a pull occurs, precomputed
bundled are reused if available. As the phase part is advisory it can be sent
to all clients. However they could be relevant changesets without phase
information. Either because they are already common or because they had no
precomputed bundle for them yet.
If receiving any 'phase-heads' parts disable subsequent phases re-trivial
parts, the client will not request phase data for all relevant changesets. For
example common changesets will not turn public.
A new bundle2 capability 'phases' has been added. If 'heads' is part of the
supported value for 'phases', the server supports reading and sending 'phase-
heads' bundle2 part.
Server is now able to process a 'phases' boolean parameter to 'getbundle'. If
'True', a 'phase-heads' bundle2 part will be included in the bundle with phase
information relevant to the whole pulled set. If this method is available the
phases listkey namespace will no longer be listed.
Beside the more efficient encoding of the data, this new method will greatly
improve the phase exchange efficiency for repositories with non-served
changesets (obsolete, secret) since we'll no longer send data about the
filtered heads.
Add a new 'devel.legacy.exchange' config item to allow fallback to the old
'listkey in bundle2' method.
Reminder: the pulled set is not just the changesets bundled by the pull. It
also contains changeset selected by the "pull specification" on the client
side (eg: everything for bare pull). One of the reason why the 'pulled set' is
important is to make sure we can move -common- nodes to public.
The next patch will use the 'phase-heads' part to exchange phase data relevant to
the pulled set.
'handlephases' currently acquires a transaction even in case of no-op pull,
which would results in an empty transaction and messing with the existing
journal.
Pass the transaction fetcher to updatephases so it can fetch it if necessary.
Valid part names are restricted to [a-zA-Z0-9_:-]+, so I'm not worried
about having quoting present in places where we should have
predominantly valid part names. This will significantly ease the
Python 3 transition, and simultaneously isn't a BC because this is
only in error messages that should never be shown.
Now that the part processing loop is tiny, let's move it to a separate function.
This will allow extensions to completely replace the part processing logic,
without having to replace the overall bundle processing logic or the stream
maintenance logic.
This will be useful for the infinitepush extension, so it can completely take
over receiving a bundle and rerouting it to a side store. This will also make it
easier to upstream the infinitepush functionality later.
Differential Revision: https://phab.mercurial-scm.org/D709
As part of refactoring bundle part processing let's move handler validation to
its own function.
Differential Revision: https://phab.mercurial-scm.org/D707
The processpart function also did some stream maintenance, so let's move it to
the part iterator as well, as part of moving all part iteration logic into the
class.
There is one place processpart is called outside of the normal loop, so we
manually handle the seek there.
The now-empty try/finally will be removed in a later patch, for ease of review.
Differential Revision: https://phab.mercurial-scm.org/D706
I've tripped on this several times now, and am tired of debugging. Now
the header parts are part of the error message when the ''.join()
fails, which makes debugging obvious.
As part of separating the part iteration logic from the part handling logic,
let's move the exception handling to the part iterator class.
Differential Revision: https://phab.mercurial-scm.org/D705
As part of moving the part iterator logic to a separate class, let's move the
part counting logic and the output for it.
Differential Revision: https://phab.mercurial-scm.org/D704
Currently, the part iterator logic is tightly coupled with the part handling
logic, which means it's hard to replace the part handling logic without
duplicating the part iterator bits.
In a future diff we'll want to be able to replace all part handling, so let's
begin refactoring the part iterator logic to it's own class.
Differential Revision: https://phab.mercurial-scm.org/D703
As part of reducing the number of changegroup creation APIs, let's replace
getchangegroup with calls to makechangegroup. This is mostly a drop in
replacement, but it does change the version specifier to be required, so it's
more obvious which callers are creating old version 1 changegroups still.
Differential Revision: https://phab.mercurial-scm.org/D669
Previously, iterparts would yield the part to users, then consume the part. This
changed the part after the user was given it and left it at the end, both of
which seem unexpected. Let's seek back to the beginning after we've consumed
it. I tried not seeking to the end at all, but that seems important for the
overall bundle2 consumption.
This is used in a future patch to let us move the bundlerepo
bundle2-changegroup-part to be handled entirely within the for loop, instead of
having to do a seek back to 0 after the entire loop finishes.
Differential Revision: https://phab.mercurial-scm.org/D289
The Mercurial core server doesn't yet include phase-heads parts in the
bundle, but our Google-internal server wants to do
that. Unfortunately, the usual exchange still happens even if
phase-heads part is included (including the short-circuited one for
old/publishing servers). That means that even if our server (again,
the Google-internal one, but also future Mercurial core servers)
includes a phase-heads part to indicate that some heads should be
drafts, that would still get overwritten by the phase updating that
happens after. So let's fix that by marking the phase step done if we
receive at least one phase-heads part in the bundle.
Differential Revision: https://phab.mercurial-scm.org/D440
Changeset aa97e972460f introduce more complex logic around
'bundleoperation.gettransaction'. In that process it turns the old "attribute"
into a proper method which breaks the code that detects the "transaction
availability".
The change was visible in 'test-acl.t', fixing this reverts the test changes.
Differential Revision: https://phab.mercurial-scm.org/D303
Fixes some Python 3 regressions.
We don't use %d here because the part id is actually an
Optional[int]. It should always be initialized to a non-None value by
the time this code executes, but we shouldn't blindly depend on that
being the case.
Differential Revision: https://phab.mercurial-scm.org/D272
pushvars extension in fbext adds a --pushvars flag to push command using which
one send strings to server which becomes environment variables there prepended
with HG_USERVAR_. These variables can then be used to run hooks on the server.
The extension is moved directly to core and unbundling of the strings and
converting them to environment variables at server is disabled by default for
security reasons. One can turn that on by following config:
[push]
pushvars.server = true
This patch also adds the test for the extension.
Differential Revision: https://phab.mercurial-scm.org/D210
When a transaction is started, we must load the hookargs from the
bundleoperation object to the transaction so that they can be used in the
transaction. Also this patch makes sure no more hookargs are added to the
bundleoperation object once the transaction starts.
This is a part of porting fb extension bundle2hooks to core.
Differential Revision: https://phab.mercurial-scm.org/D209
There are extensions like pushrebase, pushvars which run hooks on a server
before taking the lock. Since the lock is not taken, transaction is not there,
so the hookargs can't be stored on the transaction. Adding hooksargs to bundle
operation object will help in running hooks before taking the lock.
This is a part of moving fb's extension bundle2hooks to core.
Differential Revision: https://phab.mercurial-scm.org/D208
Now that we records "all" changes happening in a transaction (in tr.changes)
we will be able to provide better report on various changes (phases turned
public, changeset obsoleted, branch merged or created, etc..)
This is far too late in the cycle to play with this, but having this existing
method called more widely will help extensions to play around with various
options during the 4.4 cycle.
Instead of calling registersummarycallback only for transactions we want, we
always call it and use the transaction name to decide when to report (eg: we
do not want `hg amend` to report new obsoleted changesets). Filtering on
transaction name does not seems great, but seems good enough for the moment.
We can change the API during the next cycle.
The previous manual call during unbundling of the bundle2 "obsmarkers" part is
no longer necessary and has been dropped.
This is done by a script [2] using RedBaron [1], a tool designed for doing
code refactoring. All "default" values are decided by the script and are
strongly consistent with the existing code.
There are 2 changes done manually to fix tests:
[warn] mercurial/exchange.py: experimental.bundle2-output-capture: default needs manual removal
[warn] mercurial/localrepo.py: experimental.hook-track-tags: default needs manual removal
Since RedBaron is not confident about how to indent things [2].
[1]: https://github.com/PyCQA/redbaron
[2]: https://github.com/PyCQA/redbaron/issues/100
[3]:
#!/usr/bin/env python
# codemod_configitems.py - codemod tool to fill configitems
#
# Copyright 2017 Facebook, Inc.
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
from __future__ import absolute_import, print_function
import os
import sys
import redbaron
def readpath(path):
with open(path) as f:
return f.read()
def writepath(path, content):
with open(path, 'w') as f:
f.write(content)
_configmethods = {'config', 'configbool', 'configint', 'configbytes',
'configlist', 'configdate'}
def extractstring(rnode):
"""get the string from a RedBaron string or call_argument node"""
while rnode.type != 'string':
rnode = rnode.value
return rnode.value[1:-1] # unquote, "'str'" -> "str"
def uiconfigitems(red):
"""match *.ui.config* pattern, yield (node, method, args, section, name)"""
for node in red.find_all('atomtrailers'):
entry = None
try:
obj = node[-3].value
method = node[-2].value
args = node[-1]
section = args[0].value
name = args[1].value
if (obj in ('ui', 'self') and method in _configmethods
and section.type == 'string' and name.type == 'string'):
entry = (node, method, args, extractstring(section),
extractstring(name))
except Exception:
pass
else:
if entry:
yield entry
def coreconfigitems(red):
"""match coreconfigitem(...) pattern, yield (node, args, section, name)"""
for node in red.find_all('atomtrailers'):
entry = None
try:
args = node[1]
section = args[0].value
name = args[1].value
if (node[0].value == 'coreconfigitem' and section.type == 'string'
and name.type == 'string'):
entry = (node, args, extractstring(section),
extractstring(name))
except Exception:
pass
else:
if entry:
yield entry
def registercoreconfig(cfgred, section, name, defaultrepr):
"""insert coreconfigitem to cfgred AST
section and name are plain string, defaultrepr is a string
"""
# find a place to insert the "coreconfigitem" item
entries = list(coreconfigitems(cfgred))
for node, args, nodesection, nodename in reversed(entries):
if (nodesection, nodename) < (section, name):
# insert after this entry
node.insert_after(
'coreconfigitem(%r, %r,\n'
' default=%s,\n'
')' % (section, name, defaultrepr))
return
def main(argv):
if not argv:
print('Usage: codemod_configitems.py FILES\n'
'For example, FILES could be "{hgext,mercurial}/*/**.py"')
dirname = os.path.dirname
reporoot = dirname(dirname(dirname(os.path.abspath(__file__))))
# register configitems to this destination
cfgpath = os.path.join(reporoot, 'mercurial', 'configitems.py')
cfgred = redbaron.RedBaron(readpath(cfgpath))
# state about what to do
registered = set((s, n) for n, a, s, n in coreconfigitems(cfgred))
toregister = {} # {(section, name): defaultrepr}
coreconfigs = set() # {(section, name)}, whether it's used in core
# first loop: scan all files before taking any action
for i, path in enumerate(argv):
print('(%d/%d) scanning %s' % (i + 1, len(argv), path))
iscore = ('mercurial' in path) and ('hgext' not in path)
red = redbaron.RedBaron(readpath(path))
# find all repo.ui.config* and ui.config* calls, and collect their
# section, name and default value information.
for node, method, args, section, name in uiconfigitems(red):
if section == 'web':
# [web] section has some weirdness, ignore them for now
continue
defaultrepr = None
key = (section, name)
if len(args) == 2:
if key in registered:
continue
if method == 'configlist':
defaultrepr = 'list'
elif method == 'configbool':
defaultrepr = 'False'
else:
defaultrepr = 'None'
elif len(args) >= 3 and (args[2].target is None or
args[2].target.value == 'default'):
# try to understand the "default" value
dnode = args[2].value
if dnode.type == 'name':
if dnode.value in {'None', 'True', 'False'}:
defaultrepr = dnode.value
elif dnode.type == 'string':
defaultrepr = repr(dnode.value[1:-1])
elif dnode.type in ('int', 'float'):
defaultrepr = dnode.value
# inconsistent default
if key in toregister and toregister[key] != defaultrepr:
defaultrepr = None
# interesting to rewrite
if key not in registered:
if defaultrepr is None:
print('[note] %s: %s.%s: unsupported default'
% (path, section, name))
registered.add(key) # skip checking it again
else:
toregister[key] = defaultrepr
if iscore:
coreconfigs.add(key)
# second loop: rewrite files given "toregister" result
for path in argv:
# reconstruct redbaron - trade CPU for memory
red = redbaron.RedBaron(readpath(path))
changed = False
for node, method, args, section, name in uiconfigitems(red):
key = (section, name)
defaultrepr = toregister.get(key)
if defaultrepr is None or key not in coreconfigs:
continue
if len(args) >= 3 and (args[2].target is None or
args[2].target.value == 'default'):
try:
del args[2]
changed = True
except Exception:
# redbaron fails to do the rewrite due to indentation
# see https://github.com/PyCQA/redbaron/issues/100
print('[warn] %s: %s.%s: default needs manual removal'
% (path, section, name))
if key not in registered:
print('registering %s.%s' % (section, name))
registercoreconfig(cfgred, section, name, defaultrepr)
registered.add(key)
if changed:
print('updating %s' % path)
writepath(path, red.dumps())
if toregister:
print('updating configitems.py')
writepath(cfgpath, cfgred.dumps())
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
cg.apply used to returns the added nodes. Callers doesn't have a use for it
anymore, remove the added node and stops recording it in the current
operation.
This information was added in the current release cycle so no extensions
breakage should happens.
updatephases have no use of the 'addednodes' parameter since 44be3dc1fec8.
However caller are still passing it for nothing, remove the parameter and
remove computing of the added nodes in caller.
If we are bundling secret changeset and the bundle will contain phase, we
request the changegroup to be applied as secret.
It will be useful for next patch as we are now sure that secrets changesets
are applied as secret and not applied as draft then forced to secret.
By default unbundled changesets are drafts. We want to reduce the number of
phases changes during unbundling by giving the possibility to the bundle to
indicate the phase of unbundled changesets.
The longer terms goal is to add phase movement tracking in tr.changes and the
'retractboundary' call is making it more complicated than we want.
This is a first basic visible usage of the changes tracking in the transaction.
We adds a new function computing the pre-existing changesets obsoleted by a
transaction and a transaction call back displaying this information.
Example output:
added 1 changesets with 1 changes to 1 files (+1 heads)
3 new obsolescence markers
obsoleted 1 changesets
The goal is to evolve the transaction summary into something bigger, gathering
existing output there and adding new useful one. This patch is a good first step
on this road. The new output is basic but give a user to the content of
tr.changes['obsmarkers'] and give an idea of the new options we haves. I expect
to revisit the message soon.
The caller recording the transaction summary should also be moved into a more
generic location but further refactoring is needed before it can happen.
