Summary:
This check is useful and detects real errors (ex. fbconduit). Unfortunately
`arc lint` will run it with both py2 and py3 so a lot of py2 builtins will
still be warned.
I didn't find a clean way to disable py3 check. So this diff tries to fix them.
For `xrange`, the change was done by a script:
```
import sys
import redbaron
headertypes = {'comment', 'endl', 'from_import', 'import', 'string',
'assignment', 'atomtrailers'}
xrangefix = '''try:
xrange(0)
except NameError:
xrange = range
'''
def isxrange(x):
try:
return x[0].value == 'xrange'
except Exception:
return False
def main(argv):
for i, path in enumerate(argv):
print('(%d/%d) scanning %s' % (i + 1, len(argv), path))
content = open(path).read()
try:
red = redbaron.RedBaron(content)
except Exception:
print(' warning: failed to parse')
continue
hasxrange = red.find('atomtrailersnode', value=isxrange)
hasxrangefix = 'xrange = range' in content
if hasxrangefix or not hasxrange:
print(' no need to change')
continue
# find a place to insert the compatibility statement
changed = False
for node in red:
if node.type in headertypes:
continue
# node.insert_before is an easier API, but it has bugs changing
# other "finally" and "except" positions. So do the insert
# manually.
# # node.insert_before(xrangefix)
line = node.absolute_bounding_box.top_left.line - 1
lines = content.splitlines(1)
content = ''.join(lines[:line]) + xrangefix + ''.join(lines[line:])
changed = True
break
if changed:
# "content" is faster than "red.dumps()"
open(path, 'w').write(content)
print(' updated')
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
```
For other py2 builtins that do not have a py3 equivalent, some `# noqa`
were added as a workaround for now.
Reviewed By: DurhamG
Differential Revision: D6934535
fbshipit-source-id: 546b62830af144bc8b46788d2e0fd00496838939
Per the last discussion, this is more reliable and consistent way than
suppressing an error. For grep, erroring out might be inconvenient, but
for curl, non-zero exit status should be detected. The latter wouldn't be
possible if non-zero status is ignored.
https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-October/105727.html
Revisions are added consecutively, so a range can easily represent them
in the changes list. This saves around 45 Bytes / revision on 64bit
platforms and reduces the memory footprint of issue5691 by 15MB.
Don't copy changes['revs'] in getobsoleted. Ranges have a very efficient
contains implementation already.
Differential Revision: https://phab.mercurial-scm.org/D1615
When the option is set, the repository will reject any transaction adding
multiple heads to the same named branch.
For now we reject all scenario with multiple heads. One could imagine handling
closed branches differently. We prefer to keep things simple for now. The
feature might get extended later. Branch closing is not the best experience
Mercurial has to offer anyway.
The next patch will wrap the conversion code, in order to write out a
requirement for 'lfs' when appropriate. Wrapping convcmd.convertsink() in an
afterloaded callback works fine when the convert extension is enabled by the
user. The problem here is that lfconvert uses the convert extension, whether or
not it was formally enabled by the user.
My first attempt was to have lfs install an afterloaded callback that would wrap
the convert sink if convert was loaded, or wrap lfconvert if it wasn't. Then
the lfconvert override could install an afterloaded callback to try wrapping the
convert sink again, before calling the original lfconvert. But that breaks down
if largefiles can't load the convert extension on the fly. [1] Further, some
tests were failing with an error indicating that the size of the afterloaded
list changed while iterating it.
Yuya mentioned that maybe some bits of convert could be moved into core, but I'm
not sure where to draw that line. The convertsink() method depends on the list
of sinks, which in turn depends on the sink classes.
[1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-November/108038.html
Repoview can have a different life cycle, causing issue in some corner
cases. The particular instance that revealed this comes from localpeer. The
localpeer hold a reference to the unfiltered repository, but calling 'local()'
will create an on-demand 'visible' repoview. That repoview can be garbaged
collected any time. Here is a simplified step by step reproduction::
1) tr = peer.local().transaction('foo')
2) tr.close()
After (1), the repoview object is garbage collected, so weakref used in (2)
point to nothing.
Thanks to Sean Farley for helping raising and debugging this issue.
When origbackuppath is set, when looking to see if a file we are backing up
conflicts with a directory in the origbackuppath, we incorrectly match on
symlinks to directories. This means we try to call vfs.rmtree on the
symlink, which fails.
Differential Revision: https://phab.mercurial-scm.org/D1311
We'll need the same logic in forthcoming changeset to handle --line-range
option in 'hg log' command.
The function lives in scmutil.py (rather than util.py) as it uses match and
pathutil modules.
This patch adds a metadata argument to cleanupnodes() which will be dict and can
be passed to obsmarker.createmarkers() and can be stored on the obsmarker.
In cases when obsolescence is not enabled, the metadata argument is useless.
This is a step towards storing a note in amend.
Differential Revision: https://phab.mercurial-scm.org/D1093
Upon pull or unbundle, we display a message with the range of new revisions
fetched. This revision range could readily be used after a pull to look out
what's new with 'hg log'. The algorithm takes care of filtering "obsolete"
revisions that might be present in transaction's "changes" but should not be
displayed to the end user.
In registersummarycallback(), we extra generic bits of the existing
"reportsummary" function into a decorator which will be used in forthcoming
changesets to add new summary callbacks.
Add an inner txmatch function in registersummarycallback() factoring out the
logic to determine if the transaction matches a particular sources set. We'll
reuse this function to add some new report logic in the new changeset.
When ui.origbackuppath is set, .orig files are stored outside of the working
copy. However conflicts can occur when files or directories end up having the
same name. These conflicts cause Mercurial to abort, even if they've been
created as a result of different backups.
Make sure we always replace files or directories in the origbackuppath if
they conflict with another file or directory.
Test Plan:
Add new unit test for conflicting paths.
Differential Revision: https://phab.mercurial-scm.org/D680
This adds basic support for extdata, a way to add external data
sources for revsets and templates. An extdata data source is simply a
list of lines of the form:
<revision identifier>[<space><freeform text>]\n
An extdata source is configured thusly:
[extdata]
name = <a url or path>
urls of the form shell: are launch shell commands to generate data.
This patch is slightly modified by Yuya Nishihara as follows:
- fix typo
- remove unused function
- remove future expansion point for parameter (which can be added later
as the extdata revset/template are experimental)
You can see the original patch at
https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-September/088426.html
The arguments are especially non-obvious because the order is
different from dirstate.status().
Differential Revision: https://phab.mercurial-scm.org/D846
This is a regression caused by 80d53a39fbcc. We have documented the behavior
in rebase help:
Rebase will destroy original commits unless you use "--keep". It will
also move your bookmarks (even if you do).
So let's restore the old behavior.
It is done by changing `scmutil.cleanupnodes` to accept more information so
a node could have different "movement destination" from "successors". It
also helps simplifying the callsite as a side effect - the special bookmark
movement logic in rebase is removed.
Differential Revision: https://phab.mercurial-scm.org/D727
The next patch will pass in overrides for bookmark moves, which is a
mapping itself, so let's rename "mapping" to "replacements" to
distinguish them better.
Differential Revision: https://phab.mercurial-scm.org/D749
We will soon want to pass in overrides for bookmark movements and this
will make that patch simpler. I also think this makes the code easier
to follow regardless of the later patch.
Differential Revision: https://phab.mercurial-scm.org/D748
The initial attempt was to discard cache when appropriate, but it appears
to be error prone. We had to carefully inspect all places where audit() is
called e.g. without actually updating filesystem, before removing files and
directories, etc.
So, this patch disables the cache of audited paths by default, and enables
it only for the following cases:
- short-lived auditor objects
- repo.vfs, repo.svfs, and repo.cachevfs, which are managed directories
and considered sort of append-only (a file/directory would never be
replaced with a symlink)
There would be more cacheable vfs objects (e.g. mq.queue.opener), but I
decided not to inspect all of them in this patch. We can make them cached
later.
Benchmark result:
- using old clone of http://selenic.com/repo/linux-2.6/ (38319 files)
- on tmpfs
- run HGRCPATH=/dev/null hg up -q --time tip && hg up -q null
- try 4 times and take the last three results
original:
real 7.480 secs (user 1.140+22.760 sys 0.150+1.690)
real 8.010 secs (user 1.070+22.280 sys 0.170+2.120)
real 7.470 secs (user 1.120+22.390 sys 0.120+1.910)
clearcache (the other series):
real 7.680 secs (user 1.120+23.420 sys 0.140+1.970)
real 7.670 secs (user 1.110+23.620 sys 0.130+1.810)
real 7.740 secs (user 1.090+23.510 sys 0.160+1.940)
enable cache only for vfs and svfs (this series):
real 8.730 secs (user 1.500+25.190 sys 0.260+2.260)
real 8.750 secs (user 1.490+25.170 sys 0.250+2.340)
real 9.010 secs (user 1.680+25.340 sys 0.280+2.540)
remove cache function at all (for reference):
real 9.620 secs (user 1.440+27.120 sys 0.250+2.980)
real 9.420 secs (user 1.400+26.940 sys 0.320+3.130)
real 9.760 secs (user 1.530+27.270 sys 0.250+2.970)
Now that we records "all" changes happening in a transaction (in tr.changes)
we will be able to provide better report on various changes (phases turned
public, changeset obsoleted, branch merged or created, etc..)
This is far too late in the cycle to play with this, but having this existing
method called more widely will help extensions to play around with various
options during the 4.4 cycle.
Instead of calling registersummarycallback only for transactions we want, we
always call it and use the transaction name to decide when to report (eg: we
do not want `hg amend` to report new obsoleted changesets). Filtering on
transaction name does not seems great, but seems good enough for the moment.
We can change the API during the next cycle.
The previous manual call during unbundling of the bundle2 "obsmarkers" part is
no longer necessary and has been dropped.
This is done by a script [2] using RedBaron [1], a tool designed for doing
code refactoring. All "default" values are decided by the script and are
strongly consistent with the existing code.
There are 2 changes done manually to fix tests:
[warn] mercurial/exchange.py: experimental.bundle2-output-capture: default needs manual removal
[warn] mercurial/localrepo.py: experimental.hook-track-tags: default needs manual removal
Since RedBaron is not confident about how to indent things [2].
[1]: https://github.com/PyCQA/redbaron
[2]: https://github.com/PyCQA/redbaron/issues/100
[3]:
#!/usr/bin/env python
# codemod_configitems.py - codemod tool to fill configitems
#
# Copyright 2017 Facebook, Inc.
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
from __future__ import absolute_import, print_function
import os
import sys
import redbaron
def readpath(path):
with open(path) as f:
return f.read()
def writepath(path, content):
with open(path, 'w') as f:
f.write(content)
_configmethods = {'config', 'configbool', 'configint', 'configbytes',
'configlist', 'configdate'}
def extractstring(rnode):
"""get the string from a RedBaron string or call_argument node"""
while rnode.type != 'string':
rnode = rnode.value
return rnode.value[1:-1] # unquote, "'str'" -> "str"
def uiconfigitems(red):
"""match *.ui.config* pattern, yield (node, method, args, section, name)"""
for node in red.find_all('atomtrailers'):
entry = None
try:
obj = node[-3].value
method = node[-2].value
args = node[-1]
section = args[0].value
name = args[1].value
if (obj in ('ui', 'self') and method in _configmethods
and section.type == 'string' and name.type == 'string'):
entry = (node, method, args, extractstring(section),
extractstring(name))
except Exception:
pass
else:
if entry:
yield entry
def coreconfigitems(red):
"""match coreconfigitem(...) pattern, yield (node, args, section, name)"""
for node in red.find_all('atomtrailers'):
entry = None
try:
args = node[1]
section = args[0].value
name = args[1].value
if (node[0].value == 'coreconfigitem' and section.type == 'string'
and name.type == 'string'):
entry = (node, args, extractstring(section),
extractstring(name))
except Exception:
pass
else:
if entry:
yield entry
def registercoreconfig(cfgred, section, name, defaultrepr):
"""insert coreconfigitem to cfgred AST
section and name are plain string, defaultrepr is a string
"""
# find a place to insert the "coreconfigitem" item
entries = list(coreconfigitems(cfgred))
for node, args, nodesection, nodename in reversed(entries):
if (nodesection, nodename) < (section, name):
# insert after this entry
node.insert_after(
'coreconfigitem(%r, %r,\n'
' default=%s,\n'
')' % (section, name, defaultrepr))
return
def main(argv):
if not argv:
print('Usage: codemod_configitems.py FILES\n'
'For example, FILES could be "{hgext,mercurial}/*/**.py"')
dirname = os.path.dirname
reporoot = dirname(dirname(dirname(os.path.abspath(__file__))))
# register configitems to this destination
cfgpath = os.path.join(reporoot, 'mercurial', 'configitems.py')
cfgred = redbaron.RedBaron(readpath(cfgpath))
# state about what to do
registered = set((s, n) for n, a, s, n in coreconfigitems(cfgred))
toregister = {} # {(section, name): defaultrepr}
coreconfigs = set() # {(section, name)}, whether it's used in core
# first loop: scan all files before taking any action
for i, path in enumerate(argv):
print('(%d/%d) scanning %s' % (i + 1, len(argv), path))
iscore = ('mercurial' in path) and ('hgext' not in path)
red = redbaron.RedBaron(readpath(path))
# find all repo.ui.config* and ui.config* calls, and collect their
# section, name and default value information.
for node, method, args, section, name in uiconfigitems(red):
if section == 'web':
# [web] section has some weirdness, ignore them for now
continue
defaultrepr = None
key = (section, name)
if len(args) == 2:
if key in registered:
continue
if method == 'configlist':
defaultrepr = 'list'
elif method == 'configbool':
defaultrepr = 'False'
else:
defaultrepr = 'None'
elif len(args) >= 3 and (args[2].target is None or
args[2].target.value == 'default'):
# try to understand the "default" value
dnode = args[2].value
if dnode.type == 'name':
if dnode.value in {'None', 'True', 'False'}:
defaultrepr = dnode.value
elif dnode.type == 'string':
defaultrepr = repr(dnode.value[1:-1])
elif dnode.type in ('int', 'float'):
defaultrepr = dnode.value
# inconsistent default
if key in toregister and toregister[key] != defaultrepr:
defaultrepr = None
# interesting to rewrite
if key not in registered:
if defaultrepr is None:
print('[note] %s: %s.%s: unsupported default'
% (path, section, name))
registered.add(key) # skip checking it again
else:
toregister[key] = defaultrepr
if iscore:
coreconfigs.add(key)
# second loop: rewrite files given "toregister" result
for path in argv:
# reconstruct redbaron - trade CPU for memory
red = redbaron.RedBaron(readpath(path))
changed = False
for node, method, args, section, name in uiconfigitems(red):
key = (section, name)
defaultrepr = toregister.get(key)
if defaultrepr is None or key not in coreconfigs:
continue
if len(args) >= 3 and (args[2].target is None or
args[2].target.value == 'default'):
try:
del args[2]
changed = True
except Exception:
# redbaron fails to do the rewrite due to indentation
# see https://github.com/PyCQA/redbaron/issues/100
print('[warn] %s: %s.%s: default needs manual removal'
% (path, section, name))
if key not in registered:
print('registering %s.%s' % (section, name))
registercoreconfig(cfgred, section, name, defaultrepr)
registered.add(key)
if changed:
print('updating %s' % path)
writepath(path, red.dumps())
if toregister:
print('updating configitems.py')
writepath(cfgpath, cfgred.dumps())
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
The 'successors' part of the mappings used of be a tuple. This avoid issue from
code consuming the generator "by mistake". For example, an extension inspecting the
mapping content used to be able to iterate over the successors mapping without
consequence.
Since the mapping are small we do not expect any performance impact we use tuple
again for this.
cleanupnodes takes care of bookmark movement, and bookmark movement could
cause bookmark divergent resolution as a side effect. This patch adds such
bookmark divergent resolution logic so future rebase migration will be
easier.
The revset is carefully written to be equivalent to what rebase does today.
Although I think it might make sense to remove divergent bookmarks more
aggressively, for example:
F book@1
|
E book@2
|
| D book
| |
| C
|/
B book@3
|
A
When rebase -s C -d E, "book@1" will be removed, "book@3" will be kept,
and the end result is:
D book
|
C
|
F
|
E book@2 (?)
|
B book@3
|
A
The question is should we keep book@2? The current logic keeps it. If we
choose not to (makes some sense to me), the "deleterevs" revset could be
simplified to "newnode % oldnode".
For now, I just make it compatible with the existing behavior. If we want to
make the "deleterevs" revset simpler, we can always do it in the future.
In some valid usecases, the "mapping" received by scmutil.cleanupnodes have
filtered nodes. Use unfiltered repo to access them correctly.
The added test case will fail with the old cleanupnodes code.
This is important to migrate histedit to use the cleanupnodes API.
We do "'%d:%s' % (ctx...)" at several places, so let's formalize it. A low-
level function, formatrevnode(ui, rev, node), is extracted so we can pass
a manifest rev/node pair.
Note that hex() for manifest output can be replaced with hexfunc() because
it is printed only when debugflag is set.
i18n/de.po is updated so test-log.t passes with no error.
When ui.origbackuppath is set, .orig files are stored outside of the working
copy, however they still have a .orig suffix appended to them. This can cause
unexpected conflicts, particularly when tracked files or directories have .orig
at the end.
This change removes the .orig suffix from files stored in an out-of-tree
origbackuppath.
Test Plan:
Update and run unit tests.
Differential Revision: https://phab.mercurial-scm.org/D679
This patch defines `SRC` (a single source revision) and `ALLSRC` (all source
revisions) to be valid names in `--dest` revset if `--src` or `--rev` is
used. So destination could be defined differently according to source
revisions. The names are capitalized to make it clear they are "dynamically
defined", distinguishable from normal revsets (Thanks Augie for the
suggestion).
This is useful, for example, `-r 'orphan()' -d 'calc-dest(SRC)'` to solve
instability, which seems to be a highly wanted feature.
The feature is not completed, namely if `-d` overlaps with `-r`, things
could go wrong. A later patch will handle that case.
The feature is also gated by `experimental.rebase.multidest` config option
which is default off.
Differential Revision: https://phab.mercurial-scm.org/D469
This is a first basic visible usage of the changes tracking in the transaction.
We adds a new function computing the pre-existing changesets obsoleted by a
transaction and a transaction call back displaying this information.
Example output:
added 1 changesets with 1 changes to 1 files (+1 heads)
3 new obsolescence markers
obsoleted 1 changesets
The goal is to evolve the transaction summary into something bigger, gathering
existing output there and adding new useful one. This patch is a good first step
on this road. The new output is basic but give a user to the content of
tr.changes['obsmarkers'] and give an idea of the new options we haves. I expect
to revisit the message soon.
The caller recording the transaction summary should also be moved into a more
generic location but further refactoring is needed before it can happen.
It's now common that an old node gets replaced by zero or more new nodes,
that could happen with amend, rebase, histedit, etc. And it's a common
requirement to do bookmark movements, strip or obsolete nodes and even
moving working copy parent.
Previously, amend, rebase, history have their own logic doing the above.
This patch is an attempt to unify them and future code.
This enables new developers to be able to do "replace X with Y" thing
correctly, without any knowledge about bookmarks, strip or obsstore.
The next step will be migrating rebase to the new API, so it works inside a
transaction, and its code could be simplified.
This will allow us to map repo["ff..."] to workingctx. _partialmatch() will
be updated later. I tried "return wdirrev" in place of raising the exception,
but earlier exception seemed better.
It seemed silly to convert ctx.hex() back to binary to use node.hex/short(),
or to use [:12] instead of node.short() because ctx.node() could be None.
Eventually I want to change wctx.rev() and wctx.node() to return wdirrev and
wdirid respectively, but that's quite big API breakage and can't be achieved
without some compatibility wrappers.
This makes it slightly easier to sort basectx objects by key=scmutil.intrev.
We're most likely to have ctx objects where changectx/workingctx abstraction
is necessary, so this won't increase the abstraction overhead.
To ease migration from files with version numbers in their first lines,
we want simplekeyvaluefile to support a non-key-value first line. In this
way, old versions of Mercurial will read such files, discover a newer version
than the one they know how to handle and fail gracefully, rather than with
exception. Shelve's shelvestate file is an example.
