Adding a possibility to configure error hint to be shown in the case of problems with SSH. Example of such hint can be "Please see http://company/internalwiki/ssh.html".
Test Plan:
- Ran hg pull with broken link and verified the output has no hint by default:
```
pulling from ssh://brokenrepository.com//repo
remote: ssh: Could not resolve hostname brokenrepository.com: Name or service not known
abort: no suitable response from remote hg!
```
- Run hg pull --config ui.ssherrorhint="Please see http://company/internalwiki/ssh.html":
```
pulling from ssh://brokenrepository.com//repo
remote: ssh: Could not resolve hostname brokenrepository.com: Name or service not known
abort: no suitable response from remote hg!
(Please see http://company/internalwiki/ssh.html)
```
Differential Revision: https://phab.mercurial-scm.org/D1431
There's been a persistent issue with flakiness on BSD systems (like OSX) where
the 'no suitable response from remote hg' message would sometimes not appear.
This was caused by one of the earlier calls failing with a "IOError: Broken
pipe". Catching those errors and printing the same message removes the
flakiness.
Differential Revision: https://phab.mercurial-scm.org/D687
The wirepeer class provides concrete implementations of peer interface
methods for calling wire protocol commands. It makes sense for this
class to inherit from the peer abstract base class. So we change
that.
Since httppeer and sshpeer have already been converted to the new
interface, peerrepository is no longer adding any value. So it has
been removed. httppeer and sshpeer have been updated to reflect the
loss of peerrepository and the inheritance of the abstract base
class in wirepeer.
The code changes in wirepeer are reordering of methods to group
by interface.
Some Python code in tests was updated to reflect changed APIs.
.. api::
peer.peerrepository has been removed. Use repository.peer abstract
base class to represent a peer repository.
Differential Revision: https://phab.mercurial-scm.org/D338
We need the same @property conversion of ui like we did for localpeer.
We renamed _capabilities() to capabilities() to satisfy the new
naming requirement.
However, since we're inheriting from wireproto.wirepeer which inherits
from peer.peerrepository and provides its own code accessing
_capabilities(), we need to keep the old alias around. This wonkiness
will disappear once wirepeer is cleaned up in subsequent commits.
We also implement methods for basepeer that are identical to the
defaults in peer.peerrepository in preparation for the removal of
peerrepository.
Differential Revision: https://phab.mercurial-scm.org/D336
Peer types are supposed to conform to a formal interface defined by
peer.peerrepository and wireproto.wirepeer. Every "public" attribute on
*peer types makes it harder to understand what attributes are part
of the interface and what are instance specific.
This commit converts a number of "public" instance attributes and
methods on sshpeer to internal so they can't be confused to be part of
the peer API.
The URL-related instance attributes were introduced in 904c418bea16
in 2005. AFAICT most of them aren't used and could potentially be
removed. But I kept them around anyway.
I also reorded some code to make things slightly easier to read.
.. api::
Rename attributes on sshpeer to reflect peer API
Differential Revision: https://phab.mercurial-scm.org/D331
Locking over the wire protocol and the "addchangegroup" wire
protocol command has been deprecated since f8e443eb02c9, which was
first part of Mercurial 0.9.1.
Support for handling these commands from sshserver was dropped in
93297d5f4df2 in 2015, effectively locking out pre 0.9.1 clients
from new servers.
However, client-side code for calling lock and addchangegroup is
still present in exchange.py and the various peer classes to
facilitate pushing to pre 0.9.1 servers.
The lock-based pushing mechanism is extremely brittle. 0.9.1 was
released in July 2006 and I highly doubt anyone is still running
such an ancient version of Mercurial on a server. I'm about to
refactor the peer API and I don't think it is worth keeping
support for this ancient protocol feature. So, this commit removes
client support for the lock-based pushing mechanism. This means
modern clients will no longer be able to push to pre 0.9.1 servers.
Differential Revision: https://phab.mercurial-scm.org/D264
This is done by a script [2] using RedBaron [1], a tool designed for doing
code refactoring. All "default" values are decided by the script and are
strongly consistent with the existing code.
There are 2 changes done manually to fix tests:
[warn] mercurial/exchange.py: experimental.bundle2-output-capture: default needs manual removal
[warn] mercurial/localrepo.py: experimental.hook-track-tags: default needs manual removal
Since RedBaron is not confident about how to indent things [2].
[1]: https://github.com/PyCQA/redbaron
[2]: https://github.com/PyCQA/redbaron/issues/100
[3]:
#!/usr/bin/env python
# codemod_configitems.py - codemod tool to fill configitems
#
# Copyright 2017 Facebook, Inc.
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
from __future__ import absolute_import, print_function
import os
import sys
import redbaron
def readpath(path):
with open(path) as f:
return f.read()
def writepath(path, content):
with open(path, 'w') as f:
f.write(content)
_configmethods = {'config', 'configbool', 'configint', 'configbytes',
'configlist', 'configdate'}
def extractstring(rnode):
"""get the string from a RedBaron string or call_argument node"""
while rnode.type != 'string':
rnode = rnode.value
return rnode.value[1:-1] # unquote, "'str'" -> "str"
def uiconfigitems(red):
"""match *.ui.config* pattern, yield (node, method, args, section, name)"""
for node in red.find_all('atomtrailers'):
entry = None
try:
obj = node[-3].value
method = node[-2].value
args = node[-1]
section = args[0].value
name = args[1].value
if (obj in ('ui', 'self') and method in _configmethods
and section.type == 'string' and name.type == 'string'):
entry = (node, method, args, extractstring(section),
extractstring(name))
except Exception:
pass
else:
if entry:
yield entry
def coreconfigitems(red):
"""match coreconfigitem(...) pattern, yield (node, args, section, name)"""
for node in red.find_all('atomtrailers'):
entry = None
try:
args = node[1]
section = args[0].value
name = args[1].value
if (node[0].value == 'coreconfigitem' and section.type == 'string'
and name.type == 'string'):
entry = (node, args, extractstring(section),
extractstring(name))
except Exception:
pass
else:
if entry:
yield entry
def registercoreconfig(cfgred, section, name, defaultrepr):
"""insert coreconfigitem to cfgred AST
section and name are plain string, defaultrepr is a string
"""
# find a place to insert the "coreconfigitem" item
entries = list(coreconfigitems(cfgred))
for node, args, nodesection, nodename in reversed(entries):
if (nodesection, nodename) < (section, name):
# insert after this entry
node.insert_after(
'coreconfigitem(%r, %r,\n'
' default=%s,\n'
')' % (section, name, defaultrepr))
return
def main(argv):
if not argv:
print('Usage: codemod_configitems.py FILES\n'
'For example, FILES could be "{hgext,mercurial}/*/**.py"')
dirname = os.path.dirname
reporoot = dirname(dirname(dirname(os.path.abspath(__file__))))
# register configitems to this destination
cfgpath = os.path.join(reporoot, 'mercurial', 'configitems.py')
cfgred = redbaron.RedBaron(readpath(cfgpath))
# state about what to do
registered = set((s, n) for n, a, s, n in coreconfigitems(cfgred))
toregister = {} # {(section, name): defaultrepr}
coreconfigs = set() # {(section, name)}, whether it's used in core
# first loop: scan all files before taking any action
for i, path in enumerate(argv):
print('(%d/%d) scanning %s' % (i + 1, len(argv), path))
iscore = ('mercurial' in path) and ('hgext' not in path)
red = redbaron.RedBaron(readpath(path))
# find all repo.ui.config* and ui.config* calls, and collect their
# section, name and default value information.
for node, method, args, section, name in uiconfigitems(red):
if section == 'web':
# [web] section has some weirdness, ignore them for now
continue
defaultrepr = None
key = (section, name)
if len(args) == 2:
if key in registered:
continue
if method == 'configlist':
defaultrepr = 'list'
elif method == 'configbool':
defaultrepr = 'False'
else:
defaultrepr = 'None'
elif len(args) >= 3 and (args[2].target is None or
args[2].target.value == 'default'):
# try to understand the "default" value
dnode = args[2].value
if dnode.type == 'name':
if dnode.value in {'None', 'True', 'False'}:
defaultrepr = dnode.value
elif dnode.type == 'string':
defaultrepr = repr(dnode.value[1:-1])
elif dnode.type in ('int', 'float'):
defaultrepr = dnode.value
# inconsistent default
if key in toregister and toregister[key] != defaultrepr:
defaultrepr = None
# interesting to rewrite
if key not in registered:
if defaultrepr is None:
print('[note] %s: %s.%s: unsupported default'
% (path, section, name))
registered.add(key) # skip checking it again
else:
toregister[key] = defaultrepr
if iscore:
coreconfigs.add(key)
# second loop: rewrite files given "toregister" result
for path in argv:
# reconstruct redbaron - trade CPU for memory
red = redbaron.RedBaron(readpath(path))
changed = False
for node, method, args, section, name in uiconfigitems(red):
key = (section, name)
defaultrepr = toregister.get(key)
if defaultrepr is None or key not in coreconfigs:
continue
if len(args) >= 3 and (args[2].target is None or
args[2].target.value == 'default'):
try:
del args[2]
changed = True
except Exception:
# redbaron fails to do the rewrite due to indentation
# see https://github.com/PyCQA/redbaron/issues/100
print('[warn] %s: %s.%s: default needs manual removal'
% (path, section, name))
if key not in registered:
print('registering %s.%s' % (section, name))
registercoreconfig(cfgred, section, name, defaultrepr)
registered.add(key)
if changed:
print('updating %s' % path)
writepath(path, red.dumps())
if toregister:
print('updating configitems.py')
writepath(cfgpath, cfgred.dumps())
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
Resolves test failures on FreeBSD, but I'm not happy about the fix.
A previous version of this also wrapped readline by putting the hack
in the _call method on doublepipe. That was confusing for readers and
wasn't necessary - just doing this on read() is sufficient to fix the
bugs I'm observing. We can always come back and do readline later if
needed.
Both wireproto.py and sshpeer.py had code for producing the value to
the "cmds" argument used by the "batch" command. This patch extracts
that code to a standalone function and uses it.
Unfortunately, the ssh and http implementations are slightly different
due to differences in their _callstream implementations, which
prevents ssh from behaving streamily. We should probably introduce a
new batch command that can stream results over ssh at some point in
the near future.
The streamy behavior of batch over http(s) is an enormous win for
remotefilelog over http: in my testing, it's saving about 40% on file
fetches with a cold cache against a server on localhost.
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
The output pipe does not have manually managed read buffer (actually, no read
anything). To also use the doublepipe for outgoing write (useful to consume
remote output when pushing large set of data) we need the doublepipe to work on
standard pipe too.
This class is responsible for ensuring we still process the server output
streamed through the ssh's 'stderr' pipe during the initial wait for other
protocol streams.
It currently only works on posix system because of its use of 'select.select'.
This is necessary to use non-blocking IO base on polling. Such polling is
needed to restore real time output with ssh peer.
Changeset 6a565336bae3 is talking about 5x regression on Mac OS X when playing
with this value. So we introduced our own buffering layer in previous
changesets. This seems to keep the regression away (we are even issuing much
less read).
We need this pipe to still be buffered when will switch to unbuffered pipe.
(switch motivated by the need of using polling to restore real time output from
ssh server). This is the only pipe that needs to be wrapped because this is the
one who do extensive usage of 'readline'. The stderr pipe of the process is
alway read in non blocking raw chunk, so it won't benefit from the
buffering.
When we'll be using the ssh's 'stderr' for realtime output successfully, it will
no longer be possible to use 'stderr' to carry the error message (because it
is already consumed by the real time output logic.
This feature is very rarely used (test says largefile only) and I think
breaking its output pattern is worth the benefit of having real time output with
ssh.
Reading all available data from a pipe has a platform-dependent
implementation.
This patch establishes platform.readpipe() by copying the
inline implementation in sshpeer.readerr(). The implementations
for POSIX and Windows are currently identical. The POSIX
implementation will be changed in a subsequent patch.
This implements the call needed by bundle2. The error handling is a bit flaky
right now, but bundle2 error handling in general is still flaky anyway. Bundle2
is still disabled by default, I do not expect this to be a problem.
We already have multiple call function for multiple return type. The
`_decompress` function is only used for http and seems like a layer violation.
We drop it in favor of a new call type dedicated to "stream that may be useful to
compress".
Previously, if another command was run with --verbose, and for whatever reason
that invoked sshpeer, we'd get a 'running ssh' message from sshpeer. This extra
line would interfere with that command's output and cause dumb parsers to
break.
For example, hg annotate can be run with --verbose to get full usernames. This,
combined with the third-party remotefilelog extension which can cause ssh
connections to be created, leads to an extra 'running ssh' line that breaks
most parsers.
This patch is (BC) because hg pull --verbose will no longer print out exactly
what ssh command it is running.
No tests are affected by this change.
When running 'hg pull --rebase', I was seeing this exception 100% of the
time as the python process was closing down:
Exception TypeError: TypeError("'NoneType' object is not callable",) in
<bound method Popen.__del__ of <subprocess.Popen object at 0x937c10>> ignored
By storing the subprocess on the sshpeer, the subprocess seems to clean up
correctly, and I no longer see the exception. I have no idea why this actually
works, but I get a 0% repro if I store the subprocess in self.subprocess,
and a 100% repro if I store None in self.subprocess.
Possibly related to issue 2240.
This change separates peer implementations from the repository implementation.
localpeer currently is a simple pass-through to localrepository, except for
legacy calls, which have already been removed from localpeer. This ensures that
the local client code only uses the most modern peer API when talking to local
repos.
Peers have a .local() method which returns either None or the underlying
localrepository (or descendant thereof). Repos have a .peer() method to return
a freshly constructed localpeer. The latter is used by hg.peer(), and also to
allow folks to pass either a peer or a repo to some generic helper methods.
We might want to get rid of .peer() eventually.
The only user of locallegacypeer is debugdiscovery, which uses it to pose as a
pre-setdiscovery client. But we decided to leave the old API defined in
locallegacypeer for clarity and maybe for other uses in the future.
It might be nice to actually define the peer API directly in peer.py as stub
methods. One problem there is, however, that localpeer implements
lock/addchangegroup, whereas the true remote peers implement unbundle.
It might be desireable to get rid of this distinction eventually.
