This is probably only used in the 'unbundle' command, but the code
ended up being cleaner to make it generic and treat *all* httppostargs
with a non-args request body as though they were file-like in
nature. It also means we get test coverage more or less for free. A
previous version of this change didn't use io.BytesIO, and it was a
lot more complicated.
This also fixes a server-side bug, so anyone using httppostargs should
update all of their servers to this revision or later *before* this
gets to their clients, otherwise servers will hang trying to over-read
the POST body.
Differential Revision: https://phab.mercurial-scm.org/D231
The wirepeer class provides concrete implementations of peer interface
methods for calling wire protocol commands. It makes sense for this
class to inherit from the peer abstract base class. So we change
that.
Since httppeer and sshpeer have already been converted to the new
interface, peerrepository is no longer adding any value. So it has
been removed. httppeer and sshpeer have been updated to reflect the
loss of peerrepository and the inheritance of the abstract base
class in wirepeer.
The code changes in wirepeer are reordering of methods to group
by interface.
Some Python code in tests was updated to reflect changed APIs.
.. api::
peer.peerrepository has been removed. Use repository.peer abstract
base class to represent a peer repository.
Differential Revision: https://phab.mercurial-scm.org/D338
Peer instances are supposed to conform to a well-defined API so
consumers can be agnostic about the underlying peer type.
To reinforce this, this commit renames a handful of instance
attributes on httpeer so they no longer have a "public" name.
Differential Revision: https://phab.mercurial-scm.org/D268
Locking over the wire protocol and the "addchangegroup" wire
protocol command has been deprecated since f8e443eb02c9, which was
first part of Mercurial 0.9.1.
Support for handling these commands from sshserver was dropped in
93297d5f4df2 in 2015, effectively locking out pre 0.9.1 clients
from new servers.
However, client-side code for calling lock and addchangegroup is
still present in exchange.py and the various peer classes to
facilitate pushing to pre 0.9.1 servers.
The lock-based pushing mechanism is extremely brittle. 0.9.1 was
released in July 2006 and I highly doubt anyone is still running
such an ancient version of Mercurial on a server. I'm about to
refactor the peer API and I don't think it is worth keeping
support for this ancient protocol feature. So, this commit removes
client support for the lock-based pushing mechanism. This means
modern clients will no longer be able to push to pre 0.9.1 servers.
Differential Revision: https://phab.mercurial-scm.org/D264
This is done by a script [2] using RedBaron [1], a tool designed for doing
code refactoring. All "default" values are decided by the script and are
strongly consistent with the existing code.
There are 2 changes done manually to fix tests:
[warn] mercurial/exchange.py: experimental.bundle2-output-capture: default needs manual removal
[warn] mercurial/localrepo.py: experimental.hook-track-tags: default needs manual removal
Since RedBaron is not confident about how to indent things [2].
[1]: https://github.com/PyCQA/redbaron
[2]: https://github.com/PyCQA/redbaron/issues/100
[3]:
#!/usr/bin/env python
# codemod_configitems.py - codemod tool to fill configitems
#
# Copyright 2017 Facebook, Inc.
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
from __future__ import absolute_import, print_function
import os
import sys
import redbaron
def readpath(path):
with open(path) as f:
return f.read()
def writepath(path, content):
with open(path, 'w') as f:
f.write(content)
_configmethods = {'config', 'configbool', 'configint', 'configbytes',
'configlist', 'configdate'}
def extractstring(rnode):
"""get the string from a RedBaron string or call_argument node"""
while rnode.type != 'string':
rnode = rnode.value
return rnode.value[1:-1] # unquote, "'str'" -> "str"
def uiconfigitems(red):
"""match *.ui.config* pattern, yield (node, method, args, section, name)"""
for node in red.find_all('atomtrailers'):
entry = None
try:
obj = node[-3].value
method = node[-2].value
args = node[-1]
section = args[0].value
name = args[1].value
if (obj in ('ui', 'self') and method in _configmethods
and section.type == 'string' and name.type == 'string'):
entry = (node, method, args, extractstring(section),
extractstring(name))
except Exception:
pass
else:
if entry:
yield entry
def coreconfigitems(red):
"""match coreconfigitem(...) pattern, yield (node, args, section, name)"""
for node in red.find_all('atomtrailers'):
entry = None
try:
args = node[1]
section = args[0].value
name = args[1].value
if (node[0].value == 'coreconfigitem' and section.type == 'string'
and name.type == 'string'):
entry = (node, args, extractstring(section),
extractstring(name))
except Exception:
pass
else:
if entry:
yield entry
def registercoreconfig(cfgred, section, name, defaultrepr):
"""insert coreconfigitem to cfgred AST
section and name are plain string, defaultrepr is a string
"""
# find a place to insert the "coreconfigitem" item
entries = list(coreconfigitems(cfgred))
for node, args, nodesection, nodename in reversed(entries):
if (nodesection, nodename) < (section, name):
# insert after this entry
node.insert_after(
'coreconfigitem(%r, %r,\n'
' default=%s,\n'
')' % (section, name, defaultrepr))
return
def main(argv):
if not argv:
print('Usage: codemod_configitems.py FILES\n'
'For example, FILES could be "{hgext,mercurial}/*/**.py"')
dirname = os.path.dirname
reporoot = dirname(dirname(dirname(os.path.abspath(__file__))))
# register configitems to this destination
cfgpath = os.path.join(reporoot, 'mercurial', 'configitems.py')
cfgred = redbaron.RedBaron(readpath(cfgpath))
# state about what to do
registered = set((s, n) for n, a, s, n in coreconfigitems(cfgred))
toregister = {} # {(section, name): defaultrepr}
coreconfigs = set() # {(section, name)}, whether it's used in core
# first loop: scan all files before taking any action
for i, path in enumerate(argv):
print('(%d/%d) scanning %s' % (i + 1, len(argv), path))
iscore = ('mercurial' in path) and ('hgext' not in path)
red = redbaron.RedBaron(readpath(path))
# find all repo.ui.config* and ui.config* calls, and collect their
# section, name and default value information.
for node, method, args, section, name in uiconfigitems(red):
if section == 'web':
# [web] section has some weirdness, ignore them for now
continue
defaultrepr = None
key = (section, name)
if len(args) == 2:
if key in registered:
continue
if method == 'configlist':
defaultrepr = 'list'
elif method == 'configbool':
defaultrepr = 'False'
else:
defaultrepr = 'None'
elif len(args) >= 3 and (args[2].target is None or
args[2].target.value == 'default'):
# try to understand the "default" value
dnode = args[2].value
if dnode.type == 'name':
if dnode.value in {'None', 'True', 'False'}:
defaultrepr = dnode.value
elif dnode.type == 'string':
defaultrepr = repr(dnode.value[1:-1])
elif dnode.type in ('int', 'float'):
defaultrepr = dnode.value
# inconsistent default
if key in toregister and toregister[key] != defaultrepr:
defaultrepr = None
# interesting to rewrite
if key not in registered:
if defaultrepr is None:
print('[note] %s: %s.%s: unsupported default'
% (path, section, name))
registered.add(key) # skip checking it again
else:
toregister[key] = defaultrepr
if iscore:
coreconfigs.add(key)
# second loop: rewrite files given "toregister" result
for path in argv:
# reconstruct redbaron - trade CPU for memory
red = redbaron.RedBaron(readpath(path))
changed = False
for node, method, args, section, name in uiconfigitems(red):
key = (section, name)
defaultrepr = toregister.get(key)
if defaultrepr is None or key not in coreconfigs:
continue
if len(args) >= 3 and (args[2].target is None or
args[2].target.value == 'default'):
try:
del args[2]
changed = True
except Exception:
# redbaron fails to do the rewrite due to indentation
# see https://github.com/PyCQA/redbaron/issues/100
print('[warn] %s: %s.%s: default needs manual removal'
% (path, section, name))
if key not in registered:
print('registering %s.%s' % (section, name))
registercoreconfig(cfgred, section, name, defaultrepr)
registered.add(key)
if changed:
print('updating %s' % path)
writepath(path, red.dumps())
if toregister:
print('updating configitems.py')
writepath(cfgpath, cfgred.dumps())
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
Another raising PeerTransportError for "incomplete response" in
httppeer.py uses this (changed) hint message. This unification reduces
cost of translation.
As part of writing test-http-bad-server.t, I noticed that some
requests include an empty Vary HTTP request header.
The Vary HTTP request header indicates which headers should be taken
into account when determining if a cached response can be used. It also
accepts the special value of "*".
The previous code unconditionally added a Vary header. This could lead
to an empty header value. While I don't believe this violates the HTTP
spec, this is weird and just wastes bytes. So this patch changes
behavior to only send a Vary header when it has a value.
Some low-level wire protocol byte reporting tests changed. In some
cases, the exact point of data termination changed. However, the
behavior being tested - that clients react when the connection is
closed in the middle of an HTTP request line or header - remains
unchanged.
Now that the response instance itself is wrapped with error
handling, we no longer need this code. This code became dead
with the previous patch because the added code catches
HTTPException and re-raises as something else.
There were a handful of places in the code where HTTPResponse.read()
was called with no explicit error handling or with inconsistent
error handling. In order to eliminate this class of bug, we globally
swap out HTTPResponse.read() with a unified error handler.
I initially attempted to fix all call sites. However, after
going down that rabbit hole, I figured it was best to just change
read() to do what we want. This appears to be a worthwhile
change, as the tests demonstrate many of our uncaught exceptions
go away.
To better represent this class of failure, we introduce a new
error type. The main benefit over IOError is it can hold a hint.
I'm receptive to tweaking its name or inheritance.
os.fdopen() does not accepts bytes as its second argument which represent the
mode in which the file is to be opened. This patch makes sure unicodes are
passed in py3 by using pycompat.sysstr().
Now that servers expose a capability indicating they support
application/mercurial-0.2 and compression, clients can key off
this to say they support responses that are compressed with
various compression formats.
After this commit, the HTTP wire protocol client now sends an
"X-HgProto-<N>" request header indicating its support for
"application/mercurial-0.2" media type and various compression
formats.
This commit also implements support for handling
"application/mercurial-0.2" responses. It simply reads the header
compression engine identifier then routes the remainder of the
response to the appropriate decompressor.
There were some test changes, but only to logging. That points to
an obvious gap in our test coverage. This will be addressed in a
subsequent commit once server support is in place (it is hard to
test without server support).
In preparation for adding another value to it in a subsequent patch.
While I was here, I added some empty lines because walls of text
are hard to read.
In preparation for supporting multiple compression formats on the
wire protocol, we need all users of the wire protocol to use
compression engine APIs.
This commit ports the HTTP wire protocol client to use the
compression engine API.
The code for handling the HTTPException is a bit hacky. Essentially,
HTTPException could be thrown by any read() from the socket. However,
as part of porting the API, we no longer have a generator wrapping
the socket and we don't have a single place where we can trap the
exception. We solve this by introducing a proxy class that intercepts
read() and converts the exception appropriately.
In the future, we could introduce a new compression engine API that
supports emitting a generator of decompressed chunks. This would
eliminate the need for the proxy class. As I said when I introduced
the decompressorreader() API, I'm not fond of it and would support
transitioning to something better. This can be done as a follow-up,
preferably once all the code is using the compression engine API and
we have a better idea of the API needs of all the consumers.
The current HTTP transport protocol only compresses certain command
responses and requires calls to that command to call
"_callcompressable," which zlib decompresses the response
transparently.
Upcoming changes will enable *any* response to be compressed with
varying compression formats. In order to handle this better, this
commit moves the decompression bits to the main function performing
the HTTP request. We introduce an underscore-prefixed argument to
denote this behavior so it doesn't conflict with a named argument
to a command.
Some errors could in some cases show unfortunate scary and confusing warnings
from the httppeer delstructors:
abort: nodename nor servname provided, or not known
Exception AttributeError: "'httpspeer' object has no attribute 'urlopener'" in <bound method httpspeer.__del__ of <mercurial.httppeer.httpspeer object at 0x106e1f5d0>> ignored```
To mute that, take 8bdb0bb8e209 to the next level and use getattr in __del__.
The httplib library is renamed to http.client in python 3. So the
import is conditionalized and a test is added in check-code to warn
to use util.httplib
This message has been overlooked by check-code, because it starts with
non-alphabet character ('(').
Making this message translatable seems reasonable, because exception
message below in same function is already translatable
- 'cannot create new http repository'
This is also a part of preparation for making "missing _() in ui
message" detection of check-code more exact.
writebundle() writes a bundle2 bundle or a plain changegroup1. Imagine
away the "2" in "bundle2.py" for a moment and this change should makes
sense. The bundle wraps the changegroup, so it makes sense that it
knows about it. Another sign that this is correct is that the delayed
import of bundle2 in changegroup goes away.
I'll leave it for another time to remove the "2" in "bundle2.py"
(alternatively, extract a new bundle.py from it).
narrowhg (for its narrow spec) and remotefilelog (for its large batch
requests) would like to be able to make requests with argument sets so
absurdly large that they blow out total request size limit on some
http servers. As a workaround, support stuffing args at the start
of the POST body.
We will probably want to leave this behavior off by default in servers
forever, because it makes the old "POSTs are only for writes"
assumption wrong, which might break some of the simpler authentication
configurations.
A future change will alter some of the arg-sending logic in a way that matters
for request body size. Centralizing the logic now will make later patches
easier to review.
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
Python 2.6 introduced the "except type as instance" syntax, replacing
the "except type, instance" syntax that came before. Python 3 dropped
support for the latter syntax. Since we no longer support Python 2.4 or
2.5, we have no need to continue supporting the "except type, instance".
This patch mass rewrites the exception syntax to be Python 2.6+ and
Python 3 compatible.
This patch was produced by running `2to3 -f except -w -n .`.
The authentication library my extension wants to use requires using a different
opener and a different request builder. This change pulls the call to
urllib2.Request out so that my extension can replace it just like it can
replace urlopener.
The next diff will add support for writing bundle2 files to writebundle, but
the bundle2 generator wants access to a ui object. This changes the signature
and callsites to pass one in.
This fixes several push tests in test-bundle2-exchange.t that were failing on
Windows with messages like the following:
$ hg -R main push http://localhost:$HGPORT2/ -r 32af7686d403 \
--bookmark book_32af
pushing to http://localhost:$HGPORT2/
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: 1 new obsolescence markers
updating bookmark book_32af
abort: The process cannot access the file because it is being used by another
process: 'C:\path\to\tmp\bundle.hg'
[255]
We already have multiple call function for multiple return type. The
`_decompress` function is only used for http and seems like a layer violation.
We drop it in favor of a new call type dedicated to "stream that may be useful to
compress".
Pre-0.6c hgweb used text/plain for protocol responses. This meant
that a web server could serve a static file and confuse a client into
generating a nasty traceback.
Now we insist that text/plain protocol responses not include a
Content-Length, which older hgweb didn't generate but will typically
be produced for static files.
When we don't get an hgweb protocol response, we dump the response to
the user for diagnostic purposes (it might be a cgitb message, for
instance).
But if we try to clone a bundle, we don't want to show the
entire bundle in the error message. Also, we don't want fetch the
full bundle multiple times during fallback. So we only fetch 1k here.
This change separates peer implementations from the repository implementation.
localpeer currently is a simple pass-through to localrepository, except for
legacy calls, which have already been removed from localpeer. This ensures that
the local client code only uses the most modern peer API when talking to local
repos.
Peers have a .local() method which returns either None or the underlying
localrepository (or descendant thereof). Repos have a .peer() method to return
a freshly constructed localpeer. The latter is used by hg.peer(), and also to
allow folks to pass either a peer or a repo to some generic helper methods.
We might want to get rid of .peer() eventually.
The only user of locallegacypeer is debugdiscovery, which uses it to pose as a
pre-setdiscovery client. But we decided to leave the old API defined in
locallegacypeer for clarity and maybe for other uses in the future.
It might be nice to actually define the peer API directly in peer.py as stub
methods. One problem there is, however, that localpeer implements
lock/addchangegroup, whereas the true remote peers implement unbundle.
It might be desireable to get rid of this distinction eventually.
