pycompat.getenv returns os.getenvb on py3 which is not available on Windows.
This patch replaces them with encoding.environ.get and checks to ensure no
new instances of os.getenv or os.setenv are introduced.
Now that servers expose a capability indicating they support
application/mercurial-0.2 and compression, clients can key off
this to say they support responses that are compressed with
various compression formats.
After this commit, the HTTP wire protocol client now sends an
"X-HgProto-<N>" request header indicating its support for
"application/mercurial-0.2" media type and various compression
formats.
This commit also implements support for handling
"application/mercurial-0.2" responses. It simply reads the header
compression engine identifier then routes the remainder of the
response to the appropriate decompressor.
There were some test changes, but only to logging. That points to
an obvious gap in our test coverage. This will be addressed in a
subsequent commit once server support is in place (it is hard to
test without server support).
os.getenv deals with unicodes on Python 3, so we have pycompat.osgetenv to
deal with bytes. This patch replaces occurrences on os.getenv with
pycompat.osgetenv
Currently we delete proxy environment variables if ui.config contains proxy
values. This is unnecessary because urllib2.ProxyHandler class only reads proxy
from environment it is initialised by None. But url.py never passes None,
so there is no point urllib2 will take environment variables in account.
This also prevents deleting environment variables which is not safe.
This code was introduced while resolving Bug 2451 even it is in one of comments
(sixth one) on bug that we can safely remove this part.
Link to bug : https://bz.mercurial-scm.org/show_bug.cgi?id=2451
Some draconian IT setups lock accounts after a small number of incorrect
password attempts. Mercurial's implementation of the urllib2 authentication was
causing 5 retry attempts with the same credentials, without prompting the user.
The code was attempting to check whether the authorization token had changed,
but unfortunately was reading the misleading 'headers' member of the request
instead of using the 'get_header' accessor.
Modelled on fix for Python issue 8797:
https://bugs.python.org/issue8797https://hg.python.org/cpython/rev/30e8a8f22a2a
It's been broken since 799db3fe9866, which made ui argument mandatory. I've
tried several combinations of HTTP/HTTPS proxying on old/new Python versions,
but I couldn't figure out how to reach this code path. Also, wrapping HTTP
connection by SSLSocket seems wrong. My understanding is that self.realhostport
is set by _generic_start_transaction() if HTTPS connection is tunneled.
This patch removes proxy tunneling from httpconnection.connect() assuming
that it was dead code from the beginning. Note that HTTPS over tunneling
should be handled by httpsconnection class.
As a server operator, I've always wanted to know what Mercurial
version clients are running so I can track version adoption and
make informed decisions about which versions of Mercurial to
support in extensions. Unfortunately, there is no easy way to discern
this today: the best you can do is look for high-level feature usage
(e.g. bundle2) or sniff capabilities from bundle2 commands. And these
things aren't changed frequently enough to tell you anything that
interesting.
Nearly every piece of software talking HTTP sends its version in
the user agent. This includes web browsers, curl, and even Git.
This patch adds the distribution name and version to the user-agent
HTTP request header. We choose "Mercurial" for the distribution
name because that seems appropriate. The version string comes
from __version__.
The value is inside parenthesis for a few reasons:
* The version *may* contain spaces
* Alternate forms like "Mercurial/<version>" imply structure and
since the user agent should not be used by servers for protocol
or feature negotiation/detection, we don't want to even give the
illusion that the value should be parsed. A free form field is
the most hostile to parsing.
Flagging the patch as BC so it shows up in release notes. This
change should be backwards compatible. But I wouldn't be surprised if
a server somewhere is filtering on the exact old user agent string. So
I want to make noise about this change.
The httplib library is renamed to http.client in python 3. So the
import is conditionalized and a test is added in check-code to warn
to use util.httplib
Before this commit url.opener overwritten stored password
for connection with given url/user even when
new password for given connection was not filled. This
commit makes opener overwrites saved authentication only
when it contains password.
This makes http password database stored in ui object.
It allows reusing authentication information when we
use this database for creating password manager for
the new connection.
So far password manager was keeping authentication information so opening
new connection and creating new password manager made all saved authentication
information lost.
This commit separates password manager and password database to make it
possible to reuse saved authentication information.
This commit violates code checker because it adds add_password method (name
with underscore) to passwordmgr object to provide method required by urllib2.
Arguments to sslutil.wrapsocket() are partially determined by
calling sslutil.sslkwargs(). This function receives a ui and
a hostname and determines what settings, if any, need to be
applied when the socket is wrapped.
Both the ui and hostname are passed into wrapsocket(). The
other arguments to wrapsocket() provided by sslkwargs() (ca_certs
and cert_reqs) are not looked at or modified anywhere outside
of sslutil.py. So, sslkwargs() doesn't need to exist as a
separate public API called before wrapsocket().
This commit starts the process of removing external consumers of
sslkwargs() by removing the "ui" key/argument from its return.
All callers now pass the ui argument explicitly.
Now that the socket validator doesn't have any instance state,
we can make it a generic function.
The "validator" class has been converted into the "validatesocket"
function and all consumers have been updated.
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
Python 2.6 introduced the "except type as instance" syntax, replacing
the "except type, instance" syntax that came before. Python 3 dropped
support for the latter syntax. Since we no longer support Python 2.4 or
2.5, we have no need to continue supporting the "except type, instance".
This patch mass rewrites the exception syntax to be Python 2.6+ and
Python 3 compatible.
This patch was produced by running `2to3 -f except -w -n .`.
This is necessary to communicate with third-party tools through command-server
channel. This requires SSLContext backported to Python 2.7.9+.
It doesn't look nice to pass ui by sslkwargs, but I think it is the only way
to do without touching various client codes including httpclient (aka http2).
ui is mandatory if certfile is specified, so it has no default value.
BTW, test-check-commit-hg.t complains that ssl_wrap_socket() has foo_bar
naming. Should I bulk-replace it to sslwrapsocket() ?
HTTPSConnection of Python 2.7.9 creates SSLContext in __init__, which involves
a password prompt for decrypting the private key. This means the password was
asked twice, one for unused SSLContext, and next for our ssl function.
Because our httpsconnection replaces connect() method at all, we can simply
drop httplib.HTTPSConnection. Instead, class and instance attributes are copied
from it.
HTTPSConnection of Python 2.7.8 and 2.6.9 seem to have no such problem.
https://hg.python.org/cpython/file/v2.7.9/Lib/httplib.py#l1183
SNI is a common way of sharing servers across multiple domains using separate
SSL certificates. As of Python 2.7.9 SSLContext has been backported from
Python 3. This patch changes sslutil's ssl_wrap_socket to use SSLContext and
take a server hostname as and argument. It also changes the url module to make
use of this argument.
The new code for 2.7.9 achieves it's task by attempting to get the SSLContext
object from the ssl module. If this fails the try/except goes back to what was
there before with the exception that the ssl_wrap_socket functions take a
server_hostname argument that doesn't get used. Assuming the SSLContext
exists, the arguments to wrap_socket at the module level are emulated on the
SSLContext. The SSLContext is initialized with the specified ssl_version. If
certfile is not None load_cert_chain is called with certfile and keyfile.
keyfile being None is not a problem, load_cert_chain will simply expect the
private key to be in the certificate file. verify_mode is set to cert_reqs. If
ca_certs is not None load_verify_locations is called with ca_certs as the
cafile. Finally the wrap_socket method of the SSLContext is called with the
socket and server hostname.
Finally, this fails test-check-commit-hg.t because the "new" function
ssl_wrap_socket has underscores in its names and underscores in its arguments.
All the underscore identifiers are taken from the other functions and as such
can't be changed to match naming conventions.
With Python 2.7.7rc1, "hg pull" through HTTP CONNECT tunnel fails due to the
removal of _set_hostport [1].
...
File "mercurial/url.py", line 372, in https_open
return self.do_open(self._makeconnection, req)
...
File "mercurial/url.py", line 342, in connect
_generic_proxytunnel(self)
File "mercurial/url.py", line 228, in _generic_proxytunnel
self._set_hostport(self.host, self.port)
AttributeError: httpsconnection instance has no attribute '_set_hostport'
self._set_hostport(self.host, self.port) should be noop and can be removed
because:
- _set_hostport() [2] was the function to parse "host:port" string and
set them to self.host and self.port,
- and (self.host, self.port) pair should be valid since connect() is called
prior to _generic_proxytunnel().
[1]: http://hg.python.org/cpython/rev/568041fd8090
[2]: http://hg.python.org/cpython/file/3a1db0d2747e/Lib/httplib.py#l721
When users are using a revset they can get multiple password prompts.
This prompts have no extra information about which password is being requested
so I added the authuri to the prompt to make it recognizable.
As in:
$ hg log -r "outgoing('https://bitbucket.org/mg/test') -
outgoing('https://bitbucket.org/nesneros/test')"
http authorization required
realm: Bitbucket.org HTTP
user: interrupted!
I changed it to describe the url when prompting for password.
As in:
$ hg log -r "outgoing('https://bitbucket.org/mg/test') -
outgoing('https://bitbucket.org/nesneros/test')"
http authorization required for https://bitbucket.org/mg/test
realm: Bitbucket.org HTTP
user: interrupted!
This is an attempt to fix issue 2451 and its duplicates (2599 and 2949, AFAIK).
Its main idea is that it is only necessary to clean the proxy environment
variables *when* http_proxy is set in the config file (since it takes
precedence over the environment variables). Otherwise, hg shouldn't bother with
them, since they will most likely be used to reach the server.
urllib2 never handles URIs with credentials, we have to extract them and store
them in the password manager before handing the stripped URI. Half of the
changes deducing the username from the URI in f7ae45a69fcd were incorrect.
Instead, we retrieve the username from the password manager before passing to
readauthforuri().
test-hgweb-auth.py was passing because the test itself was flawed: it was
passing URIs with credentials to find_password(), which never happens.
The [auth] section was ignored when handling URLs like:
http://user@example.com/foo
Instead, we look in [auth] for an entry matching the URL and supplied user
name. Entries without username can match URL with a username. Prefix length
ties are resolved in favor of entries matching the username. With:
foo.prefix = http://example.org
foo.username = user
foo.password = password
bar.prefix = http://example.org/bar
and the input URL:
http://user@example.org/bar
the 'bar' entry will be selected because of prefix length, therefore prompting
for a password. This behaviour ensure that entries selection is consistent when
looking for credentials or for certificates, and that certificates can be
picked even if their entries do no define usernames while the URL does.
Additionally, entries without a username matched against a username are
returned as if they did have requested username set to avoid prompting again
for a username if the password is not set.
v2: reparse the URL in readauthforuri() to handle HTTP and HTTPS similarly.
v3: allow unset usernames to match URL usernames to pick certificates. Resolve
prefix length ties in favor of entries with usernames.
The new http library is wired in via an extra module
(httpconnection.py), as it requires similar but different plumbing to
connect the library to Mercurial's internals and urllib2. Eventualy we
should be able to remove all of keepalive.py and its associated tangle
in url.py and replace it all with the code in httpconnection.py.
To use the new library, set 'ui.usehttp2' to true. The underlying http
library uses the logging module liberally, so if things break you can
use 'ui.http2debuglevel' to set the log level to INFO or DEBUG to get
that logging information (for example, ui.http2debuglevel=info.)
The introduction of the new URL parsing code has created a startup
time regression. This is mainly due to the use of url.hasscheme() in
the ui class. It ends up importing many libraries that the url module
requires.
This fix helps marginally, but if we can get rid of the urllib import
in the URL parser all together, startup time will go back to normal.
perfstartup time before the URL refactoring (707e4b1e8064):
! wall 0.050692 comb 0.000000 user 0.000000 sys 0.000000 (best of 100)
current startup time (9ad1dce9e7f4):
! wall 0.070685 comb 0.000000 user 0.000000 sys 0.000000 (best of 100)
after this change:
! wall 0.064667 comb 0.000000 user 0.000000 sys 0.000000 (best of 100)
While the URL parser is very forgiving about what characters are
allowed in each component, it's useful to be strict about the scheme
so we don't accidentally interpret local paths with colons as URLs.
This restricts schemes to containing alphanumeric characters, dashes,
pluses, and dots (as specified in RFC 2396).
