This effectively backs out changeset 60b56b3206cc.
The http library behind ui.http2=true isn't specifying the hostname.
It is the day before the expected 3.8 release and we don't want to ship
a regression.
I'll try to restore this requirement in the 3.9 release cycle as part
of planned improvements to Mercurial's SSL/TLS interactions.
This main purpose of this patch is to make it clearer that fingerprint
pinning takes precedence over CA verification. This will make
subsequent refactoring to the validation code easier to read.
All callers appear to be passing the hostname. So this shouldn't
break anything. By specifying the hostname, more validation options
from the ssl module are available to us. Although this patch stops
short of using them.
Now that we have a fake SSLContext instance, we can unify the code
paths for wrapping sockets to always use the SSLContext APIs.
Because this is security code, I've retained the try..except to
make the diff easier to read. It will be removed in the next patch.
I took the liberty of updating the inline docs about supported
protocols and how the constants work because this stuff is important
and needs to be explicitly documented.
Python <2.7.9 doesn't have a ssl.SSLContext class. In this patch,
we implement the interface to the class so we can have a unified
code path for all supported versions of Python.
This is similar to the approach that urllib3 takes.
An upcoming patch will introduce a global SSLContext type so we
have a single function used to wrap sockets. Prepare for that by
introducing module level constants for disabling SSLv2 and SSLv3.
Certificate pinning via [hostfingerprints] is a useful security
feature. Currently, we only support one fingerprint per hostname.
This is simple but it fails in the real world:
* Switching certificates breaks clients until they change the
pinned certificate fingerprint. This incurs client downtime
and can require massive amounts of coordination to perform
certificate changes.
* Some servers operate with multiple certificates on the same
hostname.
This patch adds support for defining multiple certificate
fingerprints per host. This overcomes the deficiencies listed
above. I anticipate the primary use case of this feature will
be to define both the old and new certificate so a certificate
transition can occur with minimal interruption, so this scenario
has been called out in the help documentation.
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
This is necessary to communicate with third-party tools through command-server
channel. This requires SSLContext backported to Python 2.7.9+.
It doesn't look nice to pass ui by sslkwargs, but I think it is the only way
to do without touching various client codes including httpclient (aka http2).
ui is mandatory if certfile is specified, so it has no default value.
BTW, test-check-commit-hg.t complains that ssl_wrap_socket() has foo_bar
naming. Should I bulk-replace it to sslwrapsocket() ?
test-https.t was broken at d133034be253 if /usr/bin/pythonX.Y is used on
Mac OS X.
If python executable is not named as "python", run-tests.py creates a symlink
and hghave uses it. On the other hand, the installed hg executable knows the
real path to the system Python. Therefore, there was an inconsistency that
hghave said it was not an Apple python but hg knew it was.
This will make it easy to manage in-house CA certificates, which are often
used in corporate environment and installed into the Windows' certs store.
Unlike Apple python, the dummycert trick isn't necessary on Python 2.7.9.
The default web.cacerts will be set as follows:
environment web.cacerts behavior
------------- ----------- -----------------------------------------
Apple Python dummycert fall back to system's store
Python 2.7.8 '!' never use CA certs (show warning instead)
Python 2.7.9+ None load CA certs from system's store
The next patch will enable verification by using the system's CA store if
possible, which means we would have to distinguish None (=use default) from
'' (=--insecure). This smells bug-prone and provides no way to override
web.cacerts to forcibly use the system's store by --config argument.
This patch changes the meaning of web.cacerts as follows:
value behavior
------- ---------------------------------------
None/'' use default
'!' never use CA certs (set by --insecure)
<path> verify by the specified CA certificates
Values other than <path> are for internal use and therefore undocumented.
Asking for TLSv1 locks us out of TLSv1_2 etc. This is at least less
bad. Ideally we'd use ssl.create_default_context(), but that causes
more mayhem in the testsuite than I really want to deal with right
now.
We really just want to support the newest thing possible, so we may as
well consolidate that knowledge into this module. Right now this
doesn't change any behavior, but a future change will fix the defaults
for Python 2.7.9 so we can use slightly better defaults there (which
is the only place it's possible at the moment.)
SNI is a common way of sharing servers across multiple domains using separate
SSL certificates. As of Python 2.7.9 SSLContext has been backported from
Python 3. This patch changes sslutil's ssl_wrap_socket to use SSLContext and
take a server hostname as and argument. It also changes the url module to make
use of this argument.
The new code for 2.7.9 achieves it's task by attempting to get the SSLContext
object from the ssl module. If this fails the try/except goes back to what was
there before with the exception that the ssl_wrap_socket functions take a
server_hostname argument that doesn't get used. Assuming the SSLContext
exists, the arguments to wrap_socket at the module level are emulated on the
SSLContext. The SSLContext is initialized with the specified ssl_version. If
certfile is not None load_cert_chain is called with certfile and keyfile.
keyfile being None is not a problem, load_cert_chain will simply expect the
private key to be in the certificate file. verify_mode is set to cert_reqs. If
ca_certs is not None load_verify_locations is called with ca_certs as the
cafile. Finally the wrap_socket method of the SSLContext is called with the
socket and server hostname.
Finally, this fails test-check-commit-hg.t because the "new" function
ssl_wrap_socket has underscores in its names and underscores in its arguments.
All the underscore identifiers are taken from the other functions and as such
can't be changed to match naming conventions.
In light of the POODLE[0] attack on SSLv3, let's just drop the ability to
use anything older than TLSv1 entirely.
This only fixes the client side. Another commit will fix the server
side. There are still a few SSLv[23] constants hiding in httpclient,
but I'll fix those separately upstream and import them when we're not
in a code freeze.
0: http://googleonlinesecurity.blogspot.com/2014/10/this-poodle-bites-exploiting-ssl-30.html
The hack for using certificate store in addition to the provided CAs resides in
Apple's OpenSSL. Apple's own Pythons will use it, but other custom built
Pythons might use a custom built OpenSSL without that hack and will fail when
exposed to the dummy cacert introduced in ee8b7fe5e119.
There do not seem to be a simple way to check from Python if we are using a
patched OpenSSL or if it is an Apple OpenSSL.
Instead, check if the Python executable resides in /usr/bin/python* or in
/System/Library/Frameworks/Python.framework/ and assume that all Pythons found
there will be native Pythons using the patched OpenSSL.
Custom built Pythons will not get the benefit of using the CAs from the
certificate store.
This will give PKI-secure behaviour out of the box, without any configuration.
Setting web.cacerts to any value or empty will disable this trick.
This dummy cert trick only works on OS X 10.6+, but 10.5 had Python 2.5 which
didn't have certificate validation at all.
Prior to this change, we default to SSLv23, which is insecure because
it allows use of SSLv2. Unfortunately, there's no constant for OpenSSL
to let us use SSLv3 or TLS - we have to pick one or the other. We
expose a knob to revert to pre-TLS SSL for the user that has an
ancient server that lacks proper TLS support.
Python docs are a little unclear, but mpm reports reading the OpenSSL
source code shows that PROTOCOL_SSLv23 allows TLS whereas
PROTOCOL_SSLv3 does not.
We can't (easily) force SSL version on older Pythons, but on 2.6 and
later we can force SSLv3, which is safer and widely supported. This
also appears to work around a bug in IIS detailed in issue 3905.
Before this patch, "sslutil.validator" may returns successfully, even
if peer certificate is not verified because there is no information in
"[hostfingerprints]" and "[web] cacerts".
To prevent from sending authentication credential to untrustable SMTP
server, validation should be aborted if peer certificate is not
verified.
This patch introduces "strict" optional argument, and
"sslutil.validator" will abort if it is True and peer certificate is
not verified.
Some platforms, notably Plan 9 from Bell Labs are stuck on older
releases of Python. Due to restrictions in the platform, it is not
possible to backport the SSL library to the existing Python port.
This patch permits the UI to quiesce SSL verification warnings by
adding a configuration entry named reportoldssl to ui.
The previous workaround for correct handling of wrapping of failing connections
might be enough to prevent this from happening, but the check here makes this
function more robust.
This works around that ssl.wrap_socket silently skips ssl negotiation on
sockets that was connected but since then has been reset by the peer but not
yet closed at the Python level. That leaves the socket in a state where
.getpeercert() fails with an AttributeError on None. See
http://bugs.python.org/issue13721 .
A call to .cipher() is now used to verify that the wrapping really did succeed.
Otherwise it aborts with "ssl connection failed".
