The world is starting to move on from SHA-1. A few commits ago, we
gained the ability to define certificate fingerprints using SHA-256
and SHA-512.
Let's start printing the SHA-256 fingerprint instead of the SHA-1
fingerprint to encourage people to pin with a more secure hashing
algorithm.
There is still a bit of work to be done around the fingerprint
messaging. This will be addressed in subsequent commits.
A short time ago, validatesocket() didn't know the reasons why
cert verification was disabled. Multiple code paths could lead
to cert verification being disabled. e.g. --insecure and lack
of loaded CAs.
With the recent refactorings to sslutil.py, we now know the reasons
behind security settings. This means we can recognize when the user
requested security be disabled (as opposed to being unable to provide
certificate verification due to lack of CAs).
This patch moves the check for certificate verification being disabled
and changes the wording to distinguish it from other states. The
warning message is purposefully more dangerous sounding in order
to help discourage people from disabling security outright.
We may want to add a URL or hint to this message. I'm going to wait
until additional changes to security defaults before committing to
something.
There are various tests for behavior when CA certs aren't loaded.
Previously, we would pass --insecure to disable loading of CA
certs. This has worked up to this point because the error message
for --insecure and no CAs loaded is the same. Upcoming commits will
change the error message for --insecure and will change behavior
when CAs aren't loaded.
This commit introduces the ability to disable loading of CA certs
by setting devel.disableloaddefaultcerts. This allows a testing
backdoor to disable loading of CA certs even if system/default
CA certs are available. The flag is purposefully not exposed to
end-users because there should not be a need for this in the wild:
certificate pinning and --insecure provide workarounds to disable
cert loading/validation.
Tests have been updated to use the new method. The variable used
to disable CA certs has been renamed because the method is not
OS X specific.
This patch effectively moves the ui.insecureconnections check to
_hostsettings(). After this patch, validatesocket() no longer uses the
ui instance for anything except writing messages.
This patch also enables us to introduce a per-host config option
for disabling certificate verification.
Error messages reference the config section defining the host
fingerprint. Now that we have multiple sections where this config
setting could live, we need to point the user at the appropriate
one.
We default to the new "hostsecurity" section. But we will still
refer them to the "hostfingerprint" section if a value is defined
there.
There are some corner cases where the messaging might be off. e.g.
they could define a SHA-1 fingerprint in both sections. IMO the
messaging needs a massive overhaul. I plan to do this as part
of future refactoring to security settings.
We introduce the [hostsecurity] config section. It holds per-host
security settings.
Currently, the section only contains a "fingerprints" option,
which behaves like [hostfingerprints] but supports specifying the
hashing algorithm.
There is still some follow-up work, such as changing some error
messages.
Currently, we only support defining host fingerprints with SHA-1.
A future patch will introduce support for defining fingerprints
using other hashing algorithms. In preparation for that, we
rewrite the fingerprint verification code to support multiple
fingerprints, namely SHA-256 and SHA-512 fingerprints.
We still only display the SHA-1 fingerprint. We'll have to revisit
this code once we support defining fingerprints with other hash
functions.
As part of this, I snuck in a change to use range() instead of
xrange() because xrange() isn't necessary for such small values.
The CA file processing code has been moved from _determinecertoptions
into _hostsettings(). As part of the move, the logic has been changed
slightly and the "cacerts" variable has been renamed to "cafile" to
match the argument used by SSLContext.load_verify_locations().
Since _determinecertoptions() no longer contains any meaningful
code, it has been removed.
_determinecertoptions() and _hostsettings() are redundant with each
other. _hostsettings() is used the flexible API we want.
We start the process of removing _determinecertoptions() by moving
some of the logic for the verify_mode value into _hostsettings().
As part of this, _determinecertoptions() now takes a settings dict
as its argument. This is technically API incompatible. But since
_determinecertoptions() came into existence a few days ago as part
of this release, I'm not flagging it as such.
This patch marks the beginning of a series that introduces a new,
more configurable, per-host security settings mechanism. Currently,
we have global settings (like web.cacerts and the --insecure argument).
We also have per-host settings via [hostfingerprints].
Global security settings are good for defaults, but they don't
provide the amount of control often wanted. For example, an
organization may want to require a particular CA is used for a
particular hostname.
[hostfingerprints] is nice. But it currently assumes SHA-1.
Furthermore, there is no obvious place to put additional per-host
settings.
Subsequent patches will be introducing new mechanisms for defining
security settings, some on a per-host basis. This commits starts
the transition to that world by introducing the _hostsettings
function. It takes a ui and hostname and returns a dict of security
settings. Currently, it limits itself to returning host fingerprint
info.
We foreshadow the future support of non-SHA1 hashing algorithms
for verifying the host fingerprint by making the "certfingerprints"
key a list of tuples instead of a list of hashes.
We add this dict to the hgstate property on the socket and use it
during socket validation for checking fingerprints. There should be
no change in behavior.
As the previous commit documented, sslkwargs() doesn't add any
value since its return is treated as a black box and proxied
to wrapsocket().
We formalize its uselessness by moving its logic into a
new, internal function and make sslkwargs() return an empty
dict.
The certificate arguments that sslkwargs specified have been
removed from wrapsocket() because they should no longer be
set.
Arguments to sslutil.wrapsocket() are partially determined by
calling sslutil.sslkwargs(). This function receives a ui and
a hostname and determines what settings, if any, need to be
applied when the socket is wrapped.
Both the ui and hostname are passed into wrapsocket(). The
other arguments to wrapsocket() provided by sslkwargs() (ca_certs
and cert_reqs) are not looked at or modified anywhere outside
of sslutil.py. So, sslkwargs() doesn't need to exist as a
separate public API called before wrapsocket().
This commit starts the process of removing external consumers of
sslkwargs() by removing the "ui" key/argument from its return.
All callers now pass the ui argument explicitly.
We are doing this check in both wrapsocket() and validatesocket().
The check was added to the validator in 8f98f4f9ff93 and the commit
message justifies the redundancy with a "might." The check in
wrapsocket() was added in 102733a3c3e1, which appears to be part of
the same series. I'm going to argue the redundancy isn't needed.
I choose to keep the check in wrapsocket() because it is working
around a bug in Python's wrap_socket() and I feel the check for
the bug should live next to the function call exhibiting the bug.
Now that the socket validator doesn't have any instance state,
we can make it a generic function.
The "validator" class has been converted into the "validatesocket"
function and all consumers have been updated.
Currently, we pass a hostname and ui to sslutil.wrap_socket()
then create a separate sslutil.validator instance also from
a hostname and ui. There is a 1:1 mapping between a wrapped
socket and a validator instance. This commit lays the groundwork
for making the validation function generic by storing the
hostname and ui instance in the state dict attached to the
socket instance and then using these variables in the
validator function.
Since the arguments to sslutil.validator.__init__ are no longer
used, we make them optional and make __init__ a no-op.
All callers now specify it. So we can require it.
Requiring the argument means SNI will always work if supported
by Python.
The main reason for this change is to store state on the socket
instance to make the validation function generic. This will be
evident in subsequent commits.
The previous patch stopped setting web.cacerts=! to indicate
--insecure.
That left user configs as the only source that could introduce
web.cacerts=!.
The practical impact of this patch is we no longer honor
web.cacerts=! in configs. Instead, we always treat web.cacerts
as a path. The patch is therefore technically BC. However,
since I don't believe web.cacerts=! is documented, it should be
safe to remove. 358b7bec186f (which introduced --insecure) has
no indication that web.cacerts=! is anything but an implementation
detail, reinforcing my belief it can be removed without major
debate.
Until now, sslkwargs may set web.cacerts=! to indicate
that system certs could not be found. This is really
obtuse because sslkwargs effectively sets state on a global
object which bypasses wrapsocket() and is later consulted
by validator.__call__. This is madness.
This patch introduces an attribute on the wrapped socket
instance indicating whether system CAs were loaded. We
can set this directly inside wrapsocket() because that
function knows everything that sslkwargs() does - and more.
With this attribute set on the socket, we refactor
validator.__call__ to use it.
Since we no longer have a need for setting web.cacerts=!
in sslkwargs, we remove that.
I think the new logic is much easier to understand and will
enable behavior to be changed more easily.
Right now, web.cacerts=! means one of two things:
1) Use of --insecure
2) No CAs could be found and were loaded (see sslkwargs)
This isn't very obvious and makes changing behavior of these
different scenarios independent of the other impossible.
This patch changes the validator code to explicit handle the
case of --insecure being used.
As the inline comment indicates, there is room to possibly change
messaging and logic here. For now, we are backwards compatible.
The end result of this function is the same. We now have a more
explicit return branch.
We still keep the old code looking at web.cacerts=! a few lines
below because we're still setting web.cacerts=! and need to react
to the variable. This will be removed in an upcoming patch.
The ways in which this code can interact with socket wrapping
and validation later are mind numbing. This patch helps make it
even more clear.
The end behavior should be identical.
Before, the return of _defaultcacerts() was 1 of 3 types. This was
difficult to read. Make it return a path or None.
We had to update hghave.py in the same patch because it was also
looking at this internal function. I wasted dozens of minutes
trying to figure out why tests were failing until I found the
code in hghave.py...
This main purpose of this patch is to make it clearer that fingerprint
pinning takes precedence over CA verification. This will make
subsequent refactoring to the validation code easier to read.
All callers appear to be passing the hostname. So this shouldn't
break anything. By specifying the hostname, more validation options
from the ssl module are available to us. Although this patch stops
short of using them.
Now that we have a fake SSLContext instance, we can unify the code
paths for wrapping sockets to always use the SSLContext APIs.
Because this is security code, I've retained the try..except to
make the diff easier to read. It will be removed in the next patch.
I took the liberty of updating the inline docs about supported
protocols and how the constants work because this stuff is important
and needs to be explicitly documented.
Python <2.7.9 doesn't have a ssl.SSLContext class. In this patch,
we implement the interface to the class so we can have a unified
code path for all supported versions of Python.
This is similar to the approach that urllib3 takes.
An upcoming patch will introduce a global SSLContext type so we
have a single function used to wrap sockets. Prepare for that by
introducing module level constants for disabling SSLv2 and SSLv3.
Certificate pinning via [hostfingerprints] is a useful security
feature. Currently, we only support one fingerprint per hostname.
This is simple but it fails in the real world:
* Switching certificates breaks clients until they change the
pinned certificate fingerprint. This incurs client downtime
and can require massive amounts of coordination to perform
certificate changes.
* Some servers operate with multiple certificates on the same
hostname.
This patch adds support for defining multiple certificate
fingerprints per host. This overcomes the deficiencies listed
above. I anticipate the primary use case of this feature will
be to define both the old and new certificate so a certificate
transition can occur with minimal interruption, so this scenario
has been called out in the help documentation.
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
This is necessary to communicate with third-party tools through command-server
channel. This requires SSLContext backported to Python 2.7.9+.
It doesn't look nice to pass ui by sslkwargs, but I think it is the only way
to do without touching various client codes including httpclient (aka http2).
ui is mandatory if certfile is specified, so it has no default value.
BTW, test-check-commit-hg.t complains that ssl_wrap_socket() has foo_bar
naming. Should I bulk-replace it to sslwrapsocket() ?
test-https.t was broken at d133034be253 if /usr/bin/pythonX.Y is used on
Mac OS X.
If python executable is not named as "python", run-tests.py creates a symlink
and hghave uses it. On the other hand, the installed hg executable knows the
real path to the system Python. Therefore, there was an inconsistency that
hghave said it was not an Apple python but hg knew it was.
This will make it easy to manage in-house CA certificates, which are often
used in corporate environment and installed into the Windows' certs store.
Unlike Apple python, the dummycert trick isn't necessary on Python 2.7.9.
The default web.cacerts will be set as follows:
environment web.cacerts behavior
------------- ----------- -----------------------------------------
Apple Python dummycert fall back to system's store
Python 2.7.8 '!' never use CA certs (show warning instead)
Python 2.7.9+ None load CA certs from system's store
The next patch will enable verification by using the system's CA store if
possible, which means we would have to distinguish None (=use default) from
'' (=--insecure). This smells bug-prone and provides no way to override
web.cacerts to forcibly use the system's store by --config argument.
This patch changes the meaning of web.cacerts as follows:
value behavior
------- ---------------------------------------
None/'' use default
'!' never use CA certs (set by --insecure)
<path> verify by the specified CA certificates
Values other than <path> are for internal use and therefore undocumented.
Asking for TLSv1 locks us out of TLSv1_2 etc. This is at least less
bad. Ideally we'd use ssl.create_default_context(), but that causes
more mayhem in the testsuite than I really want to deal with right
now.
We really just want to support the newest thing possible, so we may as
well consolidate that knowledge into this module. Right now this
doesn't change any behavior, but a future change will fix the defaults
for Python 2.7.9 so we can use slightly better defaults there (which
is the only place it's possible at the moment.)
SNI is a common way of sharing servers across multiple domains using separate
SSL certificates. As of Python 2.7.9 SSLContext has been backported from
Python 3. This patch changes sslutil's ssl_wrap_socket to use SSLContext and
take a server hostname as and argument. It also changes the url module to make
use of this argument.
The new code for 2.7.9 achieves it's task by attempting to get the SSLContext
object from the ssl module. If this fails the try/except goes back to what was
there before with the exception that the ssl_wrap_socket functions take a
server_hostname argument that doesn't get used. Assuming the SSLContext
exists, the arguments to wrap_socket at the module level are emulated on the
SSLContext. The SSLContext is initialized with the specified ssl_version. If
certfile is not None load_cert_chain is called with certfile and keyfile.
keyfile being None is not a problem, load_cert_chain will simply expect the
private key to be in the certificate file. verify_mode is set to cert_reqs. If
ca_certs is not None load_verify_locations is called with ca_certs as the
cafile. Finally the wrap_socket method of the SSLContext is called with the
socket and server hostname.
Finally, this fails test-check-commit-hg.t because the "new" function
ssl_wrap_socket has underscores in its names and underscores in its arguments.
All the underscore identifiers are taken from the other functions and as such
can't be changed to match naming conventions.
In light of the POODLE[0] attack on SSLv3, let's just drop the ability to
use anything older than TLSv1 entirely.
This only fixes the client side. Another commit will fix the server
side. There are still a few SSLv[23] constants hiding in httpclient,
but I'll fix those separately upstream and import them when we're not
in a code freeze.
0: http://googleonlinesecurity.blogspot.com/2014/10/this-poodle-bites-exploiting-ssl-30.html
The hack for using certificate store in addition to the provided CAs resides in
Apple's OpenSSL. Apple's own Pythons will use it, but other custom built
Pythons might use a custom built OpenSSL without that hack and will fail when
exposed to the dummy cacert introduced in ee8b7fe5e119.
There do not seem to be a simple way to check from Python if we are using a
patched OpenSSL or if it is an Apple OpenSSL.
Instead, check if the Python executable resides in /usr/bin/python* or in
/System/Library/Frameworks/Python.framework/ and assume that all Pythons found
there will be native Pythons using the patched OpenSSL.
Custom built Pythons will not get the benefit of using the CAs from the
certificate store.
This will give PKI-secure behaviour out of the box, without any configuration.
Setting web.cacerts to any value or empty will disable this trick.
This dummy cert trick only works on OS X 10.6+, but 10.5 had Python 2.5 which
didn't have certificate validation at all.
Prior to this change, we default to SSLv23, which is insecure because
it allows use of SSLv2. Unfortunately, there's no constant for OpenSSL
to let us use SSLv3 or TLS - we have to pick one or the other. We
expose a knob to revert to pre-TLS SSL for the user that has an
ancient server that lacks proper TLS support.
Python docs are a little unclear, but mpm reports reading the OpenSSL
source code shows that PROTOCOL_SSLv23 allows TLS whereas
PROTOCOL_SSLv3 does not.
We can't (easily) force SSL version on older Pythons, but on 2.6 and
later we can force SSLv3, which is safer and widely supported. This
also appears to work around a bug in IIS detailed in issue 3905.
Before this patch, "sslutil.validator" may returns successfully, even
if peer certificate is not verified because there is no information in
"[hostfingerprints]" and "[web] cacerts".
To prevent from sending authentication credential to untrustable SMTP
server, validation should be aborted if peer certificate is not
verified.
This patch introduces "strict" optional argument, and
"sslutil.validator" will abort if it is True and peer certificate is
not verified.
Some platforms, notably Plan 9 from Bell Labs are stuck on older
releases of Python. Due to restrictions in the platform, it is not
possible to backport the SSL library to the existing Python port.
This patch permits the UI to quiesce SSL verification warnings by
adding a configuration entry named reportoldssl to ui.
The previous workaround for correct handling of wrapping of failing connections
might be enough to prevent this from happening, but the check here makes this
function more robust.
This works around that ssl.wrap_socket silently skips ssl negotiation on
sockets that was connected but since then has been reset by the peer but not
yet closed at the Python level. That leaves the socket in a state where
.getpeercert() fails with an AttributeError on None. See
http://bugs.python.org/issue13721 .
A call to .cipher() is now used to verify that the wrapping really did succeed.
Otherwise it aborts with "ssl connection failed".
It is apparently possible to compile Python without SSL support or leave it out
when installing precompiled binaries.
Mercurial on such Pythons would crash if the user tried to use https. Now it
will be reported as "abort: Python SSL support not found" instead.
Any entries in subjectAltName would prevent fallback to using commonName, but
RFC 2818 says:
If a subjectAltName extension of type dNSName is present, that MUST
be used as the identity. Otherwise, the (most specific) Common Name
field in the Subject field of the certificate MUST be used.
We now only consider dNSNames in subjectAltName.
(dNSName is known as 'DNS' in OpenSSL/Python.)
Two imports were omitted in the restructure of the code creating
sslutil.py, socket and httplib are required when the 'ssl' module
cannot be imported, restoring these imports allows mercurial to run
on python2.4+2.5.
