Issue 5639: Support TLS SNI extension in ssl module (original) (raw)
Created on 2009-04-01 06:38 by pdp, last changed 2022-04-11 14:56 by admin. This issue is now closed.
Messages (42)
Author: Phil Pennock (pdp)
Date: 2009-04-01 06:38
With TLS it is possible to have the client use an extension (defined in RFC 4366, and RFC 3546 before that) to indicate to the server which hostname it believes it is talking to. The server can then choose TLS certificates accordingly. This makes virtual-hosting possible. Most modern GUI web-browsers support making use of this extension, Server Name Indication (SNI).
OpenSSL 0.9.8f onwards have optional support for this; OpenSSL needs to have been built with "enable-tlsext" in EXTRACONFIGURE. If that is not present, then there's a guard macro defined to say it's absent.
This patch, against Python 2.6.1, adds to the standard ssl module the ability to set the extension, using server_hostname as a arg in relevant places. This is only set for client connections and will silently be ignored if the OpenSSL library does not support it.
I have tested this on FreeBSD 7.0/amd64 with OpenSSL 0.9.8k when talking to Apache 2.2.x with the SNI patches from https://sni.velox.ch/. Below is my simple test program, to dump raw HTTP results back. With this, I can connect to various local https vhosts and get the correct content back.
I am not a Python core dev and not too enthusiastic at the thought of grabbing latest svn to port this across; I hope that it's still of use.
============= import socket import ssl import sys
def dump_https_page(hostname, uri='/'):
sock = socket.socket(socket.AF_INET) s = ssl.SSLSocket(sock=sock, ca_certs='/etc/ssl/certs', server_hostname=hostname) print 'have socket' s.connect((hostname, 443)) print 'connected'
print >>s, 'GET %s HTTP/1.0\r\nHost: %s\r\nConnection: close\r\n\r\n' % ( uri, hostname),
t = s.read() while t: print t, t = s.read()
if name == 'main': for x in sys.argv[1:]: dump_https_page(hostname=x)
Author: Phil Pennock (pdp)
Date: 2009-04-01 20:30
Note: this previous work is client-side only, as noted in the body of the report. I'll look into what's needed for clean server-side support too.
Author: Daniel Black (grooverdan) *
Date: 2009-08-31 05:32
patch against TRUNK (2.7) with self tests and doco. Essentially the same code as pdp with a SSLv2 check before using the SNI extension.
Contains some spacing cleanups that where highlighted by vim.
Author: Daniel Black (grooverdan) *
Date: 2009-08-31 05:33
py3k version
Author: Daniel Black (grooverdan) *
Date: 2009-08-31 05:43
current self tests cannot fully test the existence of the SNI extension as there is no server side support.
This client script run with argument sni.velox.ch will show the "Great! Your client ....its ClientHello: sni.velox.ch" on the output.
Author: Daniel Black (grooverdan) *
Date: 2009-08-31 06:04
The small deficiency with these patches is that the specified server_hostname is almost always the hostname that is used in the socket pair of connect. Is it appropriate to grab the hostname value and use it in the SNI extension header?
Author: Phil Pennock (pdp)
Date: 2009-09-01 02:59
(Sorry for dropping this, lost available time)
I see your point. OTOH, use of SNI needs to be something that can be disabled and people need to be able to connect to host A while supplying host B, not necessarily using IP addresses for the specificity. Use- case example: someone has a service "www" hosted on ["www-1", "www-2", "www-3"]. They have an SSL certificate for "www" and they want to have a health-checker which probes for "working service, all certs valid and not about to expire".
Unless s.connect() gains a keep_original_hostname=False option (?), this is hard to do.
Then there's the principle of least surprise -- while it would be nice to get SNI working automatically for everyone, it's still plausible that amongst the various TLS servers out there are some which break horribly for SNI and upgrading Python shouldn't break the tools in use.
So I tend towards favouring "make use of the newer, less well tested, protocol feature something that has to be explicitly enabled", even if it adds a line to boilerplate -- if SNI ever takes over the world (as Host: headers in HTTP have) then it can be defaulted on in future perhaps?
In which case, if the default is to change, the API should be sorted now, so perhaps connect() should take an override_server_hostname=False flag, which will make it pass the connect() hostname parameter instead of self.server_hostname in the call to _ssl.sslwrap()? With checking for an IP address?
Author: Daniel Black (grooverdan) *
Date: 2009-09-04 06:37
Hey Phil,
(Sorry for dropping this, lost available time) know the feeling :-(
use of SNI needs to be something that can be disabled maybe. See small rational below:
and people need to be able to connect to host A while supplying host B This seems to be a fringe case for usage and I seem to thing adding this would overly complicate the API. When testing hosts you would have all the names in DNS I'd assume.
Unless s.connect() gains a keep_original_hostname=False option (?), this is hard to do. Is this starting to look too complicated?
Options for client side SNI:
- wrapssl() - defaults to SNI enabled on SSL2/TLS1 connections (compile time/module level/object variable disable if really needed?)
- wrapssl(server_hostname=True/False) - enable SNI using the connect hostname (if domainname and not IP/socket)
- wrapssl(server_hostname=True/False/String) - True - enable SNI as above, False/None- don't use SNI or use hostname if a String.
- wrapssl(server-hostname=String)
- connect() should an override_server_hostname=False more?
Then there's the principle of least surprise -- while it would be nice to get SNI working automatically for everyone, it's still plausible that amongst the various TLS servers out there are some which break horribly for SNI and upgrading Python shouldn't break the tools in use.
Small rational to enable SNI by default:
- SNI probably won't break too much stuff. It was only in July 2009 that Apache-2.2.12 got officially released with proper SNI support. Patches existed before then however deployment was limited. mod_gnutls did implemented this earlier however I never thought this was widely used. Vista IE7 got SNI support in ~2005 (http://blogs.msdn.com/ie/archive/2005/10/22/483795.aspx) FF got SNI support in the 2.0 release (October 24, 2006, wikipedia) (https://bugzilla.mozilla.org/show_bug.cgi?id=116169#c26) The OpenSSL server side API for SNI required the registration of a callback to receive the SNI name. It is possible that there are some faulty implementations in this part of the server code. The default case that a TLS server doesn't account for SNI is very safe as it won't ever get a callback for it. The existence of wide spread client side support with limited server support hasn't broken the web.
So I tend towards favouring "make use of the newer, less well tested, protocol feature something that has to be explicitly enabled", even if it adds a line to boilerplate -- if SNI ever takes over the world (as Host: headers in HTTP have) then it can be defaulted on in future perhaps?
I think with p3k there is an opportunity to put something in that may break stuff. The release noted didn't guarantee stuff would work compatibly.
supporting SNI clients by default may actually break less stuff that not supporting SNI client. e.g. Webhosting companies may embrace the SNI features of Apache for maximum IP utilization breaking the non-SNI aware clients (which won't be the majority of web browsers).
With checking for an IP address? To be RFC compliant IP addresses shouldn't be used in SNI. Apart from socket family checking (AF_INET/AF_INET6) and doing a regex on the name I couldn't see an easy way to do this even looking at the low level socketmodule.c file. Maybe I need to look deeper. I could cheat and look at the Firefox crypto (NSS) code though.
just my current thoughts
Author: Phil Pennock (pdp)
Date: 2009-09-05 03:16
wrapssl(server_hostname=True/False/String) looks good to me.
Your arguments for enabling by default are compelling, for P3k.
Author: Antoine Pitrou (pitrou) * 
Date: 2010-04-20 20:34
Too late for 2.7 now, but looks like a good idea.
Author: Antoine Pitrou (pitrou) *
Date: 2010-05-22 20:17
The patch probably needs refreshing now that first SSL contexts are in.
I wonder whether a combined boolean/string flag is really the best solution.
I think we could instead enable SNI by default and add an optional "server_hostname" to set the hostname to SSLContext.wrap_socket(), so that people can explicitly set the hostname; and otherwise take it, if possible, from the argument given to connect().
We can also add an "enable_sni" attribute to SSLContext (True by default) to allow selective disabling. This attribute would raise an exception if SNI support isn't available, which would be a way to test for it.
Author: Jean-Paul Calderone (exarkun) *
Date: 2010-05-22 22:17
Here's another possible approach:
ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1) ctx.set_tlsext_host_name("foo.bar") skt = ctx.wrap_socket(socket.socket()) skt.connect("bar.baz")
This makes it obvious what the SNI hostname is and what the TCP address to connect to is, and they can easily be different.
Author: Daniel Black (grooverdan) *
Date: 2010-05-23 07:07
- Author: Antoine Pitrou (pitrou) Date: 2010-05-22 20:17
I quite like your proposed alternative here. Not sure when/if I'll get to implement this.
- Author: Jean-Paul Calderone (exarkun) Date: 2010-05-22 22:17 Sorry I don't like this as much. I believe following the RFC for TLS SNI should be implicit and not something the programmer need to put effort into achieving. I acknowledge this approach does go against some explicit behaviour programming quality metrics.
Author: Antoine Pitrou (pitrou) *
Date: 2010-05-23 11:53
ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1) ctx.set_tlsext_host_name("foo.bar")
Well, the hostname should be specific to a connection, so I'm not sure it makes sense to set it on the context. (besides, the OpenSSL APIs only allow it to be set on the SSL structure)
Author: Jean-Paul Calderone (exarkun) *
Date: 2010-05-23 16:37
Sorry I don't like this as much. I believe following the RFC for TLS SNI should be implicit and not something the programmer need to put effort into achieving. I acknowledge this approach does go against some explicit behaviour programming quality metrics.
It's almost always wrong for Python to enforce a particular policy, particularly in a very low level API (which is what the ssl module should be). Python's main job is to make it possible to do things. It's the application developer's job to decide what things should be done.
It would be entirely appropriate, though, for a higher-level interface (for example, the httplib module) to take care of this itself and not require users to explicitly specify things separately.
Well, the hostname should be specific to a connection, so I'm not sure it makes sense to set it on the context.
That doesn't make sense to me. For example, consider the case where you're talking to a web service. The hostname lookup might result in 10 A records, which you then drop into a connection pool. Your application doesn't care which server you talk to (and maybe it talks to serveral, or all, of them). But it does want to specify the same hostname for each.
(besides, the OpenSSL APIs only allow it to be set on the SSL structure)
Nope, I checked before making the suggestion. There's an SSL_CTX_ version of this API (in addition to the SSL_ version).
Author: Jean-Paul Calderone (exarkun) *
Date: 2010-05-23 16:42
Nope, I checked before making the suggestion. There's an SSL_CTX_ version of this API (in addition to the SSL_ version).
Sorry, I just checked again, and it seems you're right. Perhaps I saw SSL_CTX_set_tlsext_servername_callback and got the two confused.
Author: Antoine Pitrou (pitrou) *
Date: 2010-05-24 16:18
Python's main job is to make it possible to do things. It's the application developer's job to decide what things should be done.
It would be entirely appropriate, though, for a higher-level interface (for example, the httplib module) to take care of this itself and not require users to explicitly specify things separately.
Ok, I find this argument rather convincing. Also, enabling implicit SNI with the connect() argument could make user code stop working if he decides to pass the IP instead, without him being able to diagnose precisly what happens.
As you said, httplib/urllib should probably enable client-side SNI by default.
Author: Antoine Pitrou (pitrou) *
Date: 2010-10-21 21:51
Here is a patch for py3k, including http.client and urllib support.
Author: Antoine Pitrou (pitrou) *
Date: 2010-10-22 18:20
Committed with docs in r85793.
Author: Dolf Andringa (Dolf.Andringa)
Date: 2011-08-11 16:47
I see the patch has been applied python3 in r85793, but is there any chance there will also be patches for python 2.6 or 2.7? And if so, what release of python (any version) might this patch be included in?
Author: Antoine Pitrou (pitrou) *
Date: 2011-08-11 16:49
I see the patch has been applied python3 in r85793, but is there any chance there will also be patches for python 2.6 or 2.7
No, Python 2 only receives bug fixes.
Author: Dolf Andringa (Dolf.Andringa)
Date: 2011-08-12 09:00
And python3? Any idea which version the patch will be included there? This might be a good reason to finally take action on migrating my code from python 2.7 to python 3.
On 11 August 2011 18:49, Antoine Pitrou <report@bugs.python.org> wrote:
Antoine Pitrou <pitrou@free.fr> added the comment:
I see the patch has been applied python3 in r85793, but is there any chance there will also be patches for python 2.6 or 2.7
No, Python 2 only receives bug fixes.
Python tracker <report@bugs.python.org> <http://bugs.python.org/issue5639>
Author: Antoine Pitrou (pitrou) *
Date: 2011-08-12 10:50
And python3? Any idea which version the patch will be included there?
It was included in Python 3.2.
Author: Mark Kubacki (markk)
Date: 2013-07-03 12:05
Python 2.7 is still used in production.
Given the scarcity of IPv4-addresses — and with CDNs (think: Amazon, Akamai, EdgeCast…) starting to offer HTTP+SSL — the need for SNI arises in order to avoid pitfalls such as shared certificates.
The lack of ubiquitous support for TLS SNI could cause delays in HTTPS-everywhere–deployments. Therefore, in the light of the latest revelations about mass surveillance, I'd like even to argue that this is a matter of security and privacy.
Author: Antoine Pitrou (pitrou) *
Date: 2013-07-03 12:10
Mark, thanks for the patch. However, unless exceptional situations, we don't backport features to bugfix branches. The next Python 2.7 version will probably not be released before 2014, so even if your patch were integrated, widespread deployment would still be delayed significantly. By contrast, Python 3.2 has SNI support and it was released in February 2011.
Author: Mark Kubacki (markk)
Date: 2013-07-03 12:38
Antoine, thank you for the heads-up. As long as I've reminded distribution maintainers of this issue and this or a similar patch (always send a server_hostname with TLS, if one is missing) will be integrated (please do!) I've accomplished my goal.
BTW, today I've encountered a similar certificate. Semper aliquid haeret:
subjectAltName=DNS:cdn.cloudtop.org,DNS:barely-legal-spam.com,DNS:*.banging-ham.com,DNS:jimmyforcongress2014.com ;-)
Author: Dima Tisnek (Dima.Tisnek) *
Date: 2013-11-25 10:35
Is this really not going into Python2 series?
It's not a Python feature or a language feature, it's a matter of exporting OpenSSL feature.
Furthermore it's a matter of security, same as support for session tickets is a matter of performance.
SNI was first introduced in 2004, RFC in 2006, libcurl supported it from 2008, you had a patch since 2009 and still it's not in?
Are you guys intentionally trying to cripple Python2?
What do you think is likelier outcome, faster Python3 adoption or Python labelled insecure?
Author: Antoine Pitrou (pitrou) *
Date: 2013-11-25 11:06
It's not a Python feature or a language feature, it's a matter of exporting OpenSSL feature.
It's a feature regardless (from our POV), and Python 2.x has been in bug fix mode for a long time now. Please understand that this is how our release process works.
Author: Mark Nottingham (mnot)
Date: 2014-01-08 03:49
This is not a feature request, it's a bug fix in the underlying protocols.
Client sides that do not send SNI are actively hurting the Web and the Internet by constraining the deployment of TLS.
The closest analogy would be if Python's HTTP client side didn't emit a Host header, and the excuse were "But we only advertise ourselves as HTTP/1.0." The biggest difference being that this has additional security impact.
The pain of lack of support for SNI is completely borne by the server-side, not the client (here, Python). As such, this is not a feature for Python client-side developers, but an interop / scaling / security issue for the Web and Internet overall.
Author: Sam Gleske (sag47)
Date: 2014-02-28 05:48
Are you kidding me? I can't believe SNI isn't being backported to python 2.x. This is ridiculous in my opinion. The bug fix needs to be back ported.
Author: Alyssa Coghlan (ncoghlan) *
Date: 2014-03-20 03:16
I'd be happy to add a disclaimer to the Python 2.7 docs directing users to use the requests module instead (https://pypi.python.org/pypi/requests).
People really really really should be using requests on the client side when doing HTTP/HTTPS in Python 2.x - the standard library support is now too old to have kept up with the evolution of web security and standard.
Author: Donald Stufft (dstufft) *
Date: 2014-03-20 03:30
To be clear, to get SNI with requests on 2.x you need requests, pyopenssl, ndg-httpsclient, and pyasn1 (which also pulls in cryptography, six, cffi, and pycparser). So that's 8 dependencies to get SNI on Python 2.x.
At least it's doable but it's kind of really unfriendly :/ Also the error message you get when you need SNI and it's not available is basically the most obtuse thing ever. You get told that the SSL verification failed for that isn't what you asked for but when you go to it in your Browser it'll all work kosher with no indication it was using SNI.
It's generally a good idea to install those extra things anyways because the SSL lib on Python 2.7 has other actual security issues which those address (IIRC it still has TLS compression on, I think it's default cipher list is rather poor, doesn't support TLS 1.2, etc).
Author: Dima Tisnek (Dima.Tisnek) *
Date: 2014-03-20 07:50
+dstufft is absolutely right.
SNI needs to be enabled on lower level than "user" python code. if it is, requests and most other http client libs get it for free without dependencies.
Author: Antoine Pitrou (pitrou) *
Date: 2014-03-20 10:53
Nick: rather than direct users to use requests, we should direct them to use Python 3, which has had SNI support for 3+ years now.
If client programs choose to remain on Python 2, it's their fault, not Python's.
Author: Dima Tisnek (Dima.Tisnek) *
Date: 2014-03-20 10:59
Antoine, was Python 2.x a mistake?
I don't think so.
SNI is not a language feature, it's not even a python extension feature. It's a feature of and existing protocol and the underlying library.
Author: Antoine Pitrou (pitrou) *
Date: 2014-03-20 11:01
Antoine, was Python 2.x a mistake?
Really, can you stop arguing about this? If you want to know what Python considers features and bug fixes, then get acquainted with the development process instead of bickering.
Author: Alyssa Coghlan (ncoghlan) *
Date: 2014-03-20 12:31
I'm currently discussing some options with Donald and Christian. While it's annoying that a developer from a certain large corporate user of Python (a director of the PSF, no less) is whining at volunteers on the internet instead of actually helping by encouraging their employer or the board to help fund the creation and publication of an up to date TLS module for Python 2, griping about the endemic problem of corporate users taking community developed software for granted won't make the underlying problem go away: an unfortunate amount of Python code is currently improperly secured because it is using outdated SSL support, and there isn't currently a good alternative available for users that aren't in a position to immediately migrate to Python 3.
Author: Antoine Pitrou (pitrou) *
Date: 2014-03-20 12:45
I'm missing some context to appreciate your message, Nick, but please note that SNI is not in itself a security feature. It just enables interoperability with TLS virtual hosts (aka. hosting several TLS-enabled domains behind a single IP and port).
Author: Donald Stufft (dstufft) *
Date: 2014-03-20 12:58
It's somewhat of a grey area of security feature. It's not directly a security feature but if you don't have SNI and you hit a site that requires it then your error message is going to be something like what people run into with PyPI[1] which is "Cannot verify pypi.python.org, does not match hostname *.a.ssl.fastly.net". At this point most people go "What?" and assume the site is at fault and disable verification. Even more frustrating is this is going to work fine in their browser. The answer of how to actually verify this is without SNI is (once you even figure out the problem is SNI, which is non obvious) verify against what's actually in the CN of the cert, and send a Host header for what site you actually want. So while it is not strictly a security feature, it is fairly important for reasonably securely connecting to a site that requires SNI for the lay person.
[1] PyPI's problem is no SNI but that some clients don't support SAN certificates, but the error message is exactly the same.
Author: Antoine Pitrou (pitrou) *
Date: 2014-03-20 13:11
Understood, but that's no different from trying to connect with an old Windows or MSIE version (which I'm sure will also fail on some websites).
Client-side SNI support has been added in Python 3.2, and 3.4 is now out. People who migrated their code to Python 3 have been enjoying SNI support for years now, and they're gradually getting more TLS features at every new feature release.
Author: STINNER Victor (vstinner) *
Date: 2014-03-20 13:13
Please discuss the Python 2 documentation in a new issue, this one is now closed and so hidden from the list of bugs.
Author: Dima Tisnek (Dima.Tisnek) *
Date: 2014-04-11 08:50
Hopefully pep-466 resolves this for 2.x series.
History
Date
User
Action
Args
2022-04-11 14:56:47
admin
set
github: 49889
2014-04-11 08:50:52
Dima.Tisnek
set
messages: +
2014-03-20 13:13:50
vstinner
set
nosy: + vstinner
messages: +
2014-03-20 13:11:41
pitrou
set
messages: +
2014-03-20 12:58:01
dstufft
set
messages: +
2014-03-20 12:45:52
pitrou
set
messages: +
2014-03-20 12:31:48
ncoghlan
set
messages: +
2014-03-20 11:01:24
pitrou
set
messages: +
2014-03-20 10:59:36
Dima.Tisnek
set
messages: +
2014-03-20 10:53:27
pitrou
set
messages: +
2014-03-20 07:50:25
Dima.Tisnek
set
messages: +
2014-03-20 03:30:47
dstufft
set
nosy: + dstufft
messages: +
2014-03-20 03:16:29
ncoghlan
set
nosy: + ncoghlan
messages: +
2014-02-28 05:48:52
sag47
set
nosy: + sag47
messages: +
versions: + Python 3.1, Python 2.7
2014-01-08 03:49:16
mnot
set
nosy: + mnot
messages: +
2013-11-25 11:06:11
pitrou
set
messages: +
2013-11-25 10:35:28
Dima.Tisnek
set
nosy: + Dima.Tisnek
messages: +
2013-07-05 00:05:03
markk
set
files: + python-2.7.5-tlssni.patch
2013-07-04 23:58:02
markk
set
files: - python-2.7.5-tlssni.patch
2013-07-04 23:57:17
markk
set
files: + python-2.7.5-tlssni.patch
2013-07-04 23:54:48
markk
set
files: - python-2.7.5-tlssni.patch
2013-07-03 12:38:09
markk
set
messages: +
2013-07-03 12:10:04
pitrou
set
messages: +
2013-07-03 12:05:45
markk
set
nosy: + markk
messages: +
2013-07-03 11:53:11
markk
set
files: + python-2.7.5-tlssni.patch
2011-08-12 10:50:13
pitrou
set
messages: +
versions: - Python 2.6, Python 2.7
2011-08-12 09:00:55
Dolf.Andringa
set
files: + unnamed
messages: +
2011-08-11 16:49:52
pitrou
set
messages: +
2011-08-11 16:47:09
Dolf.Andringa
set
nosy: + Dolf.Andringa
messages: +
versions: + Python 2.6, Python 2.7
2011-01-07 13:43:16
pitrou
unlink
2010-10-22 18:20:03
pitrou
set
status: open -> closed
resolution: fixed
messages: +
stage: patch review -> resolved
2010-10-21 21:51:42
pitrou
set
files: + sni.patch
messages: +
2010-06-19 09:46:51
scott.tsai
set
nosy: + scott.tsai
2010-05-24 16🔞48
pitrou
set
messages: +
2010-05-23 16:42:41
exarkun
set
messages: +
2010-05-23 16:37:49
exarkun
set
messages: +
2010-05-23 11:53:39
pitrou
set
messages: +
2010-05-23 07:07:50
grooverdan
set
messages: +
2010-05-22 22:17:46
exarkun
set
nosy: + exarkun
messages: +
2010-05-22 20:17:16
pitrou
set
nosy: + giampaolo.rodola
messages: +
2010-04-23 12:42:18
jcea
set
nosy: + jcea
2010-04-20 20:47:50
pitrou
link
2010-04-20 20:34:30
pitrou
set
priority: normal
versions: - Python 2.7
nosy: + pitrou
messages: +
stage: patch review
2009-09-05 03:16:08
pdp
set
messages: +
2009-09-04 06:37:39
grooverdan
set
messages: +
2009-09-01 02:59:43
pdp
set
messages: +
2009-08-31 06:04:13
grooverdan
set
messages: +
2009-08-31 05:43:52
grooverdan
set
files: + pytest.py
nosy: + janssen
messages: +
type: enhancement
2009-08-31 05:33:32
grooverdan
set
versions: + Python 2.7, Python 3.2, - Python 2.6
2009-08-31 05:33:10
grooverdan
set
files: + python-3K-74602-ssl_client_sni.path
messages: +
2009-08-31 05:32:23
grooverdan
set
files: + python-HEAD-74602-ssl_client_sni.path
nosy: + grooverdan
messages: +
2009-04-01 20:30:50
pdp
set
messages: +
2009-04-01 06:38:23
pdp
create