[Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7 (original) (raw)

Ben Darnell ben at bendarnell.com
Fri Jun 2 09:43:21 EDT 2017

The PEP's rationale is now "This PEP will help facilitate the future adoption of :pep:543 across all supported Python versions, which will improve security for both Python 2 and Python 3 users."

What exactly are these security improvements? My understanding (which may well be incorrect) is that the security improvements come in the future when the PEP 543 APIs are implemented on top of the various platform-native security libraries. These changes will not be present until some future 3.x release, and will not be backported to 2.7 (without another PEP, which I expect would be even more contentious than this one). What then is the security improvement for Python 2 users?

In Tornado, I have not felt any urgency to replace wrap_socket with MemoryBIO. Is there a security-related reason I should do so sooner rather than later? (I'd also urge Cory and any other wrap_socket skeptics on the requests team to reconsider - Tornado's SSLIOStream works well. The asynchronous use of wrap_socket isn't so subtle and risky with buffering)

-Ben

On Fri, Jun 2, 2017 at 8:07 AM Antoine Pitrou <solipsis at pitrou.net> wrote:

Thanks you for all the explanations. So to summarize my opinion, I'm still -0.5 on this PEP. I would also like to see the issues Jython, Ubuntu et al. have mentioned solved before this is accepted. Regards Antoine.

On Fri, 2 Jun 2017 11:42:58 +0200 Victor Stinner <victor.stinner at gmail.com> wrote: > Thanks Cory for the long explanation. Let me try to summarize (tell me > if I'm wrong). > > We have 3 options: > > * Do nothing: reject the PEP 546 and let each project handles security > on its own (current status co) > * Write new C code, maybe using certitude as a starting point, to > offload certifcate validation on Windows and macOS > * Backport existing code from master to 2.7: MemoryBIO and SSLObject > > Writing new code seems more risky and error-prone than backporting > already "battle-tested" MemoryBIO from master. I also expect that > writing code to validate certificate will be longer than the "100 > lines of C code in (probably)" expected by Steve Dower. > > rust-certitude counts around 700 lines of Rust and 80 lines of Python > code. But maybe I misunderstood the purpose of certitude: Steve Dower > asked to only validate a certificate, not load or export CA. > > I counted 150 Python lines for SSLObject and 230 C lines for MemoryBIO. > > Since the long term plan is to not use stdlib ssl but a new > implementation on Windows and macOS, it seems worthless to backport > MemoryBIO on Python 2.7. The PEP 546 (backport MemoryBIO) is a > practical solution to provide a smooth transition from ssl to a new > TLS API. The experience showed that hard changes like "run 2to3 and > drop your Python 2 code" doesn't work in practice. Users want a > transition plan with small steps. > > Victor > > 2017-06-02 11:08 GMT+02:00 Cory Benfield <cory at lukasa.co.uk>: > > > > On 1 Jun 2017, at 20:59, Steve Dower <steve.dower at python.org> wrote: > > > > On 01Jun2017 1010, Nathaniel Smith wrote: > > > > I believe that for answering this question about the ssl module, it's really > > only Linux users that matter, since pip/requests/everyone else pushing for > > this only want to use ssl.MemoryBIO on Linux. Their plan on Windows/MacOS > > (IIUC) is to stop using the ssl module entirely in favor of new ctypes > > bindings for their respective native TLS libraries. > > (And yes, in principle it might be possible to write new ctypes-based > > bindings for openssl, but (a) this whole project is already teetering on the > > verge of being impossible given the resources available, so adding any major > > extra deliverable is likely to sink the whole thing, and (b) compared to the > > proprietary libraries, openssl is much harder and riskier to wrap at the > > ctypes level because it has different/incompatible ABIs depending on its > > micro version and the vendor who distributed it. This is why manylinux > > packages that need openssl have to ship their own, but pip can't and > > shouldn't ship its own openssl for many hopefully obvious reasons.) > > > > > > How much of a stop-gap would it be (for Windows at least) to override > > OpenSSL's certificate validation with a call into the OS? This leaves most > > of the work with OpenSSL, but lets the OS say yes/no to the certificates > > based on its own configuration. > > > > For Windows, this is under 100 lines of C code in (probably) ssl, and while > > I think an SChannel based approach is the better way to go long-term,[1] > > offering platform-specific certificate validation as the default in 2.7 is > > far more palatable than backporting new public API. > > > > > > It’s entirely do-able. This is where I reveal just how long I’ve been > > fretting over this problem: https://pypi.python.org/pypi/certitude. Ignore > > the description, it’s wildly out-of-date: let me summarise the library > > instead. > > > > Certitude is a Python library that uses CFFI and Rust to call into the > > system certificate validation libraries on macOS and Windows using a single > > unified API. Under the covers it has a whole bunch of Rust code that > > translates from what OpenSSL can give you (a list of certificates in the > > peer cert chain in DER format) and into what those two operating systems > > expect. The Rust code for Windows is here[1] and is about as horrifying a > > chunk of Rust as you can imagine seeing (the Windows API does not translate > > very neatly into Rust so the word “unsafe” appears a lot), but it does > > appear to work, at least in the mainline cases and in the few tests I have. > > The macOS code is here[2] and is moderately less horrifying, containing no > > instances of the word “unsafe”. > > > > I lifted this approach from Chrome, because at the moment this is what they > > do: they use their custom fork of OpenSSL (BoringSSL) to do the actual TLS > > protocol manipulation, but hand the cert chain verification off to > > platform-native libraries on Windows and macOS. > > > > I have never productised this library because ultimately I never had the > > time to spend writing a sufficiently broad test-case to confirm to me that > > it worked in all cases. There are very real risks in calling these APIs > > directly because if you get it wrong it’s easy to fail open. > > > > It should be noted that right now certitude only works with, surprise, > > PyOpenSSL. Partly this is because the standard library does not expose > > SSLgetpeercertchain, but even if it did that wouldn’t be enough as > > OpenSSL with VERIFYNONE does not actually save the peer cert chain > > anywhere. That means even with PyOpenSSL the only way to get the peer cert > > chain is to hook into the verify callback and save off the certs as they > > come in, a gloriously absurd solution that is impossible with pure-Python > > code from the ssl module. > > > > While this approach could work with ssl.c, it ultimately doesn’t resolve > > the issue. It involves writing a substantial amount of new code which needs > > to be maintained by the ssl module maintainers. All of this code needs to be > > tested thoroughly, because python-dev would be accepting responsibility > > for the incredibly damaging potential CVEs in that code. And it doesn’t get > > python-dev out of the business of shipping OpenSSL on macOS and Windows, > > meaning that python-dev continues to bear the burden of OpenSSL CVEs, as > > well as the brand new CVEs that it is at risk of introducing. > > > > Oh, and it can’t be backported to Python 2.7 or any of the bugfix-only > > Python 3 releases, and as I just noted the ssl module has never made it > > possible to use this approach from outside CPython. So it’s strictly just as > > bad as the situation PEP 543 is in, but with more C code. Doesn’t sound like > > a winning description to me. ;) > > > > Cory > > > > [1]: > > https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/windows.rs > > [2]: > > https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/osx.rs > > _> > ________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > > _> ________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ben%40bendarnell.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20170602/9d28a3bb/attachment-0001.html>

More information about the Python-Dev mailing list