[Python-Dev] PyPy 1.7 - widening the sweet spot (original) (raw)
Amaury Forgeot d'Arc amauryfa at gmail.com
Fri Nov 25 19:21:48 CET 2011
- Previous message: [Python-Dev] PyPy 1.7 - widening the sweet spot
- Next message: [Python-Dev] PyPy 1.7 - widening the sweet spot
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
2011/11/25 Brett Cannon <brett at python.org>
On Thu, Nov 24, 2011 at 07:46, Nick Coghlan <ncoghlan at gmail.com> wrote: On Thu, Nov 24, 2011 at 10:20 PM, Maciej Fijalkowski <fijall at gmail.com> wrote: > The problem is not with maintaining the modified directory. The > problem was always things like changing interface between the C > version and the Python version or introduction of new stuff that does > not run on pypy because it relies on refcounting. I don't see how > having a subrepo helps here.
Indeed, the main thing that can help on this front is to get more modules to the same state as heapq, io, datetime (and perhaps a few others that have slipped my mind) where the CPython repo actually contains both C and Python implementations and the test suite exercises both to make sure their interfaces remain suitably consistent (even though, during normal operation, CPython users will only ever hit the C accelerated version). This not only helps other implementations (by keeping a Python version of the module continuously up to date with any semantic changes), but can help people that are porting CPython to new platforms: the C extension modules are far more likely to break in that situation than the pure Python equivalents, and a relatively slow fallback is often going to be better than no fallback at all. (Note that ctypes based pure Python modules aren't particularly useful for this purpose, though - due to the libffi dependency, ctypes is one of the extension modules most likely to break when porting). And the other reason I plan to see this through before I die is to help distribute the maintenance burden. Why should multiple VMs fix bad assumptions made by CPython in their own siloed repos and then we hope the change gets pushed upstream to CPython when it could be fixed once in a single repo that everyone works off of?
PyPy copied the CPython stdlib in a directory named "2.7", which is never modified; instead, adaptations are made by copying the file into "modified-2.7", and fixed there. Both directories appear in sys.path This was done for this very reason: so that it's easy to identify the differences and suggest changes to push upstream.
But this process was not very successful for several reasons:
The definition of "bad assumptions" used to be very strict. It's much much better nowadays, thanks to the ResourceWarning in 3.x for example (most changes in modified-2.7 are related to the garbage collector), and wider acceptance by the core developers of the "@impl_detail" decorators in tests.
2.7 was already in maintenance mode, and such changes were not considered as bug fixes, so modified-2.7 never shrinks. It was a bit hard to find the motivation to fix only the 3.2 version of the stdlib, which you can not even test with PyPy!
Some modules in the stdlib rely on specific behaviors of the VM or extension modules that are not always easy to implement correctly in PyPy. The ctypes module is the most obvious example to me, but also the pickle/copy modules which were modified because of subtle differences around built-in methods (or was it the builtins module?) And oh, I almost forgot distutils, which needs to parse some Makefile which of course does not exist in PyPy.
Differences between C extensions and pure Python modules are sometimes considered "undefined behaviour" and are rejected. See issue13274, this one has an happy ending, but I remember that the _pyio.py module chose to not fix some obscure reentrancy issues (which I completely agree with)
-- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20111125/b6d66f38/attachment-0001.html>
- Previous message: [Python-Dev] PyPy 1.7 - widening the sweet spot
- Next message: [Python-Dev] PyPy 1.7 - widening the sweet spot
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]