[Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements (original) (raw)
Raymond Hettinger raymond.hettinger at gmail.com
Tue Apr 5 21:57:13 CEST 2011
- Previous message: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
- Next message: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
[Brett]
This PEP requires that in these instances that both the Python and C code must be semantically identical
Are you talking about the guaranteed semantics promised by the docs or are you talking about every possible implementation detail?
ISTM that even with pure python code, we get problems with people relying on implementation specific details.
Two functions accept a sequence, but one accesses it using len and getitem while the other uses iter. (This is like the Spam example in the PEP).
Given pure python library code like: if x < y: ... I've seen people only implement __lt__ but not __gt__, making it impossible to make even minor adjustments to the code such as: if y > x: ...
We also suffer from inconsistency in choice of exceptions (i.e. overly large sequence indices raising either an IndexError, OverflowError, or ValueError).
With C code, I wonder if certain implementation differences go with the territory:
Concurrency issues are a common semantic difference. For example, deque.pop() is atomic because the C code holds the GIL but a pure python equivalent would have to use locks to achieve same effect (and even then might introduce liveness or deadlock issues).
Heapq is one of the rare examples of purely algorithmic code. Much of the code in CPython does accesses libraries (i.e. the math module), interfaces with the OS, access binary data structures, links to third-party tools (sqlite3 and Tkinter) or does something else that doesn't have pure python equivalents (at least without using C types).
The C API for parsing argument tuples and keywords do not readily parallel the way the same are written in Python. And with iterators, the argument checking in the C versions tends to happen when the iterator is instantiated, but code written with pure python generators doesn't have its setup and checking section run until next() is called the first time.
We've had a very difficult time bridging the gulf between python's infinite precision numbers and and C's fixed width numbers (for example, it took years to get range() to handle values greater than a word size).
C code tends to be written in a way that takes advantage of that language's features instead of in a form that is a direct translation of pure python. For example, I think the work being done on a C implementation of decimal has vastly different internal structures and it would be a huge challenge to make it semantically identical to the pure python version with respect to its implementation details. Likewise, a worthwhile C implementation of OrderedDict can only achieve massive space savings by having majorly different implementation details.
Instead of expressing the wishful thought that C versions and pure Python versions are semantically identical with respect to implementation details, I would like to see more thought put into specific limitations on C coding techniques and general agreement on which implementation specific details should be guaranteed:
I would like to see a restriction on the use of the concrete C API such that it is only used when a exact type match has been found or created (i.e. if someone writes Py_ListNew(), then it is okay to use Py_ListSetItem()). See http://bugs.python.org/issue10977 for a discussion of what can go wrong. The original json C was an example of code that used the concrete C API is a way that precluded pure python subclasses of list and dict.
I would like to see better consistency on when to use OverflowError vs ValueError vs IndexError.
There should also be a discussion of whether the possible exceptions should be a guaranteed part of the API as it is in Java. Because there were no guarantees (i.e. ord(x) can raise this, that, and the other), people tend to run an experiment and then rely on whatever C Python happens to do.
There should be a discussion on when it is okay for a C implementation to handle only a value range that fits in a word.
When there is C code, when is it okay for a user to assume atomic access? Even with pure python code, we're not always consistent about it (i.e. OrderedDict implementation is not threadsafe but the LRU_Cache is).
There should be some agreement that people implementing rich comparisons will implement all six operations so that client code doesn't become dependent on (xx). For example, we had to add special-case logic to heapq years ago because Twisted implemented a task object that defined le instead of lt, so it was usable only with an older version of heapq but not with min, sort, etc.
A good PEP should address these issues head-on. Just saying that C and python code have to be semantically identical in all implementation details doesn't really address the issue.
[Brett]
(sorry, Raymond, for picking on heapq, but is was what bit the PyPy people most recently =).
No worries, it wasn't even my code. Someone donated it. The was a discusion on python-dev and collective agreement to allow it to have semantic differences that would let it run faster. IIRC, the final call was made by Uncle Timmy.
That being said, I would like to see a broader set of examples rather rather than extrapolating from a single piece 7+ year-old code. It is purely algorithmic, so it really just represents the simplest case. It would be much more interesting to discuss something what should be done with future C implementations for threading, decimal, OrderedDict, or some existing non-trivial C accelerators like that for JSON or XML.
Brett, thanks for bringing the issue up. I've been bugged for a good while about issues like overbroad use of the concrete C API.
Raymond
- Previous message: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
- Next message: [Python-Dev] PEP 399: Pure Python/C Accelerator Module Compatibiilty Requirements
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]