[Python-Dev] Need advice, maybe support (original) (raw)

Guido van Rossum guido@python.org
Sun, 18 May 2003 17:04:40 -0400

In the last months, I made very much progress with Stackless 3.0 . Finally, I was able to make much more of Python stackless (which means, does not use recursive interpreter calls) than I could achive with 1.0 .

There is one drawback with this, and I need advice: Compared to older Python versions, Py 2.2.2 and up uses more indirection through C function pointers than ever. This blocked my implementation of stackless versions, in the first place. Then the idea hit me like a blizzard: Most problems simply vanish if I add another slot to the PyMethodDef structure, which is NULL by default: mlmethnr is a function pointer with the same semantics as mlmeth, but it tries to perform its action without doing a recursive call. It tries instead to push a frame and to return PyUnwindToken. Doing this change made Stackless crystal clear and simple: A C extension not aware of Stackless does what it does all the time: call mlmeth. Stackless aware C code (like my modified ceval.c code) calls the mlmethnr slots, instead, which either defaults to the mlmeth code, or has a special version which avoids recursive interpreter calls. I also added a tpcallnr slot to typeobject, for similar reasons. While this is just great for me, yielding complete source code compatability, it is a slight drawback, since almost all extension modules make use of the PyMethodDef structure. Therefore, binary compatability of Stackless has degraded, dramatically. I'm now in some kind of dilemma: On the one side, I'm happy with this solution (while I have to admit that it is not too inexpensive, but well, all the new descriptor objects are also not cheap, but just great), on the other hand, simply replacing python22.dll is no longer sufficient. You need to re-compile everything, which might be a hard thing on Windows (win32 extensions, wxPython). Sure, I would stand this, if there is no alternative, I would have to supply a complete replacement package of everything. Do you (does anybody) have an alternative suggestion how to efficiently maintain a "normal" and a "non-recursive" version of a method without changing the PyMethodDef struc? Alternatively, would it be reasonable to ask the Python core developers, if they would accept to augment PyMethodDef and PyTypeObject with an extra field (default NULL, no maintenance), just for me and Stackless? Many thanks for any reply - sincerely -- chris

I don't think we can just add an extra field to PyMethodDef, because it would break binary incompatibility. Currently, in most cases, a 3r party extension module compiled for an earlier Python version can still be used with a later version. Because PyMethodDef is used as an array, adding a field to it would break this.

I have less of a problem with extending PyTypeObject, it grows all the time and the tp_flags bits tell you how large the one you've got is. (I still have some problems with this, because things that are of no use to the regular Python core developers tend to either confuse them, or be broken on a regular basis.)

Maybe you could get away with defining an alternative structure for PyMethodDef and having a flag in tp_flags say which it is; there are plenty of unused bits and I don't mind reserving one for you. Then you'd have to change all the code that uses tp_methods, but there isn't much of that; in fact, the only place I see is in typeobject.c.

If this doesn't work for you, maybe you could somehow fold the two implementation functions into one, and put something special in the argument list to signal that the non-recursive version is wanted? (Thinking aloud here -- I don't know exactly what the usage pattern of the nr versions will be.)

--Guido van Rossum (home page: http://www.python.org/~guido/)