[Python-Dev] Psyco requests and patches (original) (raw)

Armin Rigo arigo@ulb.ac.be
Wed, 02 Oct 2002 01:33:47 +0200

Hello everybody,

As I have been putting more work into Psyco to make sure it will work well with Python 2.3, I came up with a few support requests.

Non-technical stuff first. People tend to think about Psyco as an obscure and complex piece of software for gurus -- should they know about it at all. Of course I would argue instead that it is exactly the piece of software that Python users should not have to worry about, because it should be perfectly transparent. As you can guess I would even say that the first widely-used dynamic programming language to natively include this kind of techniques would see a huge widening of its application domain. For Psyco to be a first step in that direction, here are a few goals that I would like you to comment.

Publicity: well, I could do with a little more advertising, e.g. on www.python.org or better yet in the docs and tutorials, at places where performance is discussed. Right now I am not sure that you can learn about Psyco's existence unless you read python-dev or comp.lang.python.

Ease of use: currently the psyco.bind() interface is easy to use, but it could be trivial instead of just easy. It should actually be fully transparent. (That's what psyco.jit() promised to do, but right now it badly sucks; keeping it around may have been an error, as people is actually trying it and getting bad results.)

Portability: the easy part! Indeed, an empty module named 'psyco' portably provides exactly the same functionality as Psyco everywhere... On non-supported platforms you just get nothing better than the current interpreter performance, but nothing worse either. (Besides, emitting portable pseudo-code could still be a future option. Of course, emitting a platform's machine code is not that easy at all, but this fact doesn't compromise portability.)

Ease of maintenance: the hard part. Psyco is not currently ready for that. The implementation is way too obscure. You have no guarantee if I tell -- and I do -- that I plan to upgrade Psyco to all newer Python versions for at least as long as the 2.x series is alive. We need more than one person involved. (Note that upgrades are not as time-consuming as, say, in the case of Christian Tismer's first Stackless implementation that required a complete rewrite of the interpreter.) Let's defer this point to the bottom of this e-mail.

Inclusion in the Standard Library or the interpreter core itself: not right now! This is a guideline for the goals discussed above. I might be inclined to think that completion of all the previous goals would be enough for consideration...?

-=- Python patches -=-

Psyco requires no change in the interpreter core -- however, there are a couple of things that needed obfuscated workarounds or just don't completely work at all. A few minor changes would make it all much cleaner. I submitted three patches to SF (http://www.python.org/sf/617309, http://www.python.org/sf/617311, http://www.python.org/sf/617312). The patches should be clear enough, not introducing obscure "features" or measurable performance loss. The diffs are currently against the 2.2 branch -- it would greatly help Psyco if the patches made their way into Python 2.2.2, which I understood to be the candidate Python-in-a-tie. I will submit 2.3 versions of the patches as well, of course.

-=- Maintenance of Psyco -=-

Psyco is currently highly obscure. I want to rewrite it in Python. Still, however, there will always be the problem that for any feature supported by Psyco someone must track the changes in the corresponding feature in the interpreter, and port them. (This doesn't concern the features not supported by Psyco: for these it always calls their up-to-date C implementation.) This process might be semi-automated with a tool that would be able to dig into C code and analyse it, but this would be sort of a hack.

We might do some thinking about this synchronization problem when the discussion about Python 3 is raised again: how hard would it be e.g. to write some parts of the interpreter directly in Python -- or at least in some specific dialect that can be turned into C (a la Pyrex) for ease of compilation? It may actually help make the interpreter more understandable. This looks a bit far-fetched, but I will never give up my claims that high-level languages are just better than low-level ones for most purposes, and writing an interpreter is no exception!

For example, there was a December 2000 proposal to rewrite the "cmath" module in Python (http://www.python.org/sf/210838). It never actually made its way into the Standard library, which is too bad for Psyco, because Psyco-with-cmath-in-Python would be faster than Psyco-with-cmath-in-C.

Practically, the whole point of having a single high-level source code is that you can do more than just one thing with it. With C code, it is a horrible hack to do anything else than compile and execute it. With Python code it's a different matter. You can use this single source to produce both the equivalent C code and to produce Psyco itself. In each case there is still an important part written directly in C, of course; I am not (yet;-) suggesting that Python should be written only in Python! Very roughly, the separation would be between the built-in objects + abstract protocols in 'Objects/' (which could be written in Python) and the core in 'Python/' (which could not).

The framework required to do such a thing looks complicated. I would argue that it probably isn't (and in any case it might be worth being carefully considered); this extra complexity might be favorably balanced if only just because writing Python in Python is easier than in C :-)

A bientot,

Armin

PS: I just obtained my math Ph.D. -- it's now Psyco time again!