[Python-Dev] Trial balloon: microthreads library in stdlib (original) (raw)

Richard Tew richard.m.tew at gmail.com
Tue Feb 13 13:31:34 CET 2007

On 2/13/07, "Martin v. Löwis" <martin at v.loewis.de> wrote:

Steve Holden schrieb: > The only things that concern me are a) whether it could make sense to > add Stackless in bits and pieces and b) whether the BDFL (or even the > developer community en masse) would object in principle, thereby > rendering such efforts useless.

I think I need to try again. The 'developer community en masse' does not have a single voice, so it won't object. Not sure about the BDFL, but I personally don't object 'to a change like that' 'in principle', primarily because I don't know what the change is. I know I'm almost certainly object to the change 'incorporate Stackless Python into Python as-is', because history shows that any change of that size would need multiple iterations until it was acceptable (despite the code being in-use for multiple years). Very few people can get away with getting their code unedited into Python (ctypes and elementtree being the most prominent examples); in these cases, I could accept what I consider flaws in the code because: a) the authors promised to maintain the code, and react to actual bug reports, and b) these come as extension modules, so it's easy to ignore them if you don't like them.

This makes a lot of sense. You have given a lot of food for thought and a path forward.

If I gave an impression in my mails that I expected a consensus of objection, it was because I believed objection to Stackless to be an established thing and that it didn't matter whether there was consensus or not because someone would step forward and reiterate that. I didn't want to push the issue or invest wasted effort if that proved to be the case as mistakenly expected.

At no time have I expected that Stackless would be added as-is.

> My (limited) understanding is that with Stackless installed all > existing programs that don't import stackless should continue to run > unchanged. If that's true then it seems to me it would be a desirable > addition as long as maintainability wasn't compromised. I suppose only > the patches will allow a sensible judgment on that issue.

My understanding is that incorporating Stackless will reduce the portability: it cannot work on certain platforms, and even for platforms it works on, it cannot work with certain compilers or compiler switches (the Windows SEH being the primary example; the SPARC register stack another one - although this might now be supported through assembler code). On platforms where it isn't supported, it still may compile, but might crash the interpreter when used.

If there is no Stackless 'hard switching' available (the stack switching done with assistance of assembler) for the platform it is being compiled on, then compilation proceeds without Stackless compiled into Python. The module is not available. The process of adding support for additional platforms is rather straightforward and with access to them I can do it if required.

I don't know about SEH but I believe support for SPARC was added in 2002. http://svn.python.org/view/stackless/trunk/Stackless/platf/switch_sparc_sun_gcc.h

Please understand that this is from a shallow inspection a few years ago (actually, a description of the working mechanics that Christian gave me). It would make me feel uncomfortable if Python had modules that may randomly crash the interpreter, and there is no procedure to tell beforehand whether a certain applications is supported or not.

Right. I imagine that if there are still problems like these, they will be brought to light when and if patches are submitted. But for now I will address what you mention just so that people don't assume they are still the case, especially since Stackless has been rewritten since then.

Also, I would not like to see modules that monkey-patch other modules in the standard library. If socket.connect is to behave differently, the code that makes that so should be in socket.connect, and the documentation of socket.connect should state that it has that modified behavior. Of course, people may object to massive library changes of the nature "if running in stackless mode, this library routine behaves differently".

Perhaps my pursuit of better support for asynchronous calls has led to some confusion. Stackless is an implementation with minimal changes to the core. It does not include any modules which monkey patch.

However I have a replacement socket module based on asyncore which is not part of the Stackless distribution. If monkey patched it allows people to use sockets suitably in their microthreads. I am experimenting with this in order to see if I can make using Stackless as close to being as straightforward as writing normal Python code as possible, while still gaining the benefits which use of microthreads should allow.

It was considered and quickly dismissed to modify the socket and file support in the core at C level to do what was needed. Primarily because it would complicate the limited changes we make to the core.

Thanks, Richard.

More information about the Python-Dev mailing list