[Python-ideas] Tulip / PEP 3156 (original) (raw)

[Python-ideas] Tulip / PEP 3156 - subprocess events

Paul Moore p.f.moore at gmail.com
Wed Jan 16 19:10:34 CET 2013

On 16 January 2013 17:52, Guido van Rossum <guido at python.org> wrote:

On Wed, Jan 16, 2013 at 7:12 AM, Paul Moore <p.f.moore at gmail.com> wrote:

I've so far been lurking on the tulip/async discussions, as although I'm interested, I have no specific need for writing high-performance network code.

However, I hit a use case today which seems to me to be ideal for an async-style approach, and yet I don't think it's covered by the current PEP. Specifically, I am looking at monitoring a subprocess.Popen object. This is basically an IO loop, but monitoring the 3 pipes to the subprocess (well, only stdout and stderr in my case...). Something like addreader/addwriter would be fine, except for the fact that (a) they are documented as low-level not for the user, and (b) they don't work in all cases (e.g. in a select-based loop on Windows). I'd like PEP 3156 to include some support for waiting on IO from (one or more) subprocesses like this in a cross-platform way. If there's something in there to do this at the moment, that's great, but it wasn't obvious to me when I looked... This is a great use case. The right approach would probably be to define a new Transport (and an event loop method to create one) that wraps pipes going into and out of a subprocess. The new method would have a standard API (probably similar to that of subprocess), whereas there would be different implementations of the Transport based on platform and event loop implementation (similar to the way the subprocess module has quite different implementations). Can you check out the Tulip source code (code.google.com/p/tulip) and come up with a patch to do this? I'll gladly review it. It's fine to only cover the UNIX case for now.

I'll have a look. There is one problem, though - I imagine it will be relatively easy to put something together that works on Unix, as waiting on pipes is covered by the existing select/poll mechanisms. But I'm on Windows, so I won't be able to test it. And on Windows, there's no mechanism in place to wait on arbitrary filehandles, so the process wait mechanism is a much harder nut to crack. Chicken and egg problem...

Maybe I'll start by looking at waiting on arbitrary filehandles, and use that to build the process approach. Unfortunately, I don't think IOCP is any more able to wait on arbitrary files than select - see my followup to an older thread on Richard's work there. Or maybe I'll set up a hacking environment in a Linux VM or something. That'd be a fun experience in any case.

I'll have to get my brain round the existing spec as well. I'm finding it hard to understand why there are so many methods on the event loop that are specific to particular use cases (for this example, your suggested method to create the new type of Transport). My instinct says that this should also be a good test case for a user coming up with a new type of "event source" and wanting to plug it into the event loop. Having to add a new method to the event loop seems to imply this isn't possible.

OK, off to do a lot of spec reading and then some coding. With luck, you'll be patient with dumb questions from me on the way :-)

Thanks, Paul

More information about the Python-ideas mailing list