[Python-Dev] subprocess crossplatformness and async communication (original) (raw)

Nick Coghlan ncoghlan at gmail.com
Mon Jan 26 23:43:24 CET 2009

Daniel Stutzbach wrote:

I'm confused. What's wrong with the following?

p = Popen('dosomething', stdin=PIPE, stdout=PIPE) p.stdin.write('la la la\n') p.stdin.flush() line = p.stdout.readline() p.stdin.write(process(line)) p.stdin.flush() If you want to see if data is available on p.stdout, use the select module (unless you're on Windows). The child process has to flush its output buffer for this to work, but that isn't Python's problem.

Anatoly covered that in his response to Guido: "You can't launch a process and communicate with it without blocking at some point. The scenario where you can periodically check output and error pipes without blocking and send input is not supported."

With the vanilla subprocess Popen implmentation, the stdin.write calls can actually both block if the stdin buffer is full (waiting for the child process to clear space) and the stdout.readline call can definitely block (waiting for the child process to end the line).

So it isn't async communication in general that is the concern: it is non-blocking async communication. (Since I'm happy with the idea of threaded programs, I'd personally just farm the task off to some worker threads and make it non-blocking that way, but that approach effectively abandons the whole concept of non-blocking IO and it's ability to avoid using threads).

As Guido said though, while there doesn't seem to be anything fundamentally wrong with the idea of adding better support for non-blocking IO to subprocess, it's difficult to have an opinion without a concrete proposal for API changes to subprocess.Popen. The linked recipe certainly can't be adopted as is (e.g. due to the dependency on pywin32)

Cheers, Nick.

-- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia

More information about the Python-Dev mailing list