[Python-Dev] subprocess crossplatformness and async communication (original) (raw)

Guido van Rossum guido at python.org
Mon Jan 26 19:36:42 CET 2009

On Mon, Jan 26, 2009 at 12:44 AM, anatoly techtonik <techtonik at gmail.com> wrote:

You can't launch a process and communicate with it without blocking at some point. The scenario where you can periodically check output and error pipes without blocking and send input is not supported.

Scenario One: control console program. This implies reading from input and writing replies asyncronously - Popen.stdin, stdout, stderr - stdin.write(), stdout.read() or stderr.read() block script execution if any of the other OS pipe buffers fills up and blocks the child process. Advice is to use communicate(), but it waits for process to exit and hence doesn't allow to send the input based on reply from previous send. Scenario Two: waiting for process that terminates with extensive output blocks - Popen.wait() blocks if the child process generates enough output to a stdout or stderr pipe, advice is to use Popen.communicate() http://docs.python.org/library/subprocess.html#subprocess.Popen.wait - Popen.communicate() - Waits for process to terminate, reads data from stdout and stderr until end-of-file. Cashes data in memory - should not be used for if the data size is large or unlimited. Solution - read(maxsize) and write(maxsize) functions that will return immediately.

Hi Anatoly -- thanks for clarifying your issues. I hope other developers more familiar with subprocess.py will chime in and either help you figure out a way to do this without changes to the subprocess module, or, if that would be too painful, help you develop additional APIs. You could start by proposing a set of changes to subprocess.py and submit them as a patch to bugs.python.org; that is easier to deal with than pointing to a recipe on the Activestate site.

-- --Guido van Rossum (home page: http://www.python.org/~guido/)

More information about the Python-Dev mailing list