[Python-Dev] subprocess crossplatformness and async communication (original) (raw)
anatoly techtonik techtonik at gmail.com
Mon Jan 26 09:44:01 CET 2009
- Previous message: [Python-Dev] subprocess crossplatformness and async communication
- Next message: [Python-Dev] subprocess crossplatformness and async communication
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Hi, Guido
You can't launch a process and communicate with it without blocking at some point. The scenario where you can periodically check output and error pipes without blocking and send input is not supported.
Scenario One: control console program. This implies reading from input and writing replies asyncronously
- Popen.stdin, stdout, stderr - stdin.write(), stdout.read() or stderr.read() block script execution if any of the other OS pipe buffers fills up and blocks the child process. Advice is to use communicate(), but it waits for process to exit and hence doesn't allow to send the input based on reply from previous send.
Scenario Two: waiting for process that terminates with extensive output blocks
- Popen.wait() blocks if the child process generates enough output to a stdout or stderr pipe, advice is to use Popen.communicate() http://docs.python.org/library/subprocess.html#subprocess.Popen.wait
- Popen.communicate() - Waits for process to terminate, reads data from stdout and stderr until end-of-file. Cashes data in memory - should not be used for if the data size is large or unlimited.
Solution - read(maxsize) and write(maxsize) functions that will return immediately.
-- anatoly t.
On Sat, Jan 24, 2009 at 5:58 PM, Guido van Rossum <guido at python.org> wrote:
Anatoly,
I'm confused. The subprocess already allows reading/writing its stdin/stdout/stderr, and AFAIK it's a platform-neutral API. I'm sure there's something missing, but your post doesn't make it clear what exactly, and the recipe you reference is too large to digest easily. Can you explain what it is that the current subprocess does't have beyond saying "async communication" (which could mean many things to many people)? --Guido On Sat, Jan 24, 2009 at 5:07 AM, anatoly techtonik <techtonik at gmail.com> wrote: Greetings,
This turned out to be a rather long post that in short can be summarized as: "please-please-please, include asynchronous process communication in subprocess module and do not allow "available only on ..." functionality", because it hurts the brain". Code to speak for itself: http://code.activestate.com/recipes/440554/
The subprocess module was a great step forward to unify various spawn and system and exec and etc. calls in one module, and more importantly - in one uniform API. But this API is partly crossplatform, and I believe I've seen recent commits to docs with more unix-only differences in this module. The main point of this module is to "allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes". PEP 324 goal is also to make "make Python an even better replacement language for over-complicated shell scripts". Citing pre-subrocess PEP 324, "Currently, Python has a large number of different functions for process creation. This makes it hard for developers to choose." Now there is one class with many methods and many platform-specific comments and notices. To make thing worse people on Unix use subprocess with fcntl and people on windows tend not to use it at all, because it looks complicated and doesn't solve the problem with asynchronous communication. That I suggest is to add either support for async crossplatfrom read/write/probing of executed process or a comment at the top of module documentation that will warn that subprocess works in blocking mode. With async mode you can emulate blocking, the opposite is not possible. This will save python users a lot of time. Thanks for reading my rant. BTW, the proposed change is top 10 python recipe on ActiveState http://code.activestate.com/recipes/langs/python/ -- --anatoly t.
Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
-- --anatoly t.
- Previous message: [Python-Dev] subprocess crossplatformness and async communication
- Next message: [Python-Dev] subprocess crossplatformness and async communication
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]