(original) (raw)

diff -r 842c42cfd1c7 pep-3145.txt --- a/pep-3145.txt Wed Dec 19 09:59:50 2012 -0800 +++ b/pep-3145.txt Thu Dec 20 00:46:27 2012 +0300 @@ -2,93 +2,99 @@ Title: Asynchronous I/O For subprocess.Popen Version: RevisionRevisionRevision Last-Modified: DateDateDate -Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson +Author: (James) Eric Pruitt, + Charles R. McCreary, + Josiah Carlson, + anatoly techtonik techtonik@gmail.com Status: Draft Type: Standards Track Content-Type: text/plain Created: 04-Aug-2009 -Python-Version: 3.2 -Post-History: +Python-Version: 3.4 +Post-History: 09-Sep-2009 Abstract: - In its present form, the subprocess.Popen implementation is prone to - dead-locking and blocking of the parent Python script while waiting on data - from the child process. This PEP proposes to make - subprocess.Popen more asynchronous to help alleviate these - problems. + In its present form, the process opened with subprocess.Popen can cause a + deadlock in the parent Python script while the script is waiting for data. + This PEP proposes a modification to subprocess.Popen to allow non-blocking + communicatiom with the child process. Motivation: - A search for "python asynchronous subprocess" will turn up numerous - accounts of people wanting to execute a child process and communicate with - it from time to time reading only the data that is available instead of - blocking to wait for the program to produce data [1] [2] [3]. The current - behavior of the subprocess module is that when a user sends or receives - data via the stdin, stderr and stdout file objects, dead locks are common - and documented [4] [5]. While communicate can be used to alleviate some of - the buffering issues, it will still cause the parent process to block while - attempting to read data when none is available to be read from the child - process. + Many people want fine-grained control over the processes executed from + Python [1]. + + This includes real-time interaction with process by reading its + output and reacting accordingly. Other usage scenario is periodically + checking if there is something available in the process output quese to + see if it is still alive and kill it when it doesn't respond. + + Currenly, subprocess allows only one way to interact with the process in a + save way - vie communicat() method, which can send data to the process only + once and buffers its input and output and not suitable for streamed + processing. + + All other ways of communicating with process by reading and writing its + pipes and prone to deadlocks, which is documented [4][5], but not + explained. The cause of the deadlocks are blocking reads provided by stdin, + stdout and stderr streams. The script is blocked when it reads from a pipe + where no data is available. If child process at the same time writes to the + other pipe, it can be blocked too if the amount of written data exceeds the + size of pipe buffer. As the script is already blocked waiting for the data + on an empty pipe, it can not extract the data for the child process to + continue causing the deadlock. Rationale: - There is a documented need for asynchronous, non-blocking functionality in - subprocess.Popen [6] [7] [2] [3]. Inclusion of the code would improve the - utility of the Python standard library that can be used on Unix based and - Windows builds of Python. Practically every I/O object in Python has a - file-like wrapper of some sort. Sockets already act as such and for - strings there is StringIO. Popen can be made to act like a file by simply - using the methods attached the the subprocess.Popen.stderr, stdout and - stdin file-like objects. But when using the read and write methods of - those options, you do not have the benefit of asynchronous I/O. In the - proposed solution the wrapper wraps the asynchronous methods to mimic a - file object. + Deadlocking behaviour of subprocess.Popen and absence of non-blocking read + causes a lot of frustration among Python users [6][7][2][3]. Crossplatform + solution for non-blocking communication with child processes in the + standard library will greatly help every Python tool that should interact + with non-Python software. + + Many stream-like objects in Python have file-like wrappers (Sockets, + StringIO, etc.). Non-blocking functionality for Popen can be added through + the additional methods attached to the subprocess.Popen.stderr, stdout and + stdin file-like objects. Proposed solution implements non-blocking read, + write methods and wraps them to expose a file API. Reference Implementation: - I have been maintaining a Google Code repository that contains all of my - changes including tests and documentation [9] as well as blog detailing - the problems I have come across in the development process [10]. + (James) Eric Pruitt worked on the code during GSoC 2009. The code is now + in Google Code repository including tests and documentation [9]. He also + described the problems from the development process in the blog [10]. - I have been working on implementing non-blocking asynchronous I/O in the - subprocess.Popen module as well as a wrapper class for subprocess.Popen - that makes it so that an executed process can take the place of a file by - duplicating all of the methods and attributes that file objects have. + The changes implement non-blocking I/O in the subprocess.Popen class. It + also provides wrapper methods to work with the executed process through + file object API. - There are two base functions that have been added to the subprocess.Popen - class: Popen.send and Popen._recv, each with two separate implementations, - one for Windows and one for Unix based systems. The Windows - implementation uses ctypes to access the functions needed to control pipes - in the kernel 32 DLL in an asynchronous manner. On Unix based systems, - the Python interface for file control serves the same purpose. The - different implementations of Popen.send and Popen._recv have identical - arguments to make code that uses these functions work across multiple - platforms. + The core non-blocking functionality in subprocess.Popen is provided by + three base methods: Popen.send, Popen.recv and Popen.recv_err. These + methods are crossplatform. On Windwws they use ctypes to access + functions in the kernel32.dll needed to control pipes in a non-blocking + manner. On Unix based systems the use the file control interface provided + vie select and fcntl modules. - When calling the Popen._recv function, it requires the pipe name be - passed as an argument so there exists the Popen.recv function that passes - selects stdout as the pipe for Popen._recv by default. Popen.recv_err - selects stderr as the pipe by default. Popen.recv and Popen.recv_err - are much easier to read and understand than Popen._recv('stdout' ...) and - Popen._recv('stderr' ...) respectively. + Both Popen.recv* methods share the same code in Popen._recv helper, + which accepts 'stdout' or 'stderr' string as argument to select pipe + required for the receiving function. - Since the Popen._recv function does not wait on data to be produced - before returning a value, it may return empty bytes. Popen.asyncread - handles this issue by returning all data read over a given time - interval. + Popen._recv function does not wait for data and may return empty response. + Popen.asyncread method is provided to wait and return all data over a + given period. Popen.asyncwrite is counterpart that pushes data to the + process until either data is depleted or the process disconnects. - The ProcessIOWrapper class uses the asyncread and asyncwrite functions to - allow a process to act like a file so that there are no blocking issues - that can arise from using the stdout and stdin file objects produced from - a subprocess.Popen call. + The implementation also exposes high-level ProcessIOWrapper class to + launch processes and in return get file object that can be accessed safely + wihout any blocking issues. The underlying logic uses .asyncread and + .asyncwrite methods References: [1] [ python-Feature Requests-1191964 ] asynchronous Subprocess - http://mail.python.org/pipermail/python-bugs-list/2006-December/ - 036524.html + http://bugs.python.org/issue1191964 [2] Daily Life in an Ivory Basement : /feb-07/problems-with-subprocess http://ivory.idyll.org/blog/feb-07/problems-with-subprocess @@ -115,7 +121,7 @@ http://code.google.com/p/subprocdev/source/browse/doc/subprocess.rst?spec=svn2c925e935cad0166d5da85e37c742d8e7f609de5&r=2c925e935cad0166d5da85e37c742d8e7f609de5#437 [9] subprocdev - Project Hosting on Google Code - http://code.google.com/p/subprocdev + https://code.google.com/p/subprocdev/source/list?name=python3k [10] Python Subprocess Dev http://subdev.blogspot.com//techtonik@gmail.com