[Python-Dev] PEP 324: popen5 - New POSIX process module (original) (raw)

Peter Astrand astrand at lysator.liu.se
Sat Jan 3 17:27:12 EST 2004

On Sat, 3 Jan 2004, Martin v. Loewis wrote:

> - One "unified" module provides all functionality from previous > functions.

I doubt this is a good thing. Different applications have different needs - having different API for them is reasonable.

I don't agree. I have used all of the existing mechanism in lots of apps, and it's just a pain. There are lots of functions to choose between, but none does what you really want.

> - Cross-process exceptions: Exceptions happening in the child > before the new process has started to execute are re-raised in > the parent. This means that it's easy to handle exec() > failures, for example. With popen2, for example, it's > impossible to detect if the execution failed.

This is a bug in popen2, IMO. Fixing it is a good thing, but does not require a new module.

"Fixing popen2" would mean a break old applications; exceptions will happen, which apps are not prepared of.

> - A hook for executing custom code between fork and exec. This > can be used for, for example, changing uid.

Such a hook could be merged as a keyword argument into the existing API.

Into which module/method/function? There is no one flexible enough. The case for redirecting only stderr is just one example; this is simple not possible with the current API.

> - All combinations of file descriptor redirection is possible. > For example, the "python-dialog" [2] needs to spawn a process > and redirect stderr, but not stdout. This is not possible with > current functions, without using temporary files.

Sounds like a new function on the popen2 module.

To support all combinations, 12 different functions are necessary. Who will remember what popen2.popen11() means?

> - Support for connecting several subprocesses (shell "pipe").

Isn't this available already, as the shell supports pipe creation, anyway?

With popen5, you can do it without using the shell.

> - Universal newline support.

This should be merged into the existing code.

There's already a bug about this; bug 788035. This is what one of the comment says:

"But this whole popen{,2,3,4} section of posixmodule.c is so fiendishly complicated with all the platform special cases that I'm loath to touch it..."

I haven't checked if this is really true, though.

> - A communicate() method, which makes it easy to send stdin data > and read stdout and stderr data, without risking deadlocks. > Most people are aware of the flow control issues involved with > child process communication, but not all have the patience or > skills to write a fully correct and deadlock-free select loop.

Isn't asyncore supposed to simplify that?

Probably not. The description says:

"This module provides the basic infrastructure for writing asynchronous socket service clients and servers."

It's not obvious to me how this module could be use as a "shell backquote"
replacement (which is what communicate() is about). It's probably possible though; I haven't tried. Even if this is possible I guess we need some kind of "entry" or "wrapper" method in the popen module to simplify things for the user. My guess is that an communicate() method that uses asyncore would be as long/complicated as the current implementation. The current implementation is only 68 lines, including comments.

So in short, I'm -1 on creating a new module, but +1 on merging most of these features into the existing code base - they are good features.

Well, I don't see how this could be done easily: The current API is not flexible enough, and some things (like cross-process exceptions) breaks compatibility.

Writing a good popen module is hard. Providing cross-platform support (for Windows, for example) is even harder. Trying to retrofit a good popen implementation into an old API without breaking compatibility seems impossible to me. I'm not prepared to try.

-- /Peter Åstrand <astrand at lysator.liu.se>

More information about the Python-Dev mailing list