Issue 4855: Popen(..., shell=True,...) should allow simple access to the command's PID (original) (raw)

When using proc = Popen() with the parameter "shell=True", the returned proc-object's Process ID (PID) obtained by reading "proc.pid" is the one of the shell that was created.

This gets very cumbersome, when the newly created process shall be signalled later on. It is very rare that one needs to signal the shell instead of the newly started executable.

Moreover, sending a signal to the shell does (for some reason) not get propagated to the child (=the executable/command), so the shell is quit/ terminated/killed/whatever and the command's thread is kept alive.

Using 'shell=False' is not an option in case the originating process has more code to execute, e.g. in order to communicate with the newly started process via pipes (i.e. using proc.stdout.readline() or similar), because in that case Python uses os.execvp() for the command, which re-uses (and therefore blocks) the current thread of control in which the Popen was issued.

Starting another thread with fork() is also not an option, as the Popen will be executed in a different thread, which makes it difficult to pass the reference to the proc-object back into the original thread where it is needed for the inter-process-communication.

All examples I found on the net which use pipes to communicate between current and newly started command use of course 'shell=True', I did not however find any simple way to get the child's PID (aside from issuing a 'ps --ppid -o pid=' and reading it's output, as I did, which is far from elegant).

So, there should be one of these alternative possibilities/behaviours in case of 'Popen(...,shell=True...)':

  1. proc gets a new field or method child_pid/child_pid() which returns the command's PID. proc.pid behaves as before
  2. Signalling the shell's process notifies all children with the same signal. proc.pid behaves as before
  3. proc.pid returns the command's PID (instead of the shell's)
  4. There is some simple command to query all the childs PIDs of a process when passing in the PPID (parent PID). proc.pid behaves as before

...with 1) being the most up to 4) being the least desireable.

I hope I did not overlook any such possibility, but searching around on the net and in the tracker turned up nothing useful.

I really do think this should be more intuitive.

This really belongs to python-list rather than the tracker. It is not correct that with shell=False Popen reuses the thread of control of the calling process. You seem to be confusing blocking and reusing the thread of control. Popen always creates a new thread of control using a fork() call. The proc.stdout.readline() will block until the process produces a full line of output, which is what anyone would expect.

It is not clear what you are trying to achieve. Here is an example session:

from subprocess import * p = Popen(['sleep', '10000']) p.kill() p.wait() -9

or using a different signal (SIGINT = 2):

p = Popen(['sleep', '10000']) p.send_signal(2) p.wait() -2

The wait function blocks until the process terminates and returns the status (negative of the signal number in case of exit on signal). What else are you trying to achieve?

Your proposals don't make much sense:

  1. "child_pid()" - what should it return if the shell command spawns more than one process? For example if the command contains |'s or multiple &'s?

  2. "Signalling the children of the shell" - read the manual page for your shell. Foreground processes receive most of the signals that way.
    Background processes only get a SIGHUP.

  3. "proc.pid returns the command's PID" - there is no such thing: a command can create any number of processes.

  4. "simple command to query all the childs PIDs" - without reimplementing shell in python to parse the command string, it is impossible to tell which subprocesses are spawned by a given Popen call.
    A function that returns all children of the given process can be written, but will be expensive on most systems because it will require a
    search over the entire process table.

In short, this report is invalid.