[Python-Dev] PEP 446 (make FD non inheritable) ready for a final review (original) (raw)

Victor Stinner victor.stinner at gmail.com
Fri Aug 23 19:07:09 CEST 2013

Hi,

I will try to answer to your worries. Tell me if I should complete the PEP with these answers.

2013/8/23 Charles-François Natali <cf.natali at gmail.com>:

Why does dup2() create inheritable FD, and not dup()?

Ah yes, there were as section explaining it. In a previous version of the PEP (and its implementation), os.dup() and os.dup2() created non-inheritable FD, but inheritable FD for standard steams (fd 0, 1, 2).

I did a research on http://code.ohloh.net/ to see how os.dup2() is used in Python projects. 99.4% (169 projects on 170) uses os.dup2() to replace standard streams: file descriptors 0, 1, 2 (stdin, stdout, stderr); sometimes only stdout or stderr. I only found a short demo script using dup2() with arbitrary file descriptor numbers (10 and 11) to keep a copy of stdout and stderr before replacing them (also with dup2).

I didn't find use cases of dup() to inherit file descriptors in the Python standard library. It's the opposite: when os.dup() (or the C function dup() is used), the FD must not be inherited. For example, os.listdir(fd) duplicates fd: a child process must not inherit the duplicated file descriptor, it may lead a security vulnerability (ex: parent process running as root, whereas the child process is running as a different used and not allowed to open the directory).

For example, a lot of code uses the guarantee that dup()/open()... returns the lowest numbered file descriptor available, so code like this:

r, w = os.pipe() if os.fork() == 0: # child os.close(r) os.close(1) dup(w) will break And that's a lot of code (e.g. that's what posixsubprocess.c uses, but since it's implemented in C it's wouldn't be affected).

Yes, it will break. As I wrote in my previous email, we cannot solve all issues listed in the Rationale section of the PEP without breaking applications (or at least breaking backward compatibility). It is even explicitly said in the "Backward Compatibility" section: http://www.python.org/dev/peps/pep-0446/#backward-compatibility "This PEP break applications relying on inheritance of file descriptors."

But I also added a hint to fix applications: "Developers are encouraged to reuse the high-level Python module subprocess which handles the inheritance of file descriptors in a portable way."

If you don't want to use subprocess, yes, you will have to add "os.set_inheritable(w)" in the child process.

About your example: I'm not sure that it is reliable/portable. I saw daemon libraries closing all file descriptors and then expecting new file descriptors to become 0, 1 and 2. Your example is different because w is still open. On Windows, I have seen cases with only fd 0, 1, 2 open, and the next open() call gives the fd 10 or 13...

I'm optimistic and I expect that most Python applications and libraries already use the subprocess module. The subprocess module closes all file descriptors (except 0, 1, 2) since Python 3.2. Developers relying on the FD inheritance and using the subprocess with Python 3.2 or later already had to use the pass_fds parameter.

Furthermore, many people use Python for system programming, and this change would be highly surprising.

Yes, it is a voluntary design choice (of the PEP). It is also said explicitly in the "Backward Compatibility" section: "Python does no more conform to POSIX, since file descriptors are now made non-inheritable by default. Python was not designed to conform to POSIX, but was designed to develop portable applications."

So no matter what the final decision on this PEP is, it must be kept in mind.

The purpose of the PEP is to explain correctly the context and the consequences of the changes, so Guido van Rossum can uses the PEP to make its final decision.

The programming languages Go, Perl and Ruby make newly created file descriptors non-inheritable by default: since Go 1.0 (2009), Perl 1.0 (1987) and Ruby 2.0 (2013). OK, but do they expose OS file descriptors?

Yes:

Last time, I said that to me, the FD inheritance issue is solved on POSIX by the subprocess module which passes closefds. In my own code, I use subprocess, which is the "official", portable and safe way to create child processes in Python. Someone using fork() + exec() should know what he's doing, and be able to deal with the consequences: I'm not only talking about FD inheritance, but also about async-signal/multi-threaded safety ;-)

The subprocess module has still a (minor?) race condition in the child process. Another C thread can create a new file descriptor after the subprocess module closed all file descriptors and before exec(). I hope that it is very unlikely, but it can happen. It's also explained in the PEP (see "Closing All Open File Descriptors"). I suppose that the race condition explains why Linux still has no closefrom() or nextfd() system calls. IMO the kernel is the best place to decide which FD should be kept, and which must not be inherited (must be closed), in the child process. I like the close-on-exec flag (and HANDLE_FLAG_INHERIT on Windows).

As for Windows, since it doesn't have fork(), it would make sense to make its FD non heritable by default.

As said in the "Inheritance of File Descriptors on Windows" section, Python gets inheritable handles and file descriptors because it does not use the Windows native API. Applications developed for Windows using the native API only create non-inheritable handles, and so don't have all these annoying inheritance issues.

Windows does not have a fork() function, but handles and file descriptors can be inherited using CreateProcess() and spawn(), see the table in the "Status of Python 3.3" section.

And then use what you describe here to selectively inherit FDs (i.e. implement keepfds): """ Since Windows Vista, CreateProcess() supports an extension of the STARTUPINFO struture: the STARTUPINFOEX structure. Using this new structure, it is possible to specify a list of handles to inherit: PROCTHREADATTRIBUTEHANDLELIST. Read Programmatically controlling which handles are inherited by new processes in Win32 (Raymond Chen, Dec 2011) for more information. """

This feature can only be used to inherit handles, and it does not need of an intermediate program. If you want to inherit only some file descriptors, you need an intermediate program which will recreates them and then use spawn() (or directly the reserved fields of the STARTUPINFO structure). Using an intermediate program has unexpected consequences, so I prefer to use this option.

The feature is also specific to Windows 7, a recent Windows version, whereas there are still millions of Windows XP installations (and I read that Python 3.4 will still supported Windows XP!).

We need more feedback from users to know their use cases before chosing the best implementation of pass_handles/pass_fds on Windows.

Richard agreed that this point can be deferred.

Victor

More information about the Python-Dev mailing list