[Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) (original) (raw)

Victor Stinner victor.stinner at gmail.com
Fri Jul 26 14:08:35 CEST 2013

2013/7/24 Guido van Rossum <guido at python.org>:

But I'm also ready to propose that all this is such a mess that we should change the default fd/handle inheritance to False, *across platforms*, and damn the torpedoes -- i.e. accept breaking all existing 3rd party UNIX code for subprocess creation that bypasses the subprocess module, as well as breaking uses of os.spawn*() on both platforms that depend on FD inheritance beyond stdin/stdout/stderr).

With the new, sane default, all we need instead of PEP 446 is a way to make an FD inheritable after it's been created, which can be a single os.makeinheritable(fd) call that you must apply to the fileno() of the stream or socket object you want inherited (or directly to a FD you created otherwise, e.g. with os.pipe()). On Windows, this should probably only work with os.spawn*(), since otherwise you need handle inheritance, not FD inheritance, and that's a non-portable concept anyway.

After having written 2 PEP on the topic, I slowly agree that make all file descriptors non-inheritable is the best compromise. It solves most, or all, issues.

The main drawback is the additionnal syscalls: on some platforms, 2 additional syscalls are need to make a file descriptor non-inheritable for each creation of file descriptor. According to my benchmark on the implementation of the PEP 433: the overhead of making a file descriptor non-inheritable is between 1% and 3% (7.8 µs => 7.9 or 8.0 µs) on Linux 3.6. http://www.python.org/dev/peps/pep-0433/#performances

Having to make a file descriptor inheritable after creating it non-inheritable is also not optimal. Making it first non-inheritable requires 0, 1 or 2 extra syscalls, and making it inheritable again require also 1 or 2 syscalls. So f=open(...); os.make_inheritable(f.fileno()) can take up to 5 syscalls (1 open + 4 fnctl), whereas it can be done in only 1 syscall (1 open). One of the motivation of the PEP 433 an 446 is to reduce the number of syscalls, even if the use case was to make sockets non-inheritable.

If we consider that the most common case is to use non-inheritable file descriptors, having to call os.make_inheritable() may be acceptable. Windows and recent operating syscalls support creating file descriptor directly non-inheritable in a single syscalls. ioctl() can be also be used instead of fcntl() to use 1 syscall instead of 2.

We can fix multiprocessing any anything else in the stdlib that this breaks, I presume.

The CGI code rely on inheritance of file descriptors 0, 1 and 2 which are pipes. The file descriptors 0, 1 and 2 are replaced with the pipes using os.dup2().

Victor

More information about the Python-Dev mailing list