[Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446) (original) (raw)

Victor Stinner victor.stinner at gmail.com
Wed Jul 24 01:40:18 CEST 2013

The multiprocessing module is an example of use case relying on inherance of handles. It calls CreateProcess() with bInheritHandles=TRUE to share a pipe between the manager (parent) and the worker (child process).

Note: subprocess and multiprocess have their own function to set the HANDLE_FLAG_INHERIT flag: they use DuplicateHandle(), whereas SetHandleInformation() could be used (to reuse the existing handle instead of creating a new handle).

2013/7/24 Victor Stinner <victor.stinner at gmail.com>:

Python functions open(), os.open() and os.dup() create file descriptors with the HANDLEFLAGINHERIT flag set (cloexec=False), whereas os.pipe() creates 2 file descriptors with the HANDLEFLAGINHERIT flag unset (cloexec=False, see also issue #4708). (...) If closefds=False, handles with the HANDLEFLAGINHERIT flag set are inherited, but all file descriptors are still closed except 0, 1 and 2.

Leaking handles in child processes is also an issue on Windows. Random examples.

http://bugs.python.org/issue17634 "Win32: shutil.copy leaks file handles to child processes" "Win32's native CopyFile API call doesn't leak file handles to child processes."

http://ghc.haskell.org/trac/ghc/ticket/2650 "The case in which I originally ran into this was System.Directory.copyFile intermittently reporting a "permission denied" error for a temp file it was using. I think it was trying to delete it, but failing because a child process somewhere was hanging on to the Handle." According to the issue, GHC calls CreateProcess with bInheritHandles=TRUE (as Python did until Python 3.2).

http://support.microsoft.com/kb/315939 "This behavior can occur if two threads simultaneously create child processes and redirect the STD handles through pipes. In this scenario, there is a race condition during the creation of the pipes and processes, in which it is possible for one child to inherit file handles intended for the other child." => Python looks to be correct, it uses the (StartupInfo) "STARTF_USESTDHANDLES" flag

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6428742 Java is still calling CreateProcess() with bInheritHandles=TRUE which causes issues like "6347873: (so) Ports opened with ServerSocketChannel blocks when using Runtime.exec". Interesting comment: "Submitter has a point. Very risky to fix." :-/

For the record, the default value of close_fds parameter was also set to True on Windows by the following changeset:

changeset: 66889:59b43dc34158 user: Gregory P. Smith <greg at mad-scientist.com> date: Tue Dec 14 13:43:30 2010 +0000 files: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py Misc/NEWS Modules/_posixsubprocess.c description: Issue #6559: fix the subprocess.Popen pass_fds implementation. Add a unittest. Issue #7213: Change the close_fds default on Windows to better match the new default on POSIX. True when possible (False if stdin/stdout/stderr are supplied).

Update the documentation to reflect all of the above.

The changeset was written to fix http://bugs.python.org/issue7213 ; another example of leak of handles.

The PEP 466 allows you to control which handles are inherited to child process when you use subprocess with closefds=False. (The subprocess parameter should be called "closehandles" on Windows to avoid confusion.)

Another advantage of the PEP 446 is that most of the time, the HANDLE_FLAG_INHERIT flag value can be set atomatically at the creation of the file descriptor (creation of the handle). It is a nice enhancement to fight against race conditions with threads ;-) "Most of the time": for example, socket are inherited by default, WSA_FLAG_NO_HANDLE_INHERIT flag was only added to Windows Vista.

Victor

More information about the Python-Dev mailing list