[Python-Dev] Status of the PEP 433? (original) (raw)

Victor Stinner victor.stinner at gmail.com
Thu Feb 14 14:12:39 CET 2013

Hi,

I read again all emails related to PEP 433. There are different opposition to this change and I tried to make a summary.

My question is: how can I move this PEP forward (accept, defer or reject it)? Can I accept it myself, or do we need a global agreement, or should someone else decide for this PEP (who? Antoine maybe)?

--

Most complains concern issues if cloexec is set by default. I don't understand why so many people feel concerned because my PEP proposes to not change anything to the default behaviour: cloexec will not be set by default, as it was always the case since Python 1.

You have to explicitly use -e command line option, PYTHONCLOEXEC=1 environment variable, or call sys.setdefaultcloexec(True). If you explicitly change the default value of the cloexec parameter, you must expect to change the behaviour of your application and maybe discover "bugs" in your application (application incompatible with cloexec set by default).

(I don't think that any library would like to play with sys.setdefaultcloexec(), since most developers agree that it might break applications.)

--

Ok, let's see all oppositions, presented as a FAQ. (I hope that I didn't misuse email quotes.)

Q: the PEP does not solve file descriptor inherance after fork() A: Correct, it only solves issues with exec(), as explained in the PEP.

Q: the PEP changes the POSIX behaviour. A: Antoine replied that "Python is not POSIX", Nick wrote "While many of Python's APIs are heavily inspired by POSIX, it still isn't POSIX" (he was talking about integer division). Perl sets cloexec flag by default since its first version, something like 25 years ago, except for explicit POSIX::* calls. Ruby 2 will also set cloexec flag by default.

Q: The PEP may break applications. A: Most developers agree that it is (very) unlikely. If file descriptor inherance matters, subprocess must be used (because it rocks!) with pass_fds. If subprocess is used without pass_fds, applications stops working since python 3.2 (since python 3.0 on Windows). pass_fds will clear the close-on-exec flag on the file descriptors with the PEP. If an application breaks because it relies on file descriptor inherance, it's easy to detect it (EBADF error) and simple to fix it (ex: just call sys.setdefaultcloexec(False) at startup, or set explicitly cloexec parameter on the few functions causing trouble).

Q: Performance overhead if cloexec is set by default (extra syscalls). A: The PEP now contains a benchmark: the overhead on a microbenchmark is between 1 and 3%. I bet that nobody will be able to notice a difference on a real application.

Q: The default value of cloexec must not be configurable: "libraries cannot rely on it and have to make file descriptors cloexec on an individual basis, since the default flag can be disabled". A: IMO inherance of file descriptor doesn't matter in most cases, so only a few function must be modified to explicitly set or clear cloexec flag. Said differently: libraries should not care of the default value of the cloexec parameter, which should only be configured by applications (not libraries).

Q: The default value of cloexec must not be configurable: "just by looking at this code, you have no way to know whether the file descriptor will be inherited by the child process." A: Guido wrote "Honestly, what happened to the idea that we're all adults here? The likelihood that someone is clueless enough to misuse the flag and yet clueful enough to find out how to change it seems remote -- and they pretty much deserve what they get."

--

Another alternative (not listed in the PEP yet) is to leave the posix module (its name depends on the platform: posix, nt or ce) unchanged, and only modify the os module to add cloexec parameter.

I implemented this alternative in the "posixmod" branch of the PEP 433 repository. It gives a better traceback on error because the cloexec flag is set in Python, so it's possible to know if the error comes from the creation of the file descriptor or setting the flag in the traceback. (Does it really matter? I never seen an error on setting or clearing the flag.)

The problem is that it only concernes posix and os modules: the PEP does also modify socket, io and select modules, which don't have "low" and "high" level modules. So it's possible possible to create a socket using the POSIX behaviour. (Does it matter, it was said that Python is not POSIX.)

I'm not convinced that we should do something special for posix/os, I prefer the actual version of the PEP (modify also posix).

Victor

More information about the Python-Dev mailing list