[Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets (original) (raw)

Victor Stinner victor.stinner at gmail.com
Fri Jul 5 19:03:35 CEST 2013

2013/7/5 Cameron Simpson <cs at zip.com.au>:

| Both set ONONBLOCK flag (UNIX)

Oh, how embarassing.

You said that the PEP is not cristal clear. Do you have a suggestion to make it more clear?

Should I mention that the close-on-exec flag is O_CLOEXEC on UNIX, and HANDLE_FLAG_INHERIT on Windows? (except that HANDLE_FLAG_INHERIT set means inheritable, whereas O_CLOEXEC set means not inheritable)

| > This is deducable from your PEP, but I was at first confused, and | > initially expected getblocking/setblocking functions in New | > Functions. | | socket objects already have getblocking() and setblocking() methods.

Ah, ok then. As far as it goes. Shouldn't there be a general purpose os.getblocking() and os.setblocking() functions that operate on any file descriptor, paralleling os.getcloexec() and os.setcloexec()?

I didn't propose to add these two functions, because I'm not sure that they are really useful.

We can add os.get_blocking() and os.set_blocking() on UNIX, but these functions would not be available on Windows. On Windows, os.set_blocking() only makes sense for sockets, and sockets already have a set_blocking() method. On UNIX, it makes sense because set_blocking(fd, True) can be implemented as a single syscall on some platforms (using ioctl), whereas most developers implement it using two syscalls (fcntl to get current flags, and fcntl to set the O_NONBLOCK flag) maybe because it is more portable.

But we cannot add a new "blocking" parameter to all functions creating file descriptors, like open(), because of Windows. Or do you like the "raise NotImplementedError on Windows" option?

But WHY? I think socket file decriptors should be treated little differently from other file descriptors.

Because of Windows. On Windows, sockets and files are two different things. sockets have no "file descriptor", they have a HANDLE. For example, sockobj.fileno() on Windows returns a huge number, not something like 3 or 10.

The C type HANDLE is larger than a file descriptor on Windows 64-bit (sizeof(void*) > sizeof(int)) and so functions must be modified to use a wider type (to parse their argument), and must support the HANDLE type (_open_osfhandle() can be used to have a single version of the code).

Otherwise, I think it should be separated out into a separate proposal if you're proposing it just for sockets; make this proposal just close-on-exec and forget the blocking stuff for this specific PEP.

The reason for addressing close-on-exec and blocking parameters in the same PEP is that new versions of operating systems support setting the two flags at the creation a new file descriptor and a new socket. Read the article linked at the end of the PEP for the background: http://udrepper.livejournal.com/20407.html

The Python call socket.socket(..., cloexec=True, blocking=False) only calls one syscall on Linux >= 2.6.27.

| > I would expect Popen and friends to need to both clear the flag to | > get the descriptors across the fork() call, and possibly to set | > the flag again after the fork. Naively, I would expect the the flag | > to be as it was before the Popen call, after the call. | | As explained in the "Inheritance of file descriptors" section, the | close-on-exec flag has no effect on fork: | | "The flag has no effect on fork(), all file descriptors are inherited | by the child process."

Agreed, see my second post where I explained I'd misthought it. However, as stated there, I think the side effects should be fairly overtly stated in the docuemntation.

Sorry, I don't understand. Which "side effect"?

The close-on-exec flag only affects inheritance of file descriptors at the execv() syscall, not at fork(). And execv() can be called without fork().

Popen must clear close-on-exec flag on files from its pass_fds parameter for convinience. It would be surprising to not inherit a file descriptor passed to Popen in pass_fds, don't you think so?

os.execv() has no pass_fds parameter, and it is a thin wrapper on the syscall. Popen is a high-level API, that's why it prepares more things before calling the new program.

| The close-on-exec flag is cleared after the fork and before the | exec(). Pseudo-code for Popen: | | pid = os.fork() | if pid: | return pid | # child process | for fd in passfds: os.setcloexec(fd, False) | os.execv(...)

Fine. No state restore is fine with me. I think it should be as clear as possible in the doco.

I don't understand. I already wrote "The flag has no effect on fork(), all file descriptors are inherited by the child process" in the PEP. It is not enough? Do you have a suggestion to explain it better?

Victor

More information about the Python-Dev mailing list