[Python-Dev] epoll implementation (original) (raw)

Ross Cohen rcohen at snurgle.org
Sat May 27 09:46:49 CEST 2006

On Sat, May 27, 2006 at 02:27:20AM +0100, Steve Holden wrote:

Greg Ewing wrote: > Rather than adding yet another platform-dependent module, > I'd like to see a unified Python interface in the stdlib > that uses whichever is the best one available. > Of course that would mean establishing which was the best available which, as we've seen this week, may not be easy.

Lots of systems (Windows? what's that?) implement both select and poll. I have yet to see a system which implements more than 1 of epoll, kqueue, /dev/poll, I/O completion ports, etc.

You'd have to search long and hard for a case where the non-select/poll interfaces didn't perform better. Correctness/completeness is, of course, a stronger criterion than performance, so if kqueue on OS X doesn't support all relevant types of file descriptors, then it shouldn't used.

Fortunately, the python select.poll interface is a good match for all of epoll, kqueue and /dev/poll. kqueue supports watching a lot more than file descriptors, but that doesn't matter much for this discussion. kqueue also mashes register/unregister/poll into a single call, which contributes to how confusing the interface is, though from what I can tell it really is basically the same as epoll.

As for Windows, the WSASelect interface is truly awful. The python library currently sets it so you can pass in up to 512 descriptors, but don't count on getting even that many. There is a hard limit on the number of sockets a user can have open (varies based on OS version) with WSASelect and it will just start returning WSAENOBUFS once you go over. You then have the task of guessing how many you need to close before it will start working again. Firewalls especially can eat into the pool.

IOCP has neither the WSAENOBUFS problem nor the 512 descriptor limit. It would be nice (if you or your customers run Windows) if someone did a select.poll interface with IOCP as a backend.

Ross

More information about the Python-Dev mailing list