Message 346369 - Python tracker (original) (raw)

Ben:

Concretely, this is a concern for Tornado (which requires add_reader()) and applications in the scientific python community (including Jupyter) which depend on it.

If you need add_reader/add_writer in Python 3.8, you can switch the default event loop to selector at the beginning of your application:

if sys.platform == 'win32': asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

Yury:

If there's no native API for that, I guess we can spawn a thread with a 'select()' call to emulate this API?

There is no native API because IOCP design is to run an asynchronous read/write and then wait for its completion. Unix select() has the opposite design: check if a file descriptor is read for read/write.

Reimplementing add_reader/add_writer using a single select() call in a thread sounds like a good idea, but we have to make sure that it can be stopped whenever using a dedicated "self-pipe" (to awake the blocked select(), so loop.close() can stop this thread.

See attached proof-of-concept: selector_thread.py: run a selector in a separated thread which pass pack events to the loop using call_soon().

I would prefer to use a single selector to better scale with the number of FD.

Note: On Windows, select() only supports sockets.

Note: select.select() may be extended to support more than 512 sockets on Windows, see bpo-28708 :-)

Yury:

Another question: if we fix this, would you allow this to go in beta2/3? Strictly speaking it's going to be a new functionality.

It's a new feature, so it can wait for Python 3.9 :-)

I don't see any regression here, as soon as you can opt-in for SelectorEventLoop.