Message 355881 - Python tracker (original) (raw)

From my understanding, the executor classes are designed around spawning the threads (or processes in the case of ProcessPoolExecutor) as needed up to max_workers, rather than spawning them upon startup. The asynchronous spawning of threads or processes would also not be compatible with the executor subclasses as far as I can tell.

I can start working on a draft/prototype for a design. It will likely take more time to implement this, but it will give us the chance to have a native asyncio ThreadPool that doesn't directly rely upon the API in concurrent.futures.

No, that would be too much work. Writing a thread pool or process pool from scratch is an extremely tedious task and it will take us years to stabilize the code. It's not simple.

We should design our API correctly though. And that means that we can't initialize our pools in init.

Something along these lines would work:

class ThreadPool:
  async def start(): ...
  async def __aenter__(self): 
    await self.start()
    return self
  
  async def aclose(): ...
  async def __aexit__(self, *exc):
    await self.aclose()

Let me know if you approve of this idea Yury and Andrew. It's quite a bit more involved than implementing a simple high level version of loop.run_in_executor(), but I think it would prove to be worthwhile in the long term.

It shouldn't be much harder than run_in_executor() as we continue to rely on concurrent.future (at least for the first version).

We need to start the discussion about this API on discourse. Please give me a few days to organize that.