[Python-Dev] futures API (original) (raw)

Thomas Nagy tnagyemail-mail at yahoo.fr
Fri Dec 10 19:51:11 CET 2010

--- El vie, 10/12/10, Brian Quinlan escribió:

On Dec 10, 2010, at 5:36 AM, Thomas Nagy wrote: > I have a process running for a long time, and which may use futures of different maxworkers count. I think it is not too far-fetched to create a new futures object each time. Yet, the execution becomes slower after each call, for example with http://freehackers.org/~tnagy/futurestest.py: > > """ > import concurrent.futures > from queue import Queue > import datetime > > class counter(object): >     def init(self, fut): >         self.fut = fut > >     def run(self): >         def lookbusy(num, obj): >   tot = 0 >   for x in range(num): >   tot += x >   obj.outq.put(tot) > >         start = datetime.datetime.utcnow() >         self.count = 0 >         self.outq = Queue(0) >         for x in range(1000): >   self.count += 1 >   self.fut.submit(lookbusy, self.count, self) > >         while self.count: >   self.count -= 1 >   self.outq.get() > >         delta = datetime.datetime.utcnow() - start >   print(delta.totalseconds()) > > fut = concurrent.futures.ThreadPoolExecutor(maxworkers=20) > for x in range(100): >     # comment the following line >     fut = concurrent.futures.ThreadPoolExecutor(maxworkers=20) >     c = counter(fut) >     c.run() > """ > > The runtime grows after each step: > 0.216451 > 0.225186 > 0.223725 > 0.222274 > 0.230964 > 0.240531 > 0.24137 > 0.252393 > 0.249948 > 0.257153 > ... > > Is there a mistake in this piece of code?

There is no mistake that I can see but I suspect that the circular references that you are building are causing the ThreadPoolExecutor to take a long time to be collected. Try adding: c = counter(fut) c.run() +    fut.shutdown() Even if that fixes your problem, I still don't fully understand this because I would expect the runtime to fall after a while as ThreadPoolExecutors are collected.

The shutdown call is indeed a good fix :-) Here is the time response of the calls to counter() when shutdown is not called: http://www.freehackers.org/~tnagy/runtime_futures.png

After trying to stop the program by using CTRL+C, the following error may appear, after which the process cannot be interrupted:

""" 19🔞12 /tmp/build> python3.2 futures_test.py 0.389657 0.417173 0.416513 0.421424 0.449666 0.482273 ^CTraceback (most recent call last):   File "futures_test.py", line 36, in     c.run()   File "futures_test.py", line 22, in run     self.fut.submit(look_busy, self.count, self)   File "/usr/local/lib/python3.2/concurrent/futures/thread.py", line 114, in submit     self._work_queue.put(w)   File "/usr/local/lib/python3.2/queue.py", line 135, in put     self.not_full.acquire() KeyboardInterrupt """

It is not expected, is it?

Thomas

More information about the Python-Dev mailing list