[Python-Dev] futures API (original) (raw)

Brian Quinlan brian at sweetapp.com
Fri Dec 10 20:24:42 CET 2010

On Dec 10, 2010, at 10:51 AM, Thomas Nagy wrote:

--- El vie, 10/12/10, Brian Quinlan escribió:

On Dec 10, 2010, at 5:36 AM, Thomas Nagy wrote:

I have a process running for a long time, and which may use futures of different maxworkers count. I think it is not too far-fetched to create a new futures object each time. Yet, the execution becomes slower after each call, for example with http://freehackers.org/~tnagy/futurestest.py:

""" import concurrent.futures from queue import Queue import datetime class counter(object): def init(self, fut): self.fut = fut def run(self): def lookbusy(num, obj): tot = 0 for x in range(num): tot += x obj.outq.put(tot) start = datetime.datetime.utcnow() self.count = 0 self.outq = Queue(0) for x in range(1000): self.count += 1 self.fut.submit(lookbusy, self.count, self) while self.count: self.count -= 1 self.outq.get() delta = datetime.datetime.utcnow() - start print(delta.totalseconds()) fut = concurrent.futures.ThreadPoolExecutor(maxworkers=20) for x in range(100): # comment the following line fut = concurrent.futures.ThreadPoolExecutor(maxworkers=20) c = counter(fut) c.run() """ The runtime grows after each step: 0.216451 0.225186 0.223725 0.222274 0.230964 0.240531 0.24137 0.252393 0.249948 0.257153 ... Is there a mistake in this piece of code? There is no mistake that I can see but I suspect that the circular references that you are building are causing the ThreadPoolExecutor to take a long time to be collected. Try adding: c = counter(fut) c.run() + fut.shutdown() Even if that fixes your problem, I still don't fully understand this because I would expect the runtime to fall after a while as ThreadPoolExecutors are collected. The shutdown call is indeed a good fix :-) Here is the time response of the calls to counter() when shutdown is not called: http://www.freehackers.org/~tnagy/runtimefutures.png

FWIW, I think that you are confusion the term "future" with
"executor". A future represents a single work item. An executor
creates futures and schedules their underlying work.

Hmmm....that is very suspicious - it looks like the
ThreadPoolExecutors are not being collected. If you are feeling bored
you could figure out why not :-)

After trying to stop the program by using CTRL+C, the following error may appear, after which the process cannot be interrupted:

""" 19🔞12 /tmp/build> python3.2 futurestest.py 0.389657 0.417173 0.416513 0.421424 0.449666 0.482273 ^CTraceback (most recent call last): File "futurestest.py", line 36, in c.run() File "futurestest.py", line 22, in run self.fut.submit(lookbusy, self.count, self) File "/usr/local/lib/python3.2/concurrent/futures/thread.py", line 114, in submit self.workqueue.put(w) File "/usr/local/lib/python3.2/queue.py", line 135, in put self.notfull.acquire() KeyboardInterrupt """ It is not expected, is it?

It isn't surprising. Python lock acquisitions are not interruptible
and anytime you interrupt a program that manipulates locks you may
kill the code that was going to cause the lock to be released.

Cheers, Brian

Thomas

More information about the Python-Dev mailing list