Issue 29733: concurrent.futures as_completed raise TimeoutError wrong (original) (raw)

when I use as_completed function to wait my futures, if I sleep more than timeout seconds in each iteration , I found that futures has been set result, but raise TimeoutError. as my test example code shows:

from concurrent import futures from multiprocessing import current_process import time

def run(count): cp = current_process() print(cp.name, 'begin', count, 'at', time.time()) time.sleep(count) print(cp.name, 'end', count, 'at', time.time()) return count

if name == 'main': ppe = futures.ProcessPoolExecutor(max_workers=4) cp = current_process()

fs = [ppe.submit(run, i) for i in range(4)]

print('begin receive at', time.time())
for f in futures.as_completed(fs, timeout=5):
    time.sleep(5)
    print(cp.name, 'receive', f.result(), 'at', time.time())
    print(cp.name, 'receive', [f.result() for f in fs], 'at', time.time())
print('end receive at', time.time())

run above-mentioned example code, it will output : begin receive at 1488799136.471536 Process-1 begin 0 at 1488799136.472969 Process-1 end 0 at 1488799136.473114 Process-3 begin 1 at 1488799136.473741 Process-2 begin 2 at 1488799136.474226 Process-4 begin 3 at 1488799136.474561 Process-3 end 1 at 1488799137.474495 Process-2 end 2 at 1488799138.475289 Process-4 end 3 at 1488799139.475696 MainProcess receive 0 at 1488799141.478663 MainProcess receive [0, 1, 2, 3] at 1488799141.478787 Traceback (most recent call last): File "test_futures.py", line 23, in for f in futures.as_completed(fs, timeout=5): File "/Users/jiangwanwei/anaconda3/lib/python3.5/concurrent/futures/_base.py", line 213, in as_completed len(pending), len(fs))) concurrent.futures._base.TimeoutError: 3 (of 4) futures unfinished

The docs ( https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.as_completed ) do seem to indicate it shouldn't do so as long as results were available before the timeout expired:

"The returned iterator raises a concurrent.futures.TimeoutError if next() is called and the result isn’t available after timeout seconds from the original call to as_completed()."

My reading of that would be that it raises the error only when:

1. The timeout has expired
2. The call would block (or possibly, would have blocked after the timeout expired), indicating no result was available

Handling "would have blocked" is hard, but might it make sense to still allow a non-blocking wait on the event even if the timeout has expired, with the exception raised only if the non-blocking wait fails?

Side-note: Looks like this code is still using time.time, not time.monotonic, so it's vulnerable to system clock adjustments; NTP updates could cause a five second timeout to expire instantly, or take seconds or even minutes longer to expire.