[Python-Dev] Another threading idea (original) (raw)

Tim Peters tim.peters at gmail.com
Wed Mar 15 00:24:56 CET 2006

[Raymond Hettinger]

FWIW, I've been working on a way to simplify the use of queues with daemon consumer threads

Sometimes, I launch one or more consumer threads that wait for a task to enter a queue and then work on the task. A recurring problem is that I sometimes need to know if all of the tasks have been completed so I can exit or do something with the result.

FWIW, instead of:

main thread

q = TaskQueue() for t in worker_threads(): t.start() for task in tasklist: q.put(task) # increments the counter and enqueues a task q.join() # all of the tasks are done (counter is zero) do_work_on_results()

I've sometimes used a separate "work finished" queue, like so:

main thread

q = TaskQueue() finished = Queue.Queue() for t in worker_threads(): t.start() for task in tasklist: q.put(task) for task in tasklist: finished.get() do_work_on_results()

When a worker thread is done with a task, it simply does:

finished.put(None) # None can just as well be 42 or "done"

No explicit count is needed, although it's easy to add one if desired. The only trick is for the main thread to insist on doing finished.get() exactly as many times as it does q.put().

This is easy and natural -- once you've seen it :-) BTW, it's more common for me to put something meaningful on the finished queue, as my main thread often wants to accumulate some info about the outcomes of tasks.

More information about the Python-Dev mailing list