| msg292346 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2017-04-26 13:52 |
| A multiprocessing Queue object managers multiple resources: * a multiprocessing Pipe * a thread * (a lock and a semaphore) If a Queue is not cleaned up properly, your application may leak many resources. Try attached queue_leak.py to see an example "leaking a thread". I suggest to emit a ResourceWarning warning in Queue destrutor. I don't know what should be the test to decide if a warning must be emitted? * if the queue wasn't closed yet? * if the thread is alive? * if the queue wasn't closed yet and/or the thread is alive? (my favorite choice) Other examples of objects emitting ResourceWarning: * io files: io.FileIO, io.TextIOWrapper, etc. * socket.socket * subprocess.Popen: I recently added a ResourceWarning on that one * asyncio transports and event loops |
|
|
| msg292347 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-04-26 13:54 |
| Example: haypo@selma$ ./python queue_leak.py number of thread diff: +1 dangling threads! before: [<_MainThread(MainThread, started 139814961067072)>] after: [<_MainThread(MainThread, started 139814961067072)>] Note: queue_leak.py resource check is based on test.support.reap_threads. |
|
|
| msg292348 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-04-26 13:55 |
| Oh, I forgot that I hitted this issue while analyzing issue #30131: test_logging leaks a "dangling" thread. It took me a while to find multiprocessing queues in the big test_logging file! |
|
|
| msg292850 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-05-03 07:18 |
| See also issue #30244: Emit a ResourceWarning in concurrent.futures executor destructors. |
|
|
| msg293001 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2017-05-04 17:03 |
| The thread seems to be stopped when the Queue object is finalized: # Send sentinel to the thread queue object when garbage collected self._close = Finalize( self, Queue._finalize_close, [self._buffer, self._notempty], exitpriority=10 ) I don't think the other resources (pipe, lock, semaphore) need explicit cleaning. |
|
|
| msg298015 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-07-10 00:31 |
| Another example: http://bugs.python.org/issue30886#msg298014 "The problem is that multiprocessing.Queue.join_thread() does nothing since the thread wasn't started by a subprocess." |
|
|
| msg298034 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2017-07-10 08:56 |
| > The problem is that multiprocessing.Queue.join_thread() does nothing since the thread wasn't started by a subprocess. I don't understand what this means. Can you clarify a bit? |
|
|
| msg298035 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2017-07-10 08:56 |
| Specifically "the thread wasn't started by a subprocess"... |
|
|
| msg298036 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-07-10 09:15 |
| > Specifically "the thread wasn't started by a subprocess"... I'm talking about this check in Queue._start_thread() of multiprocessing.queues: created_by_this_process = (self._opid == os.getpid()) if not self._joincancelled and not created_by_this_process: self._jointhread = Finalize( self._thread, Queue._finalize_join, [weakref.ref(self._thread)], exitpriority=-5 ) |
|
|
| msg298039 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-07-10 09:28 |
| Let's discuss created_by_this_process in bpo-30886. This issue is more about adding or not a ResourceWarning. |
|
|
| msg298045 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2017-07-10 09:50 |
| I don't think a ResourceWarning should be emitted. There is no risk of data loss or resource leak if you don't close a multiprocessing Queue explicitly. Actually, in the real world, I don't think I've ever seen code that closes queues explicitly. |
|
|
| msg298662 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2017-07-19 10:17 |
| I'm willing to close this issue, as I don't think a ResourceWarning is appropriate here. |
|
|