| msg298645 - (view) |
Author: Armin Rigo (arigo) *  |
Date: 2017-07-19 08:53 |
| multiprocessing.queues.SimpleQueue should have a close() method. This is needed to explicitly release the two file descriptors of the Pipe used internally. Without it, the file descriptors leak if a reference to the SimpleQueue object happens to stay around for longer than expected (e.g. in a reference cycle, or with PyPy). I think the following would do: diff -r 0b72fd1a7641 lib-python/2.7/multiprocessing/queues.py --- a/lib-python/2.7/multiprocessing/queues.py Sun Jul 16 13:41:28 2017 +0200 +++ b/lib-python/2.7/multiprocessing/queues.py Wed Jul 19 10:45:03 2017 +0200 @@ -358,6 +358,11 @@ self._wlock = Lock() self._make_methods() + def close(self): + # PyPy extension: CPython doesn't have this method! + self._reader.close() + self._writer.close() + def empty(self): return not self._reader.poll() |
|
|
| msg298649 - (view) |
Author: Xiang Zhang (xiang.zhang) * |
Date: 2017-07-19 09:44 |
| Get this solved then we could also solve #23267, which reports pipes leak in pool due to using SimpleQueue. |
|
|
| msg298651 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-07-19 09:46 |
| On Python 3.6+, regrtest is able to detect file descriptor leaks when using --huntrleaks. Is someone interested to backport this feature to 3.5 and/or 2.7 which would mean fix all FD leaks in our test suite? If someone wants to work on that, I would suggest to first fix all bugs, and when the test suite is fine: modify regrtest. |
|
|
| msg298657 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-07-19 10:06 |
| See also my bpo-30171: "Emit ResourceWarning in multiprocessing Queue destructor". |
|
|
| msg298660 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-07-19 10:14 |
| This issue is marked as also affecting Python 3.7. I don't see a SimpleQueue.close() method in master, but I don't remind any resource warning when running tests on master neither. Does it mean that our test runner miss such file descriptor leak? Or worse, that SimpleQueue is not tested? I see a TestSimpleQueue test case in Lib/test/_test_multiprocessing.py. haypo@selma$ ./python -m test test_multiprocessing_spawn -m TestSimpleQueue -R 3:3 (...) Tests result: SUCCESS > This is needed to explicitly release the two file descriptors of the Pipe used internally. Without it, the file descriptors leak if a reference to the SimpleQueue object happens to stay around for longer than expected (e.g. in a reference cycle, or with PyPy). Oh ok, the garbage collector is able to close the file descriptors. The bug is when a SimpleQueue is not collected. So again, I consider that a ResourceWarning is needed here. The purpose of a ResourceWarning is to warn when a leak may be kept alive longer than expected if it's not closed explicitly. |
|
|
| msg298661 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2017-07-19 10:16 |
| Anyone who thinks those objects stay alive too long can submit a PR for SimpleQueue and Pool to close them eagerly. |
|
|
| msg298667 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-07-19 10:31 |
| > Anyone who thinks those objects stay alive too long can submit a PR for SimpleQueue and Pool to close them eagerly. I started with SimpleQueue for the master branch: https://github.com/python/cpython/pull/2760 |
|
|
| msg367432 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2020-04-27 15:47 |
| bpo-23267 is marked as a duplicate of this issue. |
|
|
| msg367435 - (view) |
Author: miss-islington (miss-islington) |
Date: 2020-04-27 16:11 |
| New changeset 9adccc1384568f4d46e37f698cb3e3a4f6ca0252 by Victor Stinner in branch 'master': bpo-30966: Add multiprocessing.SimpleQueue.close() (GH-19735) https://github.com/python/cpython/commit/9adccc1384568f4d46e37f698cb3e3a4f6ca0252 |
|
|
| msg367445 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2020-04-27 18:53 |
| New changeset 1a275013d1ecc2e3778d64fda86174b2f13d6969 by Victor Stinner in branch 'master': bpo-30966: concurrent.futures.Process.shutdown() closes queue (GH-19738) https://github.com/python/cpython/commit/1a275013d1ecc2e3778d64fda86174b2f13d6969 |
|
|
| msg396246 - (view) |
Author: Irit Katriel (iritkatriel) * |
Date: 2021-06-21 13:49 |
| This seems complete, can it be closed? |
|
|