| msg299959 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2017-08-09 00:40 |
| It seems like test_socketserver leaks child processes on the "x86 Gentoo Refleaks 3.6" buildbot, but I'm unable to reproduce the bug. http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%203.6/builds/49/steps/test/logs/stdio 0:48:11 load avg: 3.91 [154/405/2] test_socketserver failed -- running: test_decimal (630 sec) Warning -- reap_children() reaped child process 1044 beginning 6 repetitions 123456 Warning -- reap_children() reaped child process 1104 Warning -- reap_children() reaped child process 1115 .Warning -- reap_children() reaped child process 1170 Warning -- reap_children() reaped child process 1175 Warning -- reap_children() reaped child process 1184 .Warning -- reap_children() reaped child process 1249 .Warning -- reap_children() reaped child process 1311 Warning -- reap_children() reaped child process 1316 ... test_socketserver leaked [1, 1, 1] memory blocks, sum=3 (...) test_ForkingUnixStreamServer (test.test_socketserver.SocketServerTest) ... creating server ADDR = /tmp/unix_socket.hqh5x95a CLASS = <class 'test.test_socketserver.ForkingUnixStreamServer'> server running test client 0 test client 1 test client 2 waiting for server done Warning -- reap_children() reaped child process 17938 ok test_ForkingUnixDatagramServer (test.test_socketserver.SocketServerTest) ... creating server ADDR = /tmp/unix_socket.gry6ulhp CLASS = <class 'test.test_socketserver.ForkingUnixDatagramServer'> server running test client 0 test client 1 test client 2 waiting for server done Warning -- reap_children() reaped child process 18212 ok test_ForkingUDPServer (test.test_socketserver.SocketServerTest) ... creating server ADDR = ('127.0.0.1', 43415) CLASS = <class 'socketserver.ForkingUDPServer'> server running test client 0 test client 1 test client 2 waiting for server done Warning -- reap_children() reaped child process 18281 ok test_ForkingUnixDatagramServer (test.test_socketserver.SocketServerTest) |
|
|
| msg299960 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-08-09 00:41 |
| Ah, the master branch is also affected. Example: http://buildbot.python.org/all/builders/x86%20Gentoo%20Refleaks%203.x/builds/47/steps/test/logs/stdio 1:35:05 load avg: 4.64 [212/406] test_socketserver passed -- running: test_subprocess (370 sec) beginning 6 repetitions 123456 Warning -- reap_children() reaped child process 4641 .Warning -- reap_children() reaped child process 4708 ..Warning -- reap_children() reaped child process 4832 .Warning -- reap_children() reaped child process 4916 Warning -- reap_children() reaped child process 4921 .. |
|
|
| msg300074 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-08-10 11:30 |
| The problem is that socketserver.ForkinMixin doesn't wait until all children completes. It's only calls os.waitpid() in non-blocking module (using os.WNOHANG) after each loop iteration. If a child process completes after the last call to ForkingMixIn.collect_children(), the server leaks zombie processes. The server must wait until all children completes. Attached PR implements that. The bug was be reproduced with the attached forkingmixin_sleep.patch. haypo@selma$ ./python -m test -v -u all test_socketserver --fail-env-changed -m '*Fork*' (...) Warning -- reap_children() reaped child process 17093 Warning -- reap_children() reaped child process 17094 (...) |
|
|
| msg300084 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-08-10 13:28 |
| New changeset aa8ec34ad52bb3b274ce91169e1bc4a598655049 by Victor Stinner in branch 'master': bpo-31151: Add socketserver.ForkingMixIn.server_close() (#3057) https://github.com/python/cpython/commit/aa8ec34ad52bb3b274ce91169e1bc4a598655049 |
|
|
| msg300106 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-08-10 15:38 |
| bpo-31010 has been marked as a duplicate of this issue. |
|
|
| msg302013 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-09-13 00:39 |
| I created a thread on the python-dev mailing list to discuss this issue: https://mail.python.org/pipermail/python-dev/2017-August/148826.html |
|
|
| msg302015 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-09-13 00:43 |
| I tag this issue as release blocker as a remainder that we have to agree how to handle threads/processes before Python 3.7 feature freeze. See also bpo-31233: socketserver.ThreadingMixIn leaks running threads after server_close(). |
|
|
| msg303041 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2017-09-26 13:50 |
| I proposed the bpo-31593 to fix the issue differently in Python 3.6 (and maybe also Python 2.7). |
|
|
| msg311015 - (view) |
Author: Ned Deily (ned.deily) * |
Date: 2018-01-28 22:19 |
| What's the status of this? Feature freeze is here! |
|
|
| msg311022 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2018-01-28 22:36 |
| > What's the status of this? Feature freeze is here! This issue is very to https://bugs.python.org/issue31233#msg311021 as it's status: * The commit aa8ec34ad52bb3b274ce91169e1bc4a598655049 is backward incompatible and it's not documented * There is no option to opt-in for the previous behaviour (exit without waiting for child processes) For beta1, IMHO it's ok to keep the current status. But I would like to see this issue fixed before 3.7 final. Either revert my change, or do something else. |
|
|
| msg311093 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2018-01-29 11:10 |
| New changeset db8189bb8db609ca3993dec31fb95fdbe134469a by Victor Stinner in branch 'master': bpo-31233, bpo-31151: Document socketserver changes (#5417) https://github.com/python/cpython/commit/db8189bb8db609ca3993dec31fb95fdbe134469a |
|
|
| msg316834 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2018-05-16 16:28 |
| I created bpo-33540: "socketserver: Add an opt-in option to get Python 3.6 behaviour on server_close()". I set the release blocker priority on this new issue. |
|
|
| msg318116 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2018-05-29 22:01 |
| The bug has been fixed in Python 3.7. |
|
|