Issue 31160: Enhance support.reap_children() - Python tracker (original) (raw)

Created on 2017-08-09 12:50 by vstinner, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 3036 merged vstinner,2017-08-09 12:50
PR 3040 closed vstinner,2017-08-09 13:46
PR 3043 merged vstinner,2017-08-09 15:23
PR 3044 merged vstinner,2017-08-09 15:27
PR 3045 merged vstinner,2017-08-09 15:41
PR 3046 closed vstinner,2017-08-09 16:03
PR 3056 merged vstinner,2017-08-10 10:47
PR 3060 merged vstinner,2017-08-10 13:41
PR 3063 merged vstinner,2017-08-10 14:09
PR 19263 merged vstinner,2020-03-31 20:53
PR 19308 closed Michael.Felt,2020-04-02 17:43
PR 19312 closed vstinner,2020-04-02 20:08
Messages (23)
msg299996 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 12:50
Attached PR enhances the support.reap_children() function: * reap_children() now sets environment_altered to True to detect bugs using python3 -m test --fail-env-changed * Replace bare "except:" with "except OSError:" in reap_children() * Write an unit test for reap_children() using a timeout of 60 seconds
msg299997 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 12:51
The GCC job of Travis CI failed with ENV_CHANGED: Test wait() behavior when waitpid returns WIFSTOPPED; . ... Warning -- reap_children() reaped child process 19839 ok I tested and... WOW! When run in a loop, this test leaks 100 MB per second. It creates a lot of processes.
msg299998 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 12:54
> The GCC job of Travis CI failed with ENV_CHANGED: Ooops, in fact it was a macOS job: https://travis-ci.org/python/cpython/jobs/262606830 The GCC job failed with much more errors: https://travis-ci.org/python/cpython/jobs/262606831 --- 0:00:02 load avg: 121.34 [ 7/403] test_unittest Warning -- reap_children() reaped child process 11088 Warning -- reap_children() reaped child process 11089 Warning -- reap_children() reaped child process 11090 Warning -- reap_children() reaped child process 11091 0:14:22 load avg: 136.67 [282/403/1] test_select Warning -- reap_children() reaped child process 14686 0:16:00 load avg: 110.41 [297/403/1] test_socketserver Warning -- reap_children() reaped child process 15483 Warning -- reap_children() reaped child process 15492 Warning -- reap_children() reaped child process 15499 Warning -- reap_children() reaped child process 15508 0🔞36 load avg: 105.94 [333/403/1] test_thread Warning -- reap_children() reaped child process 20670 --- For test_socketserver, see bpo-31151.
msg300000 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 13:20
A reap_children() warning was fixed in test_thread: bpo-31150. It seems like the commit 88eee44a91c991dbbdf284fa220e2928b5de105c was not enough to fix all warnings.
msg300020 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 15:44
New changeset 4baca1b0f7325032598cd38e7ceffc79b616d255 by Victor Stinner in branch 'master': bpo-31160: Fix test_builtin for zombie process (#3043) https://github.com/python/cpython/commit/4baca1b0f7325032598cd38e7ceffc79b616d255
msg300021 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 15:44
New changeset e3510d74aacc477c30f42f2b941d69689bbc478e by Victor Stinner in branch 'master': bpo-31160: regrtest now reaps child processes (#3044) https://github.com/python/cpython/commit/e3510d74aacc477c30f42f2b941d69689bbc478e
msg300023 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 15:59
New changeset da5e9301877346942fa4279612750d6a09e05153 by Victor Stinner in branch 'master': bpo-31160: Fix test_random for zombie process (#3045) https://github.com/python/cpython/commit/da5e9301877346942fa4279612750d6a09e05153
msg300024 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 16:19
> New changeset 4baca1b0f7325032598cd38e7ceffc79b616d255 by Victor Stinner in branch 'master': > bpo-31160: Fix test_builtin for zombie process (#3043) This change introduced a regression: http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/1159/steps/test/logs/stdio ====================================================================== FAIL: test_input_no_stdout_fileno (test.test_builtin.PtyTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_builtin.py", line 1624, in test_input_no_stdout_fileno lines = self.run_child(child, b"quux\r") File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_builtin.py", line 1573, in run_child self.assertEqual(status, 0) AssertionError: 1 != 0
msg300072 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-10 10:41
bpo-31173 fixed a leaked child process in test_subprocess.
msg300073 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-10 11:05
New changeset 6c8c2943d996b59a48d331f61f22cbe72933910e by Victor Stinner in branch 'master': bpo-31160: test_tempfile: Fix reap_children() warning (#3056) https://github.com/python/cpython/commit/6c8c2943d996b59a48d331f61f22cbe72933910e
msg300086 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-10 13:29
bpo-31151 fixed test_socketserver.
msg300088 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-10 14:02
New changeset 719a15b32587de6c4add4385ee3f84a20711340f by Victor Stinner in branch '3.6': [3.6] bpo-31160: Backport reap_children() fixes from master to 3.6 (#3060) https://github.com/python/cpython/commit/719a15b32587de6c4add4385ee3f84a20711340f
msg300093 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-10 14:45
New changeset 1247e2cda514d7a73187e0b53ec8c35d87a34a84 by Victor Stinner in branch '2.7': [2.7] bpo-31160: Backport reap_children fixes from master to 2.7 (#3063) https://github.com/python/cpython/commit/1247e2cda514d7a73187e0b53ec8c35d87a34a84
msg300147 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-11 00:12
I pushed the most important change: reap_children() now makes tests fail with ENV_CHANGED on warning, so I close the issue.
msg365430 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-31 22:27
New changeset 16d75675d2ad2454f6dfbf333c94e6237df36018 by Victor Stinner in branch 'master': bpo-31160: Fix race condition in test_os.PtyTests (GH-19263) https://github.com/python/cpython/commit/16d75675d2ad2454f6dfbf333c94e6237df36018
msg365474 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-04-01 14:31
With PR19263 The AIX bots are now red. ====================================================================== ERROR: test_input_no_stdout_fileno (test.test_builtin.PtyTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/test_builtin.py", line 1952, in test_input_no_stdout_fileno lines = self.run_child(child, b"quux\r") File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/test_builtin.py", line 1898, in run_child support.wait_process(pid, exitcode=0) File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/test/support/__init__.py", line 3432, in wait_process os.kill(pid, signal.SIGKILL) NameError: name 'signal' is not defined ---------------------------------------------------------------------- Ran 101 tests in 30.348s FAILED (errors=1, skipped=7) 1 test failed again: test_builtin +++++++++++++++ The Buildbot has detected a failed build on builder PPC64 AIX 3.x while building python/cpython. Full details are available at: https://buildbot.python.org/all/#builders/227/builds/565 Buildbot URL: https://buildbot.python.org/all/ Worker for this Build: edelsohn-aix-ppc64 Worker for this Build: aixtools-aix-power6
msg365475 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-01 14:41
> With PR19263 The AIX bots are now red. I know, I saw and I already pushed fixes. > NameError: name 'signal' is not defined Fixed by commit afeaea2d6e346f627b24cc9e84e2986a7266a70e. > 1 test failed again: test_builtin Fixed by commit 16d75675d2ad2454f6dfbf333c94e6237df36018.
msg365476 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-04-01 14:49
Ah - great. Sorry for the noise then.
msg365477 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-01 14:49
> Ah - great. Sorry for the noise then. It's not noise, it is useful :-)
msg365478 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-04-01 15:00
I think something is not yet what it needs to be: the bots both finish test with: test_zip_pickle (test.test_builtin.BuiltinTest) ... ok Timeout (0:15:00)! Thread 0x00000001 (most recent call first): File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/support/__init__.py", line 3435 in wait_process File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/test_builtin.py", line 1898 in run_child File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/test_builtin.py", line 1952 in test_input_no_stdout_fileno File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/case.py", line 616 in _callTestMethod File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/case.py", line 659 in run File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/case.py", line 719 in __call__ File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/suite.py", line 122 in run File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/suite.py", line 84 in __call__ File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/suite.py", line 122 in run File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/suite.py", line 84 in __call__ File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/suite.py", line 122 in run File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/suite.py", line 84 in __call__ File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/unittest/runner.py", line 176 in run File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/support/__init__.py", line 2079 in _run_suite File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/support/__init__.py", line 2201 in run_unittest File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/runtest.py", line 209 in _test_module File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/runtest.py", line 234 in _runtest_inner2 File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/runtest.py", line 270 in _runtest_inner File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/runtest.py", line 153 in _runtest File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/runtest.py", line 193 in runtest File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/main.py", line 318 in rerun_failed_tests File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/main.py", line 691 in _main File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/main.py", line 634 in main File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/libregrtest/main.py", line 712 in main File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/test/__main__.py", line 2 in File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/runpy.py", line 87 in _run_code File "/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib/runpy.py", line 197 in _run_module_as_main make: 1254-004 The error code from the last command is 1. Stop. program finished with exit code 2 elapsedTime=3501.292487 test_input_no_stdout_fileno (test.test_builtin.PtyTests) ... And the bot status is still FAIL (aka red): failed test (failure) uploading test-results.xml (failure)
msg365479 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-01 15:03
> I think something is not yet what it needs to be: (...) https://buildbot.python.org/all/#/builders/227/builds/571 build failed but it has my commit 16d75675d2ad2454f6dfbf333c94e6237df36018. Ok, something failed. Please open a new issue. This one is closed.
msg365512 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-01 21:42
> Please open a new issue. This one is closed. Pablo Galindo opened bpo-40140, let's use this one.
msg365517 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-01 22:03
> Pablo Galindo opened bpo-40140, let's use this one. Note: Oops, Batuhan created it, Pablo only commented.
History
Date User Action Args
2022-04-11 14:58:49 admin set github: 75343
2020-04-02 20:08:02 vstinner set pull_requests: + <pull%5Frequest18675>
2020-04-02 17:43:42 Michael.Felt set pull_requests: + <pull%5Frequest18670>
2020-04-01 22:03:51 vstinner set messages: +
2020-04-01 21:42:59 vstinner set messages: +
2020-04-01 15:03:24 vstinner set messages: +
2020-04-01 15:00:19 Michael.Felt set messages: +
2020-04-01 14:49:59 vstinner set messages: +
2020-04-01 14:49:15 Michael.Felt set messages: +
2020-04-01 14:41:24 vstinner set messages: +
2020-04-01 14:31:10 Michael.Felt set nosy: + Michael.Feltmessages: +
2020-03-31 22:27:23 vstinner set messages: +
2020-03-31 20:53:07 vstinner set pull_requests: + <pull%5Frequest18619>
2017-08-11 00:12:50 vstinner set status: open -> closedresolution: fixedmessages: + stage: resolved
2017-08-10 14:45:42 vstinner set messages: +
2017-08-10 14:09:58 vstinner set pull_requests: + <pull%5Frequest3097>
2017-08-10 14:02:02 vstinner set messages: +
2017-08-10 13:41:11 vstinner set pull_requests: + <pull%5Frequest3095>
2017-08-10 13:29:53 vstinner set messages: +
2017-08-10 11:05:09 vstinner set messages: +
2017-08-10 10:47:03 vstinner set pull_requests: + <pull%5Frequest3091>
2017-08-10 10:41:51 vstinner set messages: +
2017-08-09 16:19:41 vstinner set messages: +
2017-08-09 16:03:40 vstinner set pull_requests: + <pull%5Frequest3080>
2017-08-09 15:59:10 vstinner set messages: +
2017-08-09 15:44:35 vstinner set messages: +
2017-08-09 15:44:21 vstinner set messages: +
2017-08-09 15:41:57 vstinner set pull_requests: + <pull%5Frequest3079>
2017-08-09 15:27:09 vstinner set pull_requests: + <pull%5Frequest3078>
2017-08-09 15:23:08 vstinner set pull_requests: + <pull%5Frequest3077>
2017-08-09 13:46:22 vstinner set pull_requests: + <pull%5Frequest3074>
2017-08-09 13:20:00 vstinner set messages: +
2017-08-09 12:54:50 vstinner set messages: +
2017-08-09 12:51:10 vstinner set messages: +
2017-08-09 12:50:03 vstinner create