Issue 30796: Failures/crashes in test_signal on some buildbots (original) (raw)

Created on 2017-06-28 22:23 by pitrou, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 2488 merged pitrou,2017-06-29 12:07
Messages (16)
msg297237 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-28 22:23
The test added for issue #30703 crashes on some Unix buildbots (FreeBSD 9.x, OS X Tiger, AIX). Example here: http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.x%203.x/builds/275/steps/test/logs/stdio As noted in the PR for that issue, some systems may have low resolution for setitimer(). For example, the FreeBSD man page says: > Time values smaller than the resolution of the system clock are rounded up to this resolution (typically 10 milliseconds).
msg297239 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-28 22:26
koobs, perhaps you would like to give me temporary shell access to that machine?
msg297260 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-29 07:26
The process is even killed. http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.x%203.x/builds/275/steps/test/logs/stdio Re-running test 'test_signal' in verbose mode test_enums (test.test_signal.GenericTests) ... ok test_itimer_exc (test.test_signal.ItimerTest) ... ok test_itimer_prof (test.test_signal.ItimerTest) ... ok test_itimer_real (test.test_signal.ItimerTest) ... ok test_itimer_virtual (test.test_signal.ItimerTest) ... ok test_pthread_kill (test.test_signal.PendingSignalsTests) ... ok test_pthread_kill_main_thread (test.test_signal.PendingSignalsTests) ... ok test_pthread_sigmask (test.test_signal.PendingSignalsTests) ... ok test_pthread_sigmask_arguments (test.test_signal.PendingSignalsTests) ... ok test_sigpending (test.test_signal.PendingSignalsTests) ... ok test_sigpending_empty (test.test_signal.PendingSignalsTests) ... ok test_sigtimedwait (test.test_signal.PendingSignalsTests) ... ok test_sigtimedwait_negative_timeout (test.test_signal.PendingSignalsTests) ... ok test_sigtimedwait_poll (test.test_signal.PendingSignalsTests) ... ok test_sigtimedwait_timeout (test.test_signal.PendingSignalsTests) ... ok test_sigwait (test.test_signal.PendingSignalsTests) ... ok test_sigwait_thread (test.test_signal.PendingSignalsTests) ... ok test_sigwaitinfo (test.test_signal.PendingSignalsTests) ... ok test_getsignal (test.test_signal.PosixTests) ... ok test_interprocess_signal (test.test_signal.PosixTests) ... ok test_out_of_range_signal_number_raises_error (test.test_signal.PosixTests) ... ok test_setting_signal_handler_to_none_raises_error (test.test_signal.PosixTests) ... ok test_siginterrupt_off (test.test_signal.SiginterruptTest) ... ok test_siginterrupt_on (test.test_signal.SiginterruptTest) ... ok test_without_siginterrupt (test.test_signal.SiginterruptTest) ... ok test_stress_delivery_dependent (test.test_signal.StressTest) ... FAIL test_stress_delivery_simultaneous (test.test_signal.StressTest) ... FAIL test_invalid_fd (test.test_signal.WakeupFDTests) ... ok test_invalid_socket (test.test_signal.WakeupFDTests) ... ok test_set_wakeup_fd_blocking (test.test_signal.WakeupFDTests) ... ok test_set_wakeup_fd_result (test.test_signal.WakeupFDTests) ... ok test_set_wakeup_fd_socket_result (test.test_signal.WakeupFDTests) ... ok test_pending (test.test_signal.WakeupSignalTests) ... *** [buildbottest] Signal 14
msg297267 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2017-06-29 11:13
@Antoine Happy to any time. PM me public key on IRC (freenode:koobs) and I'll reply with user@host:port
msg297271 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-29 14:40
New changeset f7d090c165f6cd3d008fe60c78e5324caef53f80 by Antoine Pitrou in branch 'master': bpo-30796: Fix failures in signal delivery stress test (#2488) https://github.com/python/cpython/commit/f7d090c165f6cd3d008fe60c78e5324caef53f80
msg297275 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-29 15:14
The stable buildbots seem fine now (problem fixed on FreeBSD 9, thank you Koobs!).
msg297278 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-29 15:31
http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/979/steps/test/logs/stdio test_stress_delivery_dependent (test.test_signal.StressTest) ... Timeout (0:15:00)! Thread 0x405acdc0 (most recent call first): File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_signal.py", line 973 in measure_itimer_resolution File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_signal.py", line 985 in decide_itimer_count File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_signal.py", line 1001 in test_stress_delivery_dependent File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 615 in run File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 663 in __call__ File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__ File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__ File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__ File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/runner.py", line 176 in run File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 1896 in _run_suite File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 1940 in run_unittest File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 168 in test_runner File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 169 in runtest_inner File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 137 in runtest File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 290 in rerun_failed_tests File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 540 in _main File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 510 in main File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 585 in main File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/__main__.py", line 2 in File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/runpy.py", line 85 in _run_code File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/runpy.py", line 193 in _run_module_as_main make: *** [buildbottest] Error 1 program finished with exit code 2 elapsedTime=2727.765081
msg297279 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-29 16:09
Hmm, that function only sends 20 signals in a row. How can it timeout after 15 seconds?
msg297280 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-29 16:14
Hmm, it's not 15 seconds, it's 15 minutes...
msg297281 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-29 16:18
Ok, there's definitely something fishy here. Look at the elapsed times: 0:16:33 load avg: 0.93 [281/406] test_ossaudiodev skipped -- running: test_signal (874 sec) test_ossaudiodev skipped -- [Errno 2] No such file or directory: '/dev/dsp' 0:16:59 load avg: 0.95 [282/406/1] test_signal crashed (Exit code 1) Timeout (0:15:00)! faulthandler claims that 15 minutes have elapsed, but only 59-33 = 26 *seconds* have!
msg297282 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-29 16:29
x86 Tiger is fine too.
msg297283 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-29 16:31
David, would it be possible to have shell access to that machine?
msg297294 - (view) Author: David Bolen (db3l) * Date: 2017-06-29 18:54
Antoine, yes. Send me your public key (db3l.net at gmail) and I'll set it up.
msg297302 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-29 19:56
Ok, the timeout on the "Ubuntu Shared" buildbot is due to https://bugs.python.org/issue30807
msg297359 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-30 08:32
It seems fixing https://bugs.python.org/issue30807 has fixed the remaining buildbot issues. I will close once everything is settled.
msg297485 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-07-01 17:12
New changeset 3024c0529077f5cff0b32dc84b5923c8fba99a87 by Antoine Pitrou in branch '3.6': [3.6] bpo-30703: Improve signal delivery (GH-2415) (#2527) https://github.com/python/cpython/commit/3024c0529077f5cff0b32dc84b5923c8fba99a87
History
Date User Action Args
2022-04-11 14:58:48 admin set github: 74979
2017-07-01 17:12:08 pitrou set messages: +
2017-06-30 15:41:29 pitrou set status: open -> closedresolution: fixedstage: commit review -> resolved
2017-06-30 08:32:07 pitrou set messages: +
2017-06-29 19:56:25 pitrou set messages: +
2017-06-29 19:49:43 pitrou set stage: needs patch -> commit reviewversions: - Python 2.7, Python 3.5, Python 3.6
2017-06-29 19:49:24 pitrou set dependencies: + setitimer() can disable timer by mistakeversions: + Python 2.7, Python 3.5, Python 3.6
2017-06-29 18:54:25 db3l set messages: +
2017-06-29 16:31:36 pitrou set messages: +
2017-06-29 16:29:08 pitrou set messages: +
2017-06-29 16🔞34 pitrou set messages: +
2017-06-29 16:14:24 pitrou set messages: +
2017-06-29 16:09:03 pitrou set nosy: + db3lmessages: +
2017-06-29 15:31:54 vstinner set messages: +
2017-06-29 15:14:26 pitrou set messages: +
2017-06-29 14:40:16 pitrou set messages: +
2017-06-29 12:07:43 pitrou set pull_requests: + <pull%5Frequest2547>
2017-06-29 11:13:24 koobs set messages: +
2017-06-29 07:26:44 vstinner set messages: +
2017-06-28 22:26:16 pitrou set messages: +
2017-06-28 22:25:02 pitrou set nosy: + koobs
2017-06-28 22:23:54 pitrou create