Issue 33680: regrtest: re-run failed tests in a subprocess (original) (raw)
The -w option of regrtest (python -m test) re-run failed tests in verbose in the current process. Problem: if the test does crash, regrtest doesn't re-run following tests, it doesn't write the final tests result (like failing tests, total duration, etc.).
To make regrtest tolerant to such crashes, re-run tests should be run in verbose mode. The only constraint is to disable buffering on stdout and stderr to make sure that stdout and stderr order is kept in pipes. Or maybe stderr should be redirected to stdout to avoid ordering issue?
A random example of such crash, process killed by faulthandler timeout: http://buildbot.python.org/all/#/builders/118/builds/65
(...) 394 tests OK.
10 slowest tests:
- test_subprocess: 7 min 13 sec
- test_concurrent_futures: 2 min 47 sec
- test_multiprocessing_spawn: 2 min 29 sec
- test_urllib2net: 1 min 45 sec
- test_multiprocessing_forkserver: 1 min 34 sec
- test_nntplib: 1 min 29 sec
- test_smtpnet: 1 min 23 sec
- test_multiprocessing_fork: 1 min 22 sec
- test_tools: 1 min 19 sec
- test_tokenize: 1 min 13 sec
1 test failed: test_asyncio
20 tests skipped: test_dbm_gnu test_devpoll test_epoll test_gdb test_idle test_ioctl test_msilib test_ossaudiodev test_spwd test_startfile test_tcl test_tix test_tk test_ttk_guionly test_ttk_textonly test_turtle test_winconsoleio test_winreg test_winsound test_zipfile64 Re-running failed tests in verbose mode Re-running test 'test_asyncio' in verbose mode (...) test_sock_client_ops (test.test_asyncio.test_events.PollEventLoopTests) ... ok
Timeout (0:15:00)! Thread 0x000000080139a000 (most recent call first): File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/selectors.py", line 415 in select File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/asyncio/base_events.py", line 1713 in _run_once File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/asyncio/base_events.py", line 523 in run_forever File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/asyncio/base_events.py", line 555 in run_until_complete File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/test/test_asyncio/test_events.py", line 2135 in run_loop File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/test/test_asyncio/test_events.py", line 2249 in test_sock_sendfile_mix_with_regular_send (...)
test_sock_sendfile_mix_with_regular_send (test.test_asyncio.test_events.PollEventLoopTests) ... *** Error code 1 Stop. make: stopped in /usr/home/buildbot/python/3.7.koobs-freebsd-current/build program finished with exit code 1 elapsedTime=2612.384935