| msg104212 - (view) |
Author: STINNER Victor (vstinner) *  |
Date: 2010-04-26 10:43 |
| If a test fails, regrtest writes the backtrace to sys.stdout. If the backtrace contains a non-ASCII characters, it's encoded using sys.stdout encoding. In some conditions, sys.stdout is unable to encode some or all non-ASCII characters. Eg. if there is no locale set (empty environment or at least empty LANG variable value), sys.stdout.encoding="ascii". If regrtest fails to display a test output (error backtrace), regrtest exits directly (don't execute next tests). I propose to use backslashreplace error handler in sys.stdout, as done for sys.stderr to avoid this annoying issue. Attached patch (for py3k) replace sys.stdout by a new file using backslashreplace, just before executing the tests. I don't know if the issue concerns also Python2. |
|
|
| msg104223 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2010-04-26 11:30 |
| A better resolution IMO would be to output tracebacks on stderr instead. |
|
|
| msg104229 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2010-04-26 12:47 |
| > A better resolution IMO would be to output tracebacks on stderr instead. Yeah, that sounds easier and safer. Attached patch writes the tracebacks to stderr. I don't remember how to reproduce this issue :-/ I guess that stdout/stderr encoding should be ASCII (eg. LANG="") and the traceback (eg. the error message) should contain a non-ASCII character. |
|
|
| msg104231 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2010-04-26 12:49 |
| Write to stderr instead of stdout might change buildbot output order. |
|
|
| msg104670 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2010-04-30 22:45 |
| regrtest_traceback_stderr.patch is not enough: support._run_suite() writes output to sys.stdout instead of sys.stderr. New version of the patch fixes that. |
|
|
| msg104773 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2010-05-02 09:38 |
| Ok, let's try sys.stderr solution: commited in r80694 (py3k). If it breaks buildbot outputs, I will revert it and try the second solution. |
|
|
| msg104783 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2010-05-02 11:08 |
| Since it may reorder output, I think it's better revert the patch and try the other solution. However, I don't think you need to replace sys.stdout at all: just output the traceback more carefully. |
|
|
| msg104794 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2010-05-02 17:25 |
| As "expected", the patch doesn't work: it randomize the output order :-( I checked "sparc Ubuntu 3.x": the output order is correct. "test_xxx" lines are written to stdout, "FAIL: ..." + traceback are written to stderr, and the lines are written in the right order. But it failed on "x86 Tiger 3.x" : http://www.python.org/dev/buildbot/3.x/builders/x86 Tiger 3.x/builds/130/steps/test/logs/stdio http://www.python.org/dev/buildbot/3.x/builders/x86 Tiger 3.x/builds/131/steps/test/logs/stdio When a test is reexecuted in verbose mode, the output is written in red (why not, but it was not the case before my commit), and the stdout and stderr are in a "mixed". -- antoine> just output the traceback more carefully. Encode the traceback by hand would be possible, but it's more complex. A possible solution would be to write the output to a StringIO, encode unicode to bytes using the right encoding and backslashreplace error handler, and write the result to stdout. But I don't like buffering the output because the buildbot may hung for different reasons (search in the bug tracker for "test_multiprocessing" or "test_subprocess"...) and the buildbot master may consider the buildbot as dead (no new output since xxx seconds, whereas it's writing to a buffer). -- I prefer my first simple idea: use backslashreplace error handler for stdout. Let's try: r80703. This commit reverts my previous commit, apply regrtest_stdout_backslashreplace.patch + a fix for multiprocessing mode (regrtest.py -j ...). |
|
|
| msg104824 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2010-05-03 08:37 |
| > Let's try: r80703 This one looks ok: the output order is kept and I didn't noticed anything special in the buildbot output. Backported to 3.1 as r80711. Close the issue. |
|
|
| msg105087 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2010-05-05 21:31 |
| Reopen: r80703 (and r80711) introduces a new bug: on Windows, there is an empty string between each line. It looks like a newline error. replace_stdout() should set the newline argument to open(). But how can I get the newline attribute from sys.stdout? sys.stdout._writenl? regrtest_stdout_newline.patch should fix the newline problem. |
|
|
| msg105180 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2010-05-07 00:59 |
| I disabled my patch (replace_stdout function) on Windows until it gets fixed: r80905 (py3k) and r80906 (3.1) |
|
|
| msg116003 - (view) |
Author: Hirokazu Yamamoto (ocean-city) * |
Date: 2010-09-10 11:08 |
| Hello. How about this patch? I just mimicked create_stdio() in Python/pythonrun.c. (Mostly) Newlines looks correct as well as python2.x. I tested this on windows. |
|
|
| msg116004 - (view) |
Author: Hirokazu Yamamoto (ocean-city) * |
Date: 2010-09-10 11:10 |
| And here is more advanced (?) patch. More closer to Python/pythonrun.c 's implementaion. I tried regrtest_stdout_newline.patch, but it doesn't work. # AttributeError: '_io.TextIOWrapper' object has no attribute '_writenl' |
|
|
| msg118248 - (view) |
Author: Hirokazu Yamamoto (ocean-city) * |
Date: 2010-10-09 07:50 |
| I noticed fd must be dup-ed before passing to io.open. And I noticed direct test run like "py3k -m test.test_time" still produces unicode error. (Maybe this is reasonable limitation) I can confirm we can suppress the unicode error in such case with newly attached patch. "py3k_also_no_unicode_error_on_direct_test_run.patch" |
|
|
| msg118253 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2010-10-09 08:16 |
| py3k_also_no_unicode_error_on_direct_test_run.patch comes a little bit too late: $ LANG= ./python Lib/test/regrtest.py -v test_time == CPython 3.2a2+ (py3k, Oct 8 2010, 01:40:20) [GCC 4.4.5 20100909 (prerelease)] == Linux-2.6.32-trunk-686-i686-with-debian-squeeze-sid little-endian == Traceback (most recent call last): File "Lib/test/regrtest.py", line 1478, in main() File "Lib/test/regrtest.py", line 454, in main print("== ", os.getcwd()) UnicodeEncodeError: 'ascii' codec can't encode characters in position 20-21: ordinal not in range(128) My working directory is /home/haypo/prog/SVN/py3ké. |
|
|
| msg118256 - (view) |
Author: Hirokazu Yamamoto (ocean-city) * |
Date: 2010-10-09 08:58 |
| Oops, sorry. I'll withdraw my last patch. |
|
|
| msg118395 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2010-10-11 21:50 |
| > Oops, sorry. I'll withdraw my last patch. Why? Your patch is useful to run a single test outside regrtest. But you should not remove the hack on regrtest.py, only keep your patch on unittest/runner.py. There are not exclusive. |
|
|
| msg118396 - (view) |
Author: Hirokazu Yamamoto (ocean-city) * |
Date: 2010-10-12 00:24 |
| Thank you. I'll reattach the patch only for Lib/unittest/runner.py |
|
|
| msg123605 - (view) |
Author: Hirokazu Yamamoto (ocean-city) * |
Date: 2010-12-08 14:15 |
| Well, can this go into Python3.2? |
|
|
| msg136813 - (view) |
Author: Roundup Robot (python-dev)  |
Date: 2011-05-25 00:02 |
| New changeset 8be9eaf5829f by Victor Stinner in branch 'default': Issue #8533: regrtest replaces also sys.stdout on Windows http://hg.python.org/cpython/rev/8be9eaf5829f |
|
|
| msg137330 - (view) |
Author: STINNER Victor (vstinner) * |
Date: 2011-05-30 21:09 |
| The original issue (use backslashreplace for stdout in regrtest) is now fixed, and so I closed it. @ocean-city: Can you please open a new issue for unittest? (for py3k_also_no_unicode_error_on_direct_test_run.patch) |
|
|