Issue 8423: tiger buildbot: test_pep277 failures (original) (raw)

Issue8423

Created on 2010-04-16 16:35 by vstinner, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (10)
msg103340 - (view)	Author: STINNER Victor (vstinner) *	Date: 2010-04-16 16:35
http://www.python.org/dev/buildbot/3.x/builders/x86 Tiger 3.x/builds/6/steps/test/logs/stdio test test_pep277 failed -- multiple errors occurred; run in verbose mode for details Re-running test test_pep277 in verbose mode test_directory (test.test_pep277.UnicodeFileTests) ... ok test_failures (test.test_pep277.UnicodeFileTests) ... ok test_listdir (test.test_pep277.UnicodeFileTests) ... FAIL test_normalize (test.test_pep277.UnicodeFileTests) ... ERROR test_open (test.test_pep277.UnicodeFileTests) ... ok test_rename (test.test_pep277.UnicodeFileTests) ... ok test_directory (test.test_pep277.UnicodeNFCFileTests) ... ok test_failures (test.test_pep277.UnicodeNFCFileTests) ... ok test_listdir (test.test_pep277.UnicodeNFCFileTests) ... FAIL test_normalize (test.test_pep277.UnicodeNFCFileTests) ... ERROR test_open (test.test_pep277.UnicodeNFCFileTests) ... ok test_rename (test.test_pep277.UnicodeNFCFileTests) ... ok test_directory (test.test_pep277.UnicodeNFDFileTests) ... ok test_failures (test.test_pep277.UnicodeNFDFileTests) ... ok test_listdir (test.test_pep277.UnicodeNFDFileTests) ... ok test_normalize (test.test_pep277.UnicodeNFDFileTests) ... ERROR test_open (test.test_pep277.UnicodeNFDFileTests) ... ok test_rename (test.test_pep277.UnicodeNFDFileTests) ... ok test_directory (test.test_pep277.UnicodeNFKCFileTests) ... ok test_failures (test.test_pep277.UnicodeNFKCFileTests) ... ok test_listdir (test.test_pep277.UnicodeNFKCFileTests) ... FAIL test_normalize (test.test_pep277.UnicodeNFKCFileTests) ... ERROR test_open (test.test_pep277.UnicodeNFKCFileTests) ... ok test_rename (test.test_pep277.UnicodeNFKCFileTests) ... ok test_directory (test.test_pep277.UnicodeNFKDFileTests) ... ok test_failures (test.test_pep277.UnicodeNFKDFileTests) ... ok test_listdir (test.test_pep277.UnicodeNFKDFileTests) ... ok test_normalize (test.test_pep277.UnicodeNFKDFileTests) ... ERROR test_open (test.test_pep277.UnicodeNFKDFileTests) ... ok test_rename (test.test_pep277.UnicodeNFKDFileTests) ... ok ====================================================================== ERROR: test_normalize (test.test_pep277.UnicodeFileTests) ---------------------------------------------------------------------- test test_pep277 crashed -- <class 'UnicodeEncodeError'>: 'ascii' codec can't encode characters in position 222-239: ordinal not in range(128) Traceback (most recent call last): File "./Lib/test/regrtest.py", line 905, in runtest_inner File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_pep277.py", line 195, in test_main UnicodeNFKDFileTests, File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1000, in run_unittest _run_suite(suite) File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 974, in _run_suite result = runner.run(suite) File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/runner.py", line 158, in run result.printErrors() File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/runner.py", line 108, in printErrors self.printErrorList('ERROR', self.errors) File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/runner.py", line 116, in printErrorList self.stream.writeln("%s" % err) File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/runner.py", line 24, in writeln self.write(arg) UnicodeEncodeError: 'ascii' codec can't encode characters in position 222-239: ordinal not in range(128)
msg103469 - (view)	Author: STINNER Victor (vstinner) *	Date: 2010-04-18 08:02
See also #8422.
msg105648 - (view)	Author: Florent Xicluna (flox) *	Date: 2010-05-13 17:56
Some diagnosis information ... - happens for 2 tests inside test_pep277: test_listdir and test_normalize - happens only for 4 weird filenames (added with #8180) - happens only on "x86 Tiger" buildbot ("PPC Tiger" does not report this error) I've added some lines to regrtest.py to identify each platform: "x86 Tiger": Darwin-8.10.1-i386-32bit little-endian (OS X 10.4.10) "PPC Tiger": Darwin-8.11.0-Power_Macintosh-powerpc-32bit big-endian (OS X 10.4.11) I don't know how the endianness may be linked with this issue. Failing files (4): u'\u0385\u03d3\u03d4', u'\u00a8\u0301\u03d2\u0301\u03d2\u0308', # == NFD(u'\u0385\u03d3\u03d4') u'\u0020\u0308\u0301\u038e\u03ab', # == NFKC(u'\u0385\u03d3\u03d4') u'\u1e9b\u1fc1\u1fcd\u1fce\u1fcf\u1fdd\u1fde\u1fdf\u1fed', One solution is to skip these files on all 'darwin' platforms, without losing more time on this issue.
msg105651 - (view)	Author: Ronald Oussoren (ronaldoussoren) *	Date: 2010-05-13 18:43
It prefer if it were clear why these tests fail (but don't have time to look into this at the moment). That is, I don't mind if the tests get disabled on darwin, but only if there is a documented reason for it because otherwise disabling the test might hide a real issue.
msg105662 - (view)	Author: Mark Dickinson (mark.dickinson) *	Date: 2010-05-13 21:47
> It prefer if it were clear why these tests fail I agree. There's little point having tests if we're just going to disable them when they fail without understanding why they're failing first. At that point a green buildbot just means "Hey, we disabled all failing tests!", rather than being an indication that Python's working well on that platform.
msg105669 - (view)	Author: Florent Xicluna (flox) *	Date: 2010-05-13 23:42
The gory details... >>> import os, unicodedata >>> tempdir = u'temp_pep277' >>> name = u'\u0385\u03d3\u03d4' >>> os.mkdir(tempdir) >>> with open(os.path.join(tempdir, name), 'w') as f: ... f.write('foo') >>> unicodedata.normalize('NFD', name) u'\xa8\u0301\u03d2\u0301\u03d2\u0308' >>> os.listdir(tempdir) [u'\xa8\u0301\u03d2\u03d2\u0301\u0308'] The filename should use the Normal Decomposition (NFD) of the original filename. But the decomposition looks wrong on this platform. The problem is similar for the other special filenames. Examples: - u'\u0385\u03d3\u03d4' ==> NFD: u'\xa8\u0301\u03d2\u0301\u03d2\u0308' ==> OSX: u'\xa8\u0301\u03d2\u03d2\u0301\u0308' - u' \u0308\u0301\u038e\u03ab' ==> NFD: u' \u0308\u0301\u03a5\u0301\u03a5\u0308' ==> OSX: u' \u0308\u0301\u03a5\u03a5\u0301\u0308' - u'\u1e9b\u1fc1\u1fcd\u1fce\u1fcf\u1fdd\u1fde\u1fdf\u1fed' ==> NFD: u'\u017f\u0307\xa8\u0342\u1fbf\u0300\u1fbf\u0301\u1fbf\u0342\u1ffe\u0300\u1ffe\u0301\u1ffe\u0342\xa8\u0300' ==> OSX: u'\u017f\u0307\xa8\u0342\u1fbf\u0300\u1fbf\u0301\u1fbf\u1ffe\u1ffe\u1ffe\xa8\u0342\u0300\u0301\u0342\u0300' I have no idea how to investigate further. And the "PPC Tiger" buildbot does not fail here. Reference about OSX decomposition: http://developer.apple.com/mac/library/qa/qa2001/qa1173.html
msg105674 - (view)	Author: STINNER Victor (vstinner) *	Date: 2010-05-14 00:46
> Reference about OSX decomposition: > http://developer.apple.com/mac/library/qa/qa2001/qa1173.html Mac OS X source code: http://www.opensource.apple.com/source/xnu/xnu-1504.3.12/bsd/vfs/vfs_utfconv.c?txt
msg114280 - (view)	Author: STINNER Victor (vstinner) *	Date: 2010-08-18 21:34
#8207 was the same issue and r79426 (for trunk and r79426 in py3k) was supposed to fix it.
msg119663 - (view)	Author: STINNER Victor (vstinner) *	Date: 2010-10-27 00:37
I opened the issue #10209: "Mac OS X: Decompose filenames on encode, and precompose filenames on decode".
msg119829 - (view)	Author: STINNER Victor (vstinner) *	Date: 2010-10-29 00:08
> I opened the issue #10209: "Mac OS X: Decompose filenames on encode, > and precompose filenames on decode". It was a bad idea. I fixed test_pep277 instead with: - r85897 disables the filenames that are normalized differently by Python and by darwin - r85899 disables test_normalize and test_listdir tests And test_pep277 now pass on "x86 Tiger 3.x" buildbot, and so I can close this issue (and issue #10209).