Issue 9409: doctest in python2.7 can't handle non-ascii characters (original) (raw)

When trying to run my test suite I had a problem with python2.7. My suite ran 100% in Python2.4, Python2.5, Python2.6 and Python3.2a0, so I thought it would be a kind of doctest flaw.

Taking a look at the code, there is the following in doctest.py:1331:

        source = example.source.encode('ascii', 'backslashreplace')

The problem is that my doctest file had non-ascii files and I got trouble.

hugo@hugo-laptop:~/issue$ python2.7 example.py non-ascii.txt Doctest: non-ascii.txt ... ok ascii.txt Doctest: ascii.txt ... ERROR

====================================================================== ERROR: ascii.txt Doctest: ascii.txt

Traceback (most recent call last): File "/usr/local/lib/python2.7/doctest.py", line 2148, in runTest test, out=new.write, clear_globs=False) File "/usr/local/lib/python2.7/doctest.py", line 1382, in run return self.__run(test, compileflags, out) File "/usr/local/lib/python2.7/doctest.py", line 1272, in __run got += _exception_traceback(exc_info) File "/usr/local/lib/python2.7/doctest.py", line 244, in _exception_traceback traceback.print_exception(exc_type, exc_val, exc_tb, file=excout) File "/usr/local/lib/python2.7/traceback.py", line 125, in print_exception print_tb(tb, limit, file) File "/usr/local/lib/python2.7/traceback.py", line 69, in print_tb line = linecache.getline(filename, lineno, f.f_globals) File "/usr/local/lib/python2.7/linecache.py", line 14, in getline lines = getlines(filename, module_globals) File "/usr/local/lib/python2.7/doctest.py", line 1331, in __patched_linecache_getlines source = example.source.encode('ascii', 'backslashreplace') UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 19: ordinal not in range(128)

Ran 2 tests in 0.006s

FAILED (errors=1) hugo@hugo-laptop:~/issue$

Taking an inner look at doctest.py in python2.6 and python2.7 I realized there is another inconsistency with filenames in both (I was lucky to try at first a filename that doesn't match the regex):

__LINECACHE_FILENAME_RE = re.compile(r'<doctest '
                                     r'(?P<name>[\w\.]+)'
                                     r'\[(?P<examplenum>\d+)\]>$')

Well, is the file name, but filenames are not only composed of alphanums and dots. Maybe it should be slightly different, like:

__LINECACHE_FILENAME_RE = re.compile(r'<doctest '
                                     r'(?P<name>.+?)'
                                     r'\[(?P<examplenum>\d+)\]>$', re.UNICODE)

Because we can have several kinds of names. But it is not the top of the iceberg, anyaway.

To solve my problem, I propose moving back that first snippet to how it was in python2.6. The diff would be:

--- /usr/local/lib/python2.7/doctest.py 2010-07-28 22:07:01.272234398 -0300 +++ doctest.py 2010-07-28 22:20:42.000000000 -0300 @@ -1328,8 +1328,7 @@ m = self.__LINECACHE_FILENAME_RE.match(filename) if m and m.group('name') == self.test.name: example = self.test.examples[int(m.group('examplenum'))]