Issue 11955: 3.3 : test_argparse.py fails 'make test' (original) (raw)

Hi - I've been experiencing many errors trying to build any version of Python that will pass its test suite - see issues : #11946 , #11954 - and now I've been advised to raise bugs about each test failure - hence this bug. For details of my config and build procedure, please see : issue #11954 .

So, running the new ./python fails test_argparse :

$ LD_LIBRARY_PATH=`pwd` LD_PRELINK=`pwd`/libpython3.3.so.1.0
./python /usr/src/cpython/Lib/test/test_argparse.py ...

FAIL: test_failures_many_groups_listargs (main.TestFileTypeW)

Traceback (most recent call last):
File "/usr/src/cpython/Lib/test/test_argparse.py", line 216, in wrapper test_func(self)
File "/usr/src/cpython/Lib/test/test_argparse.py", line 235, in test_failures raises(ArgumentParserError, parser.parse_args, args)
AssertionError: ArgumentParserError not raised by parse_args

====================================================================== FAIL: test_failures_many_groups_sysargs (main.TestFileTypeW)

====================================================================== FAIL: test_failures_no_groups_listargs (main.TestFileTypeW)

====================================================================== FAIL: test_failures_no_groups_sysargs (main.TestFileTypeW)

====================================================================== FAIL: test_failures_one_group_listargs (main.TestFileTypeW)

====================================================================== FAIL: test_failures_one_group_sysargs (main.TestFileTypeW)

Ran 1608 tests in 5.293s

FAILED (failures=6) Traceback (most recent call last): File "/usr/src/cpython/Lib/test/test_argparse.py", line 4712, in test_main()
File "/usr/src/cpython/Lib/test/test_argparse.py", line 4704, in test_main support.run_unittest(name)
File "/usr/src/cpython/Lib/test/support.py", line 1208, in run_unittest
_run_suite(suite)
File "/usr/src/cpython/Lib/test/support.py", line 1191, in _run_suite
raise TestFailed(err)
test.support.TestFailed: multiple errors occurred

Now, as someone with not too recent python programming experience, I find it very difficult to understand exactly what is being tested by :

class TestFileTypeW(TempDirMixin, ParserTestCase): """Test the FileType option/argument type for writing files"""

def setUp(self):
    super(TestFileTypeW, self).setUp()
    self.create_readonly_file('readonly')

argument_signatures = [
    Sig('-x', type=argparse.FileType('w')),
    Sig('spam', type=argparse.FileType('w')),
]
failures = ['-x', '', 'readonly']
successes = [
    ('foo', NS(x=None, spam=WFile('foo'))),
    ('-x foo bar', NS(x=WFile('foo'), spam=WFile('bar'))),
    ('bar -x foo', NS(x=WFile('foo'), spam=WFile('bar'))),
    ('-x - -', NS(x=sys.stdout, spam=sys.stdout)),
]

But it seems at least one bug is self-evident : test_argparse.py's error messages are essentially meaningless, and contain no useful information (beyond the name) about what failed or why - they are all essentially duplicates of :

Traceback (most recent call last): File "/usr/src/cpython/Lib/test/test_argparse.py", line 216, in wrapper test_func(self) File "/usr/src/cpython/Lib/test/test_argparse.py", line 235, in test_failures raises(ArgumentParserError, parser.parse_args, args)

So why emit 6 copies of the same meaningless message ?

I'm curious: how do you expect those error messages to help people track down the source of the bug when every error message contains the same data and line numbers, and they are line numbers not of specific tests, but of some "error handler" routine ?

The test names are:

FAIL: test_failures_many_groups_listargs (main.TestFileTypeW)
FAIL: test_failures_many_groups_sysargs (main.TestFileTypeW)
FAIL: test_failures_no_groups_listargs (main.TestFileTypeW)
FAIL: test_failures_no_groups_sysargs (main.TestFileTypeW)
FAIL: test_failures_one_group_listargs (main.TestFileTypeW)
FAIL: test_failures_one_group_sysargs (main.TestFileTypeW)

So they differ by [many_groups, no_groups, one_group] and [listargs, sysargs]

There are about 170 tests in test_argparse that get multiplied by 6 in this way. If this replication was removed it would cut the number of tests and time to nearly a third.

This replication is not useful. listargs and sysargs differ only in how a list of arguments is passed to parse_args, either directly, or via sys.argv. It should be sufficient to test these alternatives once, not 170 times.

'one group' creates an argument group, and adds all arguments to that. 'many groups' creates an argument group for each added argument. But argument groups are not used for 'parse_args'. They are only used when formatting help. By default all the arguments of parser are already put into one of two groups, 'optional arguments' or 'positional arguments'.

There are tests for the help formatting, but they don't use this ParserTesterMetaclass.

I would recommend removing this test replication. It is both unnecessary and a source of confusion. The simplest would be to change the end of the ParserTesterMetaclass.

class ParserTesterMetaclass
    def __init__()
    ....
        for add_arguments in [no_groups]:
            for parse_args in [listargs]:
                AddTests(cls, add_arguments, parse_args)

I suspect there is some confusion among users and developers as to what argument groups do.

As for the TestFileTypeW error, I suspect it has to do with file writing privileges. I haven't encountered it, but I running tests in my own directory.