[Python-Dev] Guidance regarding tests for the standard lib (original) (raw)

Terry Reedy tjreedy at udel.edu
Wed Aug 14 06:00:58 CEST 2013

On 8/13/2013 7:51 PM, Steven D'Aprano wrote:

http://bugs.python.org/issue18606

Tests at end of statistics.patch.

and I'm about to submit a patch containing my updated code and tests, but I've run into a problem with testing. My existing tests use unittest, and follow the basic boilerplate documented here:

http://docs.python.org/3/library/test.html

+def test_main():

This is faulty, as explained below. It is not the boilerplate in http://docs.python.org/3/library/test.html#writing-unit-tests-for-the-test-package

which is correct. Compress to just

if name == "main": unittest_main()

To avoid all doubt, the tests pass if I call them like this: ./python Lib/test/teststatistics.py

That is a faulty test as explained below.

Patch applies cleanly on Win7, 32bit build. And indeed, F:\Python\dev\py34\PCbuild>python_d ../Lib/test/test_statistics.py works. (the progess dots will have to go before being applied ;-).

The above should be equivalent to ...> python -m test.test_statistics. but it is not.

F:\Python\dev\py34\PCbuild>python_d -m test.test_statistics Traceback (most recent call last): File "F:\Python\dev\py34\lib\runpy.py", line 160, in _run_module_as_main "main", fname, loader, pkg_name) File "F:\Python\dev\py34\lib\runpy.py", line 73, in _run_code exec(code, run_globals) File "F:\Python\dev\py34\lib\test\test_statistics.py", line 16, in from test_statistics_approx import NumericTestCase ImportError: No module named 'test_statistics_approx'

Your test depends on Lib/test being the current directory, at the beginning of the path. It should not. Your import must be from test.test_statistics_appox import NumericTestCase

With this fixed, the above command works.

Once the test runs from anywhere with unittest, run with regrtest,

...> python -m test test_statistics

Initially, this gives a traceback with the same last three lines. With the revision, I get

File "F:\Python\dev\py34\lib\unittest\loader.py", line 113, in loadTestsFromName parent, obj = obj, getattr(obj, part) AttributeError: 'module' object has no attribute 'test_statistics'

This is related to the first fault I mentioned above. If regrtest finds the old style 'test_main', it uses the regrtest loader. If it does not, it uses the unittest loader. This is documented in the code ;-). With the proper 2-line boilerplate as a second change, the second line now works.

but raise errors when I call them like this: ./python -m test -j3

LOL. In one jump, you changed the current directory, the test runner, the number of test files run, and the mode of testing. Try just one change at a time.

When you run the suite, you run test_statistics_approx.py, including the test case imported into test_statistics. (Is it run twice?).

Tested as part of the suite gives a bizarre message I do not pretend to understand. [290/379/3] test_statistics_approx Usage: regrtest.py [options]

regrtest.py: error: no such option: --slaveargs test test_statistics_approx crashed -- Traceback (most recent call last): File "F:\Python\dev\py34\lib\optparse.py", line 1391, in parse_args stop = self._process_args(largs, rargs, values) File "F:\Python\dev\py34\lib\optparse.py", line 1431, in _process_args self._process_long_opt(rargs, values) File "F:\Python\dev\py34\lib\optparse.py", line 1484, in _process_long_opt opt = self._match_long_opt(opt) File "F:\Python\dev\py34\lib\optparse.py", line 1469, in _match_long_opt return _match_abbrev(opt, self._long_opt) File "F:\Python\dev\py34\lib\optparse.py", line 1674, in _match_abbrev raise BadOptionError(s) optparse.BadOptionError: no such option: --slaveargs

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "F:\Python\dev\py34\lib\test\regrtest.py", line 1305, in runtest_inner test_runner() File "F:\Python\dev\py34\lib\test\test_statistics_approx.py", line 597, in test_main unittest.main() File "F:\Python\dev\py34\lib\unittest\main.py", line 124, in init self.parseArgs(argv) File "F:\Python\dev\py34\lib\unittest\main.py", line 148, in parseArgs options, args = parser.parse_args(argv[1:]) File "F:\Python\dev\py34\lib\optparse.py", line 1393, in parse_args self.error(str(err)) File "F:\Python\dev\py34\lib\optparse.py", line 1573, in error self.exit(2, "%s: error: %s\n" % (self.get_prog_name(), msg)) File "F:\Python\dev\py34\lib\optparse.py", line 1563, in exit sys.exit(status) SystemExit: 2

So back up and test it by itself first. It passes under unittest, but needs the revised boilerplate for regrtest. The two together work fine. When I rerun the suite, poof!, the mysterious tracback is gone and both pass.

The following appears after the 2nd. ''' ..

Ran 2 tests in 0.000s

OK ''' Perhaps this is from the doctest. If so, it probably should be surpressed.

One last thing: test order. Buildbots randomize the order of running files, so if your tests modify the environment in a way that affects another test, it might cause occasional failures.

-- Terry Jan Reedy

More information about the Python-Dev mailing list