[Python-Dev] Better unittest failures (original) (raw)

Ron Adam rrr at ronadam.com
Fri Sep 21 00:03:19 CEST 2007

The value of a unittest test is not in how well they pass, but in how well they fail.

While looking at possibly helping with the str_uni branch when that was going on I found that in some cases unittest failure results can take a little bit (or a lot) of work to figure out just what was failing, where and why.

While helping Eric test the new format function and class I came up with a partial solution which may be a bases for further improvements. Eric told me it did help quite a bit. So I think it's worth looking into.

Since we were running over a hundred different options over several different implementations to make sure they all passed and failed in the same way, we were using data based test cases so we could easily test the same data with each version. Unfortunately that has a drawback that the traceback doesn't show what data was used when testing exceptions.

Additionally when something did fail it was not always obvious what and why it was failing.

One of the conclusions I came to is it would be better if tests did not raise standard python exceptions unless the test itself has a problem. By having tests raise special Test_Only exceptions, it can make the output of the test very much clearer.

Here are the added Test_Only Excepitons. These would only be in the unittest module to catch the following situations.

  Wrong_Result_Returned
  Unexpected_Exception_Raised
  No_Exception_Raised
  Wrong_Exception_Raised

And two new functions that use them.

  assertTestReturns(expect, test, message)
  assertTestRaises(expect, test, message)

These additions would not effect any existing tests. To use these requires the code to be tested to be wrapped in a function with no arguments. And it is the same format for both assertTestReturns and assertTestRaises.

  for data in testdata:
     expect, a, b, c = data
     def test():
         return foo(a, b, c)
     assertTestReturns(expect, test, repr(data))

Replacing all existing tests with this form isn't reasonable but adding this as an option for those who want to use it is very easy to do.

The test file I used to generate the following output is attached.

Cheers, Ron

Test output using standard assertEquals and assertRaises.

$ python ut_test.py EEFFFFFF

ERROR: test_A (main.test1_normal_failures)

Traceback (most recent call last): File "ut_test.py", line 100, in test_A result = some_function(*args, **kwds) File "ut_test.py", line 62, in some_function baz = kwds['baz'] KeyError: 'baz'

This fails as a test "error" instead of a test "fail".

What was args and kwds here?

====================================================================== ERROR: test_B (main.test1_normal_failures)

Traceback (most recent call last): File "ut_test.py", line 108, in test_B self.assertRaises(expect, test, args, kwds) File "unittest.py", line 320, in failUnlessRaises callableObj(*args, **kwargs) File "ut_test.py", line 107, in test return some_function(*args, **kwds) File "ut_test.py", line 62, in some_function baz = kwds['baz'] KeyError: 'baz'

Same as above. Fails as a test "error", unkown arguments

values for some_function().

====================================================================== FAIL: test_C (main.test1_normal_failures)

Traceback (most recent call last): File "ut_test.py", line 114, in test_C self.assertRaises(expect, test, args, kwds) AssertionError: KeyError not raised

What was args, and kwds values?

====================================================================== FAIL: test_D (main.test1_normal_failures)

Traceback (most recent call last): File "ut_test.py", line 120, in test_D repr((n, expect, args, kwds))) AssertionError: (8, ('Total baz:', 4), [1, 2], {'baz': 'Total baz:'})

This one is ok.

Test output using the added methods and test only exceptions with

the same test data.

* Test errors only occur on actual test "errors".

* The reason for the fail is explained in all cases for test "fails".

* The only time you get an actual python exception is when the test

it self has a problem. Otherwise you get an test_exception that refers to the exception in the actual code.

====================================================================== FAIL: test_A (main.test2_new_failures)

Traceback (most recent call last): File "ut_test.py", line 131, in test_A repr((n, expect, args, kwds))) File "ut_test.py", line 36, in assertTestReturns result = test() File "ut_test.py", line 129, in test return some_function(*args, **kwds) File "ut_test.py", line 62, in some_function baz = kwds['baz'] Unexpected_Exception_Raised: KeyError('baz',)

Reference: (2, ('Total baz:', 3), [1, 2], {'raz': 'Total baz:'})

====================================================================== FAIL: test_B (main.test2_new_failures)

Traceback (most recent call last): File "ut_test.py", line 138, in test_B repr((n, expect, args, kwds))) File "ut_test.py", line 45, in assertTestRaises result = test() File "ut_test.py", line 136, in test return some_function(*args, **kwds) File "ut_test.py", line 62, in some_function baz = kwds['baz'] Wrong_Exception_Raised: KeyError('baz',)

Reference: (4, <type 'exceptions.IndexError'>, [1, 2], {'raz': 'Total baz:'})

====================================================================== FAIL: test_C (main.test2_new_failures)

Traceback (most recent call last): File "ut_test.py", line 145, in test_C repr((n, expect, args, kwds))) File "ut_test.py", line 52, in assertTestRaises raise self.No_Exception_Raised(result, ref) No_Exception_Raised: returned -> ('Total baz:', 3)

Reference: (6, <type 'exceptions.KeyError'>, [1, 2], {'baz': 'Total baz:'})

====================================================================== FAIL: test_D (main.test2_new_failures)

Traceback (most recent call last): File "ut_test.py", line 152, in test_D repr((n, expect, args, kwds))) File "ut_test.py", line 41, in assertTestReturns raise self.Wrong_Result_Returned(result, ref) Wrong_Result_Returned: ('Total baz:', 3)

Reference: (8, ('Total baz:', 4), [1, 2], {'baz': 'Total baz:'})

Ran 8 tests in 0.004s

FAILED (failures=6, errors=2)

-------------- next part -------------- A non-text attachment was scrubbed... Name: ut_test.py Type: text/x-python Size: 4850 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20070920/134d74ee/attachment.py

More information about the Python-Dev mailing list