| msg43364 - (view) |
Author: Thomas Heller (theller) *  |
Date: 2003-04-16 17:49 |
| This patch enables more useful output for unittests: If a test crashes (raises an unexpected exception), a full traceback is printed. If a test failes, the output is something like this: ======================================== FAIL: test_failUnlessEqual (__main__.FailingTests) ---------------------------------------------------------------------- TestFailed: 0 != 1 File "xunit.py", line 12, in test_failUnlessEqual self.failUnlessEqual(self.a, self.b) ======================================== Before, this was printed: ======================================== FAIL: test_failIfEqual (__main__.FailingTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "xunit.py", line 15, in test_failIfEqual self.failIfEqual(self.a, self.a) File "c:\python23\lib\unittest.py", line 300, in failIfEqual raise self.failureException, \ AssertionError: 0 == 0 ======================================== If needed, I can upload the test script I use, together with the results before and after the patch. This has shortly been discussed on c.l.p, response was mostly positive. http://tinyurl.com/9obf |
|
|
| msg43365 - (view) |
Author: Thomas Heller (theller) * |
Date: 2003-04-17 15:31 |
| Logged In: YES user_id=11105 Attaching new version of the patch (unittest-2.diff). This gives better output for failUnlessRaises, like this: ====================================================================== FAIL: test_failUnlessRaises (__main__.FailingTests) ---------------------------------------------------------------------- TestFailed: wrong exception, expected TypeError got: 'ValueError: 10' File "xunit.py", line 18, in test_failUnlessRaises self.failUnlessRaises(TypeError, self._raise, ValueError, 10) ====================================================================== FAIL: test_failUnlessRaises_2 (__main__.FailingTests) ---------------------------------------------------------------------- TestFailed: wrong exception, expected TypeError, IndexError, or AttributeError got: 'ValueError: 10' File "xunit.py", line 21, in test_failUnlessRaises_2 self.failUnlessRaises((TypeError, IndexError, AttributeError), self._raise, ValueError, 10) ---------------------------------------------------------------------- |
|
|
| msg43366 - (view) |
Author: Brett Cannon (brett.cannon) * |
Date: 2003-04-20 02:57 |
| Logged In: YES user_id=357491 I like the new output, personally. I am +1 on letting Thomas add the changes. Does this mean we no longer treat unittest as a separate project? |
|
|
| msg43367 - (view) |
Author: Steve Purcell (purcell)  |
Date: 2003-04-25 11:52 |
| Logged In: YES user_id=21477 This behaviour of trimming the traceback was implemented in a previous version of PyUnit, but dropped because it did not work with Jython. My aim is that the same 'unittest.py' should work out of the box with both CPython and Jython. |
|
|
| msg43368 - (view) |
Author: Thomas Heller (theller) * |
Date: 2003-04-25 12:05 |
| Logged In: YES user_id=11105 What a pity! What exactly does not work in Jython? Before giving up on this, there are at least two ways to proceed: - Behave as before in Jython, and use the better output in CPython. - Apply this patch only the the unittest version bundled with CPython. How are the chances for one of this? |
|
|
| msg43369 - (view) |
Author: Steve Purcell (purcell) |
Date: 2003-04-25 12:37 |
| Logged In: YES user_id=21477 After investigation, this seems to work with Jython (though not JPython, which didn't have tb_next etc.). In general I've been trying hard to keep 'unittest.py' vanilla, since a lot of people are still using it with Python 1.5 and even JPython. Hence the complete absence of string methods, list comprehensions and other new language features. Don't know if this policy makes sense in the longer term, but I value it right now. In that sense, I'm not sure if it's worth changing the message. |
|
|
| msg43370 - (view) |
Author: Thomas Heller (theller) * |
Date: 2003-04-25 12:52 |
| Logged In: YES user_id=11105 Last attempt to convince you: I could try to port the changes to Python 1.5, if you want to stay compatible. If you still reject the patch (you're the unittest boss), I'll have to live with subclassing unittest ;-) |
|
|
| msg43371 - (view) |
Author: Raymond Hettinger (rhettinger) * |
Date: 2003-05-06 07:11 |
| Logged In: YES user_id=80475 I would like to see Thomas's patch or some comformant variant go in. Usability problems are a bug. Friendlier output makes it more likely that unittest will be used in the first place. |
|
|
| msg43372 - (view) |
Author: Tim Peters (tim.peters) * |
Date: 2003-05-06 19:52 |
| Logged In: YES user_id=31435 I'm split. The current output when assertRaises fails is a frequent cause of head-scratching ("what? it's complaining because ValueError got raised? ... no, it's complaining because ValueError wasn't raised? ..."). OTOH, I see no value in trimming the traceback. Now that *could* be because the assertRaises output can be so confusing that we end up using the rest of the traceback to figure out what unittest is trying to tell us in those cases. |
|
|
| msg43373 - (view) |
Author: Thomas Heller (theller) * |
Date: 2003-05-06 20:08 |
| Logged In: YES user_id=11105 That's exactly how I was feeling. When an assertRaises test failed, I usually inserted the call it made before this line, to see the real traceback. And that's what this patch tries to fix. I don't want to see tracebacks when a test fails, I want a clear indication that it failed (the patch prints "TestFailed" instead of "Traceback:"). For the output of a failed assertRaises, see the first comment I added. IMO it clearly says what which exception was expected, and which one was raised. |
|
|
| msg43374 - (view) |
Author: Tim Peters (tim.peters) * |
Date: 2003-05-06 20:27 |
| Logged In: YES user_id=31435 That's why I'm split: I do like the new *messages* better (their content), but I don't like losing the tracebacks. Sometimes it's a bug in the test driver-- or in 20 layers of test driver code --and sometimes it's even a bug in unittest itself. The traceback is a fundamental tool when things go wrong, so I'm never in favor of hiding parts of tracebacks (hiding could be appropriate if you *knew* the true cause isn't in the part you're hiding -- but you can't know that). |
|
|
| msg43375 - (view) |
Author: Thomas Heller (theller) * |
Date: 2003-10-14 18:05 |
| Logged In: YES user_id=11105 Assigned to Steve for pronouncement (didn't he already comment on python-dev some time ago?) |
|
|
| msg43376 - (view) |
Author: Steve Purcell (purcell) |
Date: 2003-10-16 21:32 |
| Logged In: YES user_id=21477 I'm looking at all this and will certainly incorporate some of the suggestions: - I'm +1 on the clearer message for assertRaises() - I'm +1 on clearer messages for _all_ assert*()/fail*() methods - The TestFailed exception doesn't really add much, since AssertionError works well already - I'm loathe to ever suppress tracebacks, or fiddle with them much: the traceback is the canonical way for IDEs to find lines in the code |
|
|
| msg43377 - (view) |
Author: Thomas Heller (theller) * |
Date: 2003-12-05 20:39 |
| Logged In: YES user_id=11105 I've brought the patch up to date with current cvs, and made small changes: The 'Traceback: most recent call last' line is no longer removed from the print. And, according to Steve's suggestion, AssertionError is used again instead of the TestFailed exceptiom. The patch has been discussed in this thread on python-dev: http://mail.python.org/pipermail/python-dev/2003-December/040623.html New patch attached: unittest-3.diff |
|
|
| msg43378 - (view) |
Author: Thomas Heller (theller) * |
Date: 2003-12-05 20:41 |
| Logged In: YES user_id=11105 Attached a test script together with what is printed before and after the patch: test_sample.py |
|
|
| msg43379 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2003-12-05 21:40 |
| Logged In: YES user_id=6380 I love it! We're all waiting for a +1 from Steve so this can be checked into 2.4. (I'd love it in 2.3 too, but that's probably going to be blocked on the "no new features" rule. :-) |
|
|
| msg43380 - (view) |
Author: Thomas Heller (theller) * |
Date: 2003-12-05 23:07 |
| Logged In: YES user_id=11105 Cool, although after some thought I again like the TestFailed more than the AssertionError. (But I won't insist) |
|
|
| msg43381 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2003-12-05 23:18 |
| Logged In: YES user_id=6380 Thomas, Can you make the TestFailed issue a separate bug/patch/feature request? |
|
|
| msg43382 - (view) |
Author: Steve Purcell (purcell) |
Date: 2003-12-06 13:19 |
| Logged In: YES user_id=21477 Hi all, I've accepted this patch, with some modifications, and checked it into both Pyunit and Python CVS. If the callable passed to assertRaises() throws an exception other than the one expected, the test result should really be ERROR; the proposed changes to assertRaises() resulted in tracebacks for such unexpected exceptions being lost, since only the exception name and message were formatted into the failure string. I've therefore left the logic there as it was. Also, rather than hard-code the number of levels of traceback to skip, the changes I have checked in automatically determine the number of levels to skip. Please take a look and tell me if the trick I have used is excessively horrible. As for the separate TestFailed issue, for me this is not terribly attractive, since I know that many people find it very convenient to use the 'assert' keyword in their test code. (Note that by setting the 'failureException' attribute of your test cases, you can use an exception other than AssertionError.) For your reference, the output of Thomas' sample test script is now as follows: FFE ====================================================================== ERROR: test_4 (__main__.MyTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "heller.py", line 21, in test_4 self.assertRaises(ValueError, getattr, self, "spam") File "/export/home/steve/projects/pyunit/pyunit-cvs/unittest.py", line 319, in failUnlessRaises callableObj(*args, **kwargs) AttributeError: 'MyTestCase' object has no attribute 'spam' ====================================================================== FAIL: test_1 (__main__.MyTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "heller.py", line 6, in test_1 self.do_this() File "heller.py", line 9, in do_this self.do_that() File "heller.py", line 12, in do_that self.failUnlessEqual(1, 2) AssertionError: 1 != 2 ====================================================================== FAIL: test_3 (__main__.MyTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "heller.py", line 17, in test_3 self.assertRaises(AttributeError, getattr, self, "silly") AssertionError: AttributeError not raised ---------------------------------------------------------------------- Ran 3 tests in 0.008s FAILED (failures=2, errors=1) |
|
|