| msg57648 - (view) |
Author: Erik Andersén (erik_andersen) |
Date: 2007-11-19 18:32 |
| When a unittest test case raises an Exception, that test case is considered a failure. However, raising NotImplementedError is not a failure. It is something completely normal during development and simply indicates that the functionality has not yet been implemented. Compare this to a test case that genuinely fails, which means that the implementation does something different from what is is supposed to do. I therefore think test cases raising NotImplementedError should not be treated as failures, but be reported separately as what they are -- parts of the program that have not yet been implemented. Reporting such test cases as failures will give a lot of failure warnings that may detract attention from those cases that are genuinely errors. Also, SimpleTextRunner will report a lot of useless stack traces for these cases, thereby makeing it harder to find the stack traces for those tests that realy failed. I have made a patch to unittest that reports such cases as "not implemented" rather than as failures. This includes changes to the documentation and unit test case for unittest and doctest. I have included "added/changed in version 2.6" remarks in the documentation, so you have to changed these if the patch gets into some other version. Patch against 59056 attached |
|
|
| msg57649 - (view) |
Author: Guido van Rossum (gvanrossum) *  |
Date: 2007-11-19 18:38 |
| No. If you're testing something that's not implemented, it is correct to see that as a failure. Talk to anyone doing TDD. |
|
|
| msg57651 - (view) |
Author: Raghuram Devarakonda (draghuram)  |
Date: 2007-11-19 18:42 |
| I don't think unittest automatically treats all exceptions as failures. Failures are those that are explicitly flagged with assert* and fail* methods. All other exceptions result in "errors". I think what you are asking for is to special case one such error but am not sure if it is worth it. |
|
|
| msg57655 - (view) |
Author: Erik Andersén (erik_andersen) |
Date: 2007-11-19 19:20 |
| Raghuram Devarakonda skrev: > Raghuram Devarakonda added the comment: > > I don't think unittest automatically treats all exceptions as failures. > Failures are those that are explicitly flagged with assert* and fail* > methods. All other exceptions result in "errors". I think what you are > asking for is to special case one such error but am not sure if it is > worth it. > > ---------- > nosy: +draghuram > > __________________________________ > Tracker <report@bugs.python.org> > <http://bugs.python.org/issue1466> > __________________________________ > You are right, I used the wrong word. I mean that NotImplementedError should be treated as a special *error*. If you have ever written the test cases before you write the code, you will get a huge amount of uninteresting output from errors that come from tests to code you havn't written yet. This will drown the real errors and failures from the code you are actually testing and debugging. What I want is to have unittest report these differently so that I can see what are the real bugs the code, not to spam me with tracebacks I will never look at for test cases that refer to not implemented code. Erik |
|
|
| msg57656 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2007-11-19 19:21 |
| Sorry, just comment out those tests until you're ready to write the code. |
|
|
| msg57658 - (view) |
Author: Erik Andersén (erik_andersen) |
Date: 2007-11-19 19:31 |
| Possible, but cumbersome. You have to analyze each case and see if it depends on code not written. When you implement a new piece of code, you have to go through all your tests to see if that one should be included. Also you don't get any reporting on the coverage of the code. If you accidentally forget to uncomment a test case, you may never realize that you have a bug -- at least not until it happens in real life. Erik |
|
|
| msg57659 - (view) |
Author: Raghuram Devarakonda (draghuram) |
Date: 2007-11-19 19:51 |
| > Possible, but cumbersome. You have to analyze each case and see if it Have you tried to write a Test Runner that ignores NotImplementedError? You may have to override TestResult.addError. |
|
|
| msg57661 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2007-11-19 20:09 |
| Sorry Erik, but I don't think you should try to compensate for your flawed methodology by trying to change the unittest framework. End of discussion. |
|
|
| msg57662 - (view) |
Author: Erik Andersén (erik_andersen) |
Date: 2007-11-19 20:09 |
| No, I definitely don't want to ignore them, I want them reported in a manner that is relevant to them. Just say that the test was not implemented. No tracebacks for debugging since there is nothing to debug. In addition to a TestRunner I need a new TestCase and a new TestResult, which is what is in the patch, although I changed the classes rather than inherit from them. Erik |
|
|