[Python-Dev] Purpose of Doctests [Was: Best practices for Enum] (original) (raw)

Olemis Lang olemis at gmail.com
Tue May 21 01:35:34 CEST 2013

Hi ! ... sorry , I could not avoid to reply this message ...

On 5/20/13, Michael Foord <fuzzyman at voidspace.org.uk> wrote:

On 20 May 2013, at 18:26, Mark Janssen <dreamingforward at gmail.com> wrote:

I'm hoping that core developers don't get caught-up in the "doctests are bad meme".

Instead, we should be clear about their primary purpose which is to test the examples given in docstrings. In other words, doctests have a perfectly legitimate use case. But more than just one ;-) Another great use has nothing to do with docstrings: using an entire file as "a doctest". This encourages writing lots of text explaining what you're doing,. with snippets of code interspersed to illustrate that the code really does behave in the ways you've claimed. +1, very true. I think doctest excel in almost every way above UnitTests. I don't understand the popularity of UnitTests, except perhaps for GUI testing which doctest can't handle. I think people just aren't very imaginative about how to create good doctests that are also good documentation.

With enhanced doctests solution in mind ...

Doc tests have lots of problems for unit testing.

* Every line is a test with all output part of the test - in unit tests you only assert the specific details you're interested in

custom output checkers

* Unordered types are a pain with doctest unless you jump through hoops

( custom output checkers + doctest runner ) | (dutest tc global var)

* Tool support for editing within doctests is generally worse

this is true , let's do it !

* A failure on one line doesn't halt execution, so you can get many many reported errors from a single failure

it should if REPORT_ONLY_FIRST_FAILURE option [1]_ is set .

* Try adding diagnostic prints and then running your doctests!

I have ... dutest suites for my Trac plugins do so . However logging information is outputted to /path/to/trac/env/log/trac.log ... so a tail -f is always handy .

* Tools support in terms of test discovery and running individual tests is not as smooth

dutest offers two options since years ago MultiTestLoader combines multiple test loaders to load different kinds of tests at once from a module , whereas a package loader performs test discovery . These loader objects are composable , so if an instance of MultiTestLoader is supplied in to the package test loader then multiple types of tests are loaded out of modules all over across the package hierarchy .

Indeed , in +10 years of Python development I've never used unittest(2) discovery, and even recently implemented the one that's used in Apache™ Bloodhound test suite . Unfortunately I've had no much time to spend on improving all this support in dutest et al.

* Typing >>> and ... all the time is really annoying

... I have faith ... there should be something like this for vim ... I have faith ... ;)

* Doctests practically beg you to write your code first and then copy and paste terminal sessions - they're the enemy of TDD

Of course , not , all the opposite . If the approach is understood correctly then the first thing test author will do is to write the code «expected» to get something done . When everything is ok with API code style then write the code . Many problems in the API and inconsistencies are thus detected early .

* Failure messages are not over helpful and you lose the benefit of some of the new asserts (and their diagnostic output) in unittest

(custom ouput checkers) | ( dutest tc variable )

* Tests with non-linear code paths (branches) are more painful to express in doctests

that's a fact , not just branches , but also exceptions

Beyond this ...

My really short answer is that I do not agree with this . Like I just said in previous messages with enhanced support like the one offered by dutest (i.e. tc global var bound to an instance of unittest.TestCase) it's possible to invoke each and every unittest assertion method . So this may be seen all the other way round «unittest machinery is already used without even declaring a single test class» ... and so on ...

... so , in concept , there is no real benefit in using unittest over doctest if doctest module is eventually upgraded .

[...]

However doctests absolutely rock for testing documentation / docstring examples.

FWIW , +1

[...]

.. [1] doctest.REPORT_ONLY_FIRST_FAILURE (http://docs.python.org/2/library/doctest.html#doctest.REPORT_ONLY_FIRST_FAILURE)

-- Regards,

Olemis.

Apache™ Bloodhound contributor http://issues.apache.org/bloodhound

Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/

Featured article:

More information about the Python-Dev mailing list