[Python-Dev] setUpClass and setUpModule in unittest (original) (raw)

Robert Collins robertc at robertcollins.net
Tue Feb 16 10:28:40 CET 2010

On Tue, 2010-02-16 at 02:09 -0500, Glyph Lefkowitz wrote:

> > In a recent message he was talking about either breaking > > compatibility with TestSuite implementations that override run(), > > or test-reordering - both of which I consider important, core > > features of the unittest module. > > Well, by "breaking compatibility with custom TestSuite > implementations that override run" I mean that is one possible place > to put the functionality. Code that does override it will not stop > working, it just won't support the new features. >

Ah, I see. This doesn't sound too bad, but I'd personally prefer it if the distinction were a bit more clearly drawn. I'd like frameworks to be able to implement extension functionality without having to first stub out functionality. In other words, if I want a test suite without setUpClass, I'd prefer to avoid having an abstraction inversion.

+1

> If we chose this implementation strategy there would be no > compatibility issues for existing tests / frameworks that don't use > the new features.

That's very good to hear.

It does however get tougher to be 'stdlib compatible' for frameworks that extend the stdlib - at least with how extensions work today.

> If tests do want to use the new features then the framework authors > will need to ensure they are compatible with them. This seems like a > reasonable trade-off to me. We can ensure that it is easy to write > custom TestSuite objects that work with earlier versions of unittest > but are also compatible with setUpClass in 2.7 (and document the > recipe - although I expect it will just mean that TestSuite.run > should call a single method if it exists). >

This is something that I hope Jonathan Lange or Robert Collins will chime in to comment on: expanding the protocol between suite and test is an area which is fraught with peril, but it seems like it's something that test framework authors always want to do. (Personally, I really want to do it because I want to be able to run things asynchronously, so the semantics of 'run()' need to change pretty dramatically to support that...) It might be good to eventually develop a general mechanism for this, rather than building up an ad-hoc list of test-feature compatibility recipes which involve a list of if hasattr(...): foo(); checks in every suite implementation.

Please have a look at the testtools.TestCase.run - its incomplete, but its working towards making it possible for trial to not need to replace run, but instead provide a couple of hooks (registered during setUp) to handle what you need. What it currently offers is catching additional exceptions for you, which is a common form of extension. bzrlib is using this quite successfully, and we deleted a lot of code that overlapped the stdlib unittest run().

> Perhaps a better idea might be to also add startTest and stopTest > methods to TestSuite so that frameworks can build in features like > timing tests (etc) without having to override run itself. This is > already possible in the TestResult of course, which is a more common > extensibility point in my experience. >

I think timing and monitoring tests can mostly be done in the TestResult class; those were bad examples. There's stuff like synthesizing arguments for test methods, or deciding to repeat a potentially flaky test method before reporting a failure, which are not possible to do from the result. I'm not sure that startTest and stopTest hooks help with those features, the ones which really need suites; it would seem it mostly gives you a hook to do stuff that could already be done in TestResult anyway.

Also its not really possible to 'run one thing' around a test at the moment - theres no good place (without changing tests or doing somewhat convoluted stuff) to have custom code sit in the stack above the test code - this makes it harder to handle:

This is also in my hit-list of things to solve-and-propose-for-stdlib-unittest that I blogged about a while back.

-Rob -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: <http://mail.python.org/pipermail/python-dev/attachments/20100216/cf999b44/attachment.pgp>

More information about the Python-Dev mailing list