(original) (raw)

On Sun, Dec 30, 2012 at 5:33 AM, Eli Bendersky <eliben@gmail.com> wrote:

On Sat, Dec 29, 2012 at 5:34 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On Sun, Dec 30, 2012 at 10:46 AM, Benjamin Peterson <benjamin@python.org> wrote:

> 2012/12/29 Eli Bendersky <eliben@gmail.com>:

>> Hi,

>>

>> This came up while investigating some test-order-dependency failures in

>> issue 16076.

>>

>> test___all__ goes over modules that have `__all__` in them and does 'from

>> import *' on them. This leaves a lot of modules in sys.modules,

>> which may interfere with some tests that do fancy things with sys,modules.

>> In particular, the ElementTree tests have trouble with it because they

>> carefully set up the imports to get the C or the Python version of etree

>> (see issues 15083 and 15075).

>>

>> Would it make sense to save the sys.modules state and restore it in

>> test___all__ so that sys.modules isn't affected by this test?

>

> Sounds reasonable to me.

I've tried this as an inherent property of regrtest before (to resolve
some problems with test\_pydoc), and it didn't work - we have too many
modules with non-local side effects (IIRC, mostly related to the copy
and pickle registries).

Given that it checks the whole standard library, test\_\_\_all\_\_ is
likely to run into the same problem.

Yes, I'm running into all kinds of weird problems when saving/restoring sys.modules around test\_\_\_all\_\_. This is not the first time I get to fight this test-run-dependency problem and it's very frustrating.

This may be a naive question, but why don't we run the tests in separate interpreters? For example with -j we do (which creates all kinds of strange intermittent problems depending on which tests got bundled into the same process). Is this a matter of performance? Because that would help get rid of these dependencies between tests, which would probably save core devs some work and headache.

For test\_\_\_all\_\_ having it run itself within a subprocess to guarantee it is in its own interpreter is wise.

Doing that for all tests by default though is a good way to hide problems with particular tests and underhandedly encourage more bad behavior from tests messing up the global environment as there are no consequences. We should aim to keep our tests as clean as possible. That doesn't mean that they \_are\_ (we've got a \_lot\_ of old messy code) but it is a good goal.

-gps

�

After all, since a lot of the interpreter state is global (for example sys.modules), does it not make sense to run each test in a clean environment? Many tests do fancy things with the global environment which makes them difficult to keep clean and separate.

�
Hence test.support.import\_fresh\_module - it can ensure you get the
module you want, regardless of the preexisting contents of
sys.modules. (http://docs.python.org/dev/library/test#test.support.import\_fresh\_module)

Yes, this is the solution currently used in test\_xml\_etree. However, once pickling tests are added things stop working. Pickle uses \_\_import\_\_ to import the module a class belongs to, bypassing all such trickery. So if test\_\_\_all\_\_ got \_elementtree into sys.modules, pickle's \_\_import\_\_ finds it even if all the tests in test\_xml\_etree manage to ignore it for the Python version because they use import\_fresh\_module.

Eli

_______________________________________________

Python-Dev mailing list

Python-Dev@python.org

http://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org