[Python-Dev] test failures on Debian unstable (original) (raw)

Tim Peters tim.one@comcast.net
Sun, 24 Nov 2002 04:53:08 -0500

[martin@v.loewis.de, on test_normalization]

What -u would you propose?

Someone who knows what the test does should name the resource.

testbsddb3 is already in already a regrtest -u thingie, so should be in the expected-skip list on all boxes.

I don't like the expected-skip mechanism at all.

I love it: it solves real problems on Windows. The latest example is that I never would have noticed that test_bsddb3 even existed if the mechanism hadn't told me it was an unexpected skip on Windows. Ditto test_normalization. Likewise Neil's report of expected skips on Debian pointed out oddities to him. On platforms where dozens of tests are skipped, just listing the tests that were skipped isn't informative enough -- the list is so big that you simply don't notice if a new test shows up there. Before this mechanism, entire new packages didn't make it into the Windows distro because nobody noticed that their test suite was getting skipped. That hasn't happened again.

Why does being Linux (or HP-UX) has anything to do with whether we can run the testnormalization test; that test is completely platform-independent.

The expected-skip list doesn't really have to do with whether we can run a test, it has to do with whether it's a red flag when we find we can't run a test.

Why is it expected that testbz2 works on Linux?

I don't know that it is.

It won't work if you don't have the libraries.

If so, put it in the expected-skip list for Linux. regrtest doesn't whine about expected-skip entries that do run to completion, it only points out tests that don't run to completion (due to ImportError or TestSkipped) that also aren't in the platform's expected-skip list.

Instead, I would expect that testbz2 works if the bz2 module is present, and fails to work if it was not built.

Then it's an expected skip on platforms where that's a choice, in the sense the phrase is intended. "expected skip" may be a poor name for it, although it makes literal sense on Windows.

Likewise for testnormalization: If a certain file is present, it should succeed, otherwise, it should fail.

Ditto, but even more so in this case, because that "certain file" isn't present on any system unless the user specifically downloads it and places it in their current working directory before running test_normalization. That makes it so unlikely this test will run for users that it should probably be put into the expected-skip list on all boxes via the _ExpectedSkips constructor (maybe a good idea for all -u tests).