[Python-Dev] devguide: Issue #18871: make it more explicit that the test suite should be run before (original) (raw)

Antoine Pitrou solipsis at pitrou.net
Thu Aug 29 08:40:29 CEST 2013

On Wed, 28 Aug 2013 20:05:25 -0400 Terry Reedy <tjreedy at udel.edu> wrote:

On 8/28/2013 5:51 PM, antoine.pitrou wrote:

> +Does the test suite still pass? There are several assumptions packed in that question. 0. The test suite is relevant to the patch. Not true for doc patches, some idlelib patches, and probably some others.

That was implicit in the formulation: "You must :ref:run the whole test suite <runtests> to ensure that it passes before pushing any code changes" (not doc changes, spelling corrections, etc.).

The only reason to know that the test suite isn't relevant to a code change is to... run the test suite. Don't assume that you are omniscient and know that your change won't break something. Developers not being omniscient is why we have a test suite in the first place.

(btw, if idlelib isn't tested, it should!)

1. The test suite runs to completion.

For at least a couple of years, that was not true on Windows desktops. The popup Runtime Error boxes from now are now gone, but a month ago, a crashing test froze Command Prompt.

But was the test corrected? Did you open an issue at the time? Did you try to debug the error?

We are collectively responsible for Python's quality. This is not some third-party software you have no control on.

2. The test suite runs without error.

I have hardly ever seen this on Windows, ever (with sporadic runs over several years). Today, testemail and testsax failed in successive runs*. 3. If there is a new failure, it is due to the patch. There have been at least one, but I think more, intermittent failures of Windows tests in the last few months

Same answer as above.

4. The gain of answering that question is worth the cost.

Accepting responsibility for one's own changes is part of why we trust each others as committers. The cost of you running the test suite is smaller than the cost of other developers trying to investigate a sudden buildbot failure, where it comes from etc.

Having you run the test suite encourages you to be conscious of its existence, of its imperfections, and to feel responsible in making it better. Shrugging it off because it sometimes doesn't work isn't helpful. We are striving to make it better (both in coverage and in robustness).

I worry that further emphasizing an overly broad, time-consuming, and sometimes impractical rule will only discourage more participation.

IMO, this is a calculated risk that is worthwhile to take. Times have changed, and being rigorous with testing is central in most successful projects nowadays.

Regards

Antoine.

More information about the Python-Dev mailing list