[Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest) (original) (raw)
Victor Stinner victor.stinner at gmail.com
Tue Jul 4 05:55:01 EDT 2017
- Previous message (by thread): [Python-Dev] Need help to review a test_nntplib enhancement:
- Next message (by thread): [Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest)
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
On Python 2, the addition of Lib/test/bisect.py caused conflict with Lib/bisect.py when running the Python test suite :-( I chose to rename Lib/test/bisect.py to Lib/test/bisectcmd.py to reduce changes caused by this new debug tool. So only in Python 2.7, you have to run:
./python -m test.bisectcmd ...
instead of
./python -m tes.bisect ...
See http://bugs.python.org/issue30843 for more information.
Victor
2017-06-16 18:05 GMT+02:00 Victor Stinner <victor.stinner at gmail.com>:
Hi,
Last weeks, I worked on a new tool to bisect failing tests because it's painful to bisect manually reference leaks (I remove as much code as possible until the code is small enough to be reviewable manually). See the bisecttest.py script attached to this issue: http://bugs.python.org/issue29512 With the help of Louie Lu, I added new --list-cases option to "python -m test", so you can now list all test cases and write it into a text file: ./python -m test --list-cases testos > tests I also added a new --matchfile option, to filter tests using a text file which contains one pattern per line: ./python -m test --matchfile=tests testos fnmatch is used to match test names, so "*" joker character can be used in test names.
My bisection tool takes a text file with the --matchfile format (one pattern per line) and creates a random subset of tests with half of the tests. If tests still fail, use the subset. Otherwise, create a new random subset. Loop until the subset contains a single test (configurable threshold, -n command line option). The long term plan is to integrate the bisection feature directly into regrtest.
Right now, my script is hardcoded to bisect reference leak bugs, but it should be easy to modify it to bisect other test issues like test creating files without removing it ("ENVCHANGED" failure in regrtest). For example, a core file is dumped when running testsubprocess on FreeBSD buildbots: http://bugs.python.org/issue30448 But I'm unable to reproduce the issue on my FreeBSD. It would be nice to be able to automate the bisection on the buildbot directly. --list-cases and --matchfile options are now available in 2.7, 3.5, 3.6 and master (3.7) branches. TODO: doctest tests are only partially supported, see: http://bugs.python.org/issue30683 Victor
- Previous message (by thread): [Python-Dev] Need help to review a test_nntplib enhancement:
- Next message (by thread): [Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest)
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]