bpo-37707: Exclude expensive unit tests from PGO task by nascheme · Pull Request #15009 · python/cpython (original) (raw)
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Conversation7 Commits3 Checks0 Files changed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
[ Show hidden characters]({{ revealButtonHref }})
Mark some individual tests to skip when --pgo is used. The tests
marked increase the PGO task time significantly and likely don't
help improve optimization of the final executable.
Marked tests:
pickletester
check_frame_opcodes
test_framing_many_objects
test_bz2
testCompress4G
testDecompress4G
test_itertools
test_long_chain_of_empty_iterables
test_lzma
test_compressor_bigmem
test_decompressor_bigmem
test_statistics
test_inv_cdf
https://bugs.python.org/issue37707
Mark some individual tests to skip when --pgo is used. The tests marked increase the PGO task time significantly and likely don't help improve optimization of the final executable.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! Such a pgo skip decorator approach is exactly what I was thinking of recommending when you brought this up. :)
I haven't looked through exactly what these tests do, but I'd say it's worth evaluating ("guessing") whether they represent a common scenario or not. Even if they run for a long time, if it's something that's going to run a lot in user code then I'd rather keep it in.
99% of realistic Python runtime ("evaluated"/"guessed") is success cases, but a lot of the quick tests in the suite are going to exercise setup/teardown and failure paths. Let's not over-focus on those for the sake of a quick PGO run.
I intended that --pgo will give you an optimized build without too much extra build time. Before PR #14702, the PGO builds took so long that in practice it seems that many did not bother to use them. If you don't care at all about long build time, then --pgo-extended is the option for that. If we make expensive tests run in the --pgo case, I don't see the point of --pgo-extended.
Something I was thinking about previously, --pgo-extended should not not skip the tests marked with @skip_if_pgo_task. That way, people who want the most extensive code coverage can still use --pgo-extended to get the old behavior.
Here is a comparison of test run time before and after my skip decorator.
| test | before [s] | after [s] |
|---|---|---|
| test_pickle | 4.86 | 1.79 |
| test_bz2 | 1.98 | 0.51 |
| test_itertools | 1.51 | 1.12 |
| test_lzma | 2.65 | 0.19 |
| test_statistics | 1.84 | 0.57 |
I haven't looked through exactly what these tests do, but I'd say it's worth evaluating ("guessing") whether they represent a common scenario or not.
I did two PGO builds with and without this PR and pyperformance shows no significant difference (all differences were <1%). I am using core isolation and affinity for them so I trust the stability of the result but I can repeat them if we need to double check.
Something I was thinking about previously, --pgo-extended should not not skip the tests marked with @skip_if_pgo_task. That way, people who want the most extensive code coverage can still use --pgo-extended to get the old behavior.
+1
Thanks @nascheme for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request
Mark some individual tests to skip when --pgo is used. The tests marked increase the PGO task time significantly and likely don't help improve optimization of the final executable. (cherry picked from commit 52a48e6)
Co-authored-by: Neil Schemenauer nas-github@arctrix.com
nascheme added a commit that referenced this pull request
Mark some individual tests to skip when --pgo is used. The tests marked increase the PGO task time significantly and likely don't help improve optimization of the final executable. (cherry picked from commit 52a48e6)
Co-authored-by: Neil Schemenauer nas-github@arctrix.com
lisroach pushed a commit to lisroach/cpython that referenced this pull request
Mark some individual tests to skip when --pgo is used. The tests marked increase the PGO task time significantly and likely don't help improve optimization of the final executable.
DinoV pushed a commit to DinoV/cpython that referenced this pull request
Mark some individual tests to skip when --pgo is used. The tests marked increase the PGO task time significantly and likely don't help improve optimization of the final executable.
websurfer5 pushed a commit to websurfer5/cpython that referenced this pull request
Mark some individual tests to skip when --pgo is used. The tests marked increase the PGO task time significantly and likely don't help improve optimization of the final executable.