Publish C-level coverage of the CPython test suite · Issue #94759 · python/cpython (original) (raw)
Feature or enhancement
Automatically publish C-level coverage of the CPython test suite.
Pitch
Recent issues, such as the problems with frame.setlineno, have revealed that the C-level coverage of the test suite isn't perfect. While source-level coverage isn't the only metric of a good test suite, paying closer attention to it should help us have more confidence when making changes to the CPython core.
Proposed design for MVP
- Publish results nightly: This cadence should be enough for most cases. While various tools like codecov.io can run on every pull request, coverage takes a long time to run, and IME, there is enough noise in the results that developers (especially new to the project) would spend a lot of time "chasing" coverage.
- Use clang/llvm-cov rather than gcc/gcov: As @erlend-aasland pointed out the LLVM toolchain coverage provides branch counts and coverage of subexpressions which seems pretty useful. There is a risk that this isn't the same tooling as most Linux developers would use, and not what
make coverage-reportdoes. - Publish to Github Pages: The easiest thing seems to be to publish to Github Pages in a new repository in the
pythonorg for this purpose. This keeps all of the credential management within Github.
Out of scope
Creating diffs between coverage runs seems like it would be useful, but I haven't found a good tool for that yet.
Python-level coverage reports would likewise be useful, but should be a follow-on to this.
Previous discussion
@brettcannon had set up coverage reports in the past, but they have been broken for a couple of releases. This aims to be simpler than that approach by using a pure Github stack (not using codecov.io) and would just run nightly, not as part of every pull request.
More recent discussion began here: faster-cpython/ideas#426