Python Coverage Comment - GitHub Marketplace (original) (raw)

Coverage badge

Presentation

This action analyses the coverage data produced by the Pythoncoverage library and produces:

All of this runs on top of GitHub Action without extra-charges and runs on the GitHub infrastructure: your code isn't sent anywhere out of GitHub.

See this action in action

What does it do?

This action operates on an already generated .coverage file fromcoverage.

It has two main modes of operation:

PR mode

On PRs, it will analyze the .coverage file, and produce a comment that will be posted to the PR. If a comment had already previously be written, it will be updated. The comment contains information on the evolution of coverage rate attributed to this PR, as well as the rate of coverage for lines that this PR introduces. There's also a small analysis for each file in a collapsed block.

This comment will also be output as a job summary.

See an example.

Default branch mode

On repository's default branch, it will extract the coverage rate and create files that will be stored on a dedicated independent branch in your repository.

These files include:

See an example

Usage

Setup

Please ensure that your .coverage file(s) is created with the optionrelative_files = true.

Please ensure that the branch python-coverage-comment-action-data is not protected (there's no reason that it would be the case, except if you have very specific wildcard rules). If it is, either adjust your rules, or set theCOVERAGE_DATA_BRANCH parameter as described below. GitHub Actions will create this branch with initial data at the first run if it doesn't exist, and will independently commit to that branch after each commit to your default branch.

Badge

Once the action has run on your default branch, all the details for how to integrate the badge to your Readme will be displayed in:

Basic usage

The following snippet is targeted for cases where you expect PRs from users that don't have write access to the repository. Posting the comment is done in 2 steps:

  1. Checkout the repository and generate the comment to be posted. For security reasons, we don't want to give permissions to a workflow that checks out untrusted code
  2. From a trusted workflow, publish the comment on the PR

.github/workflows/ci.yml

name: CI

on: pull_request: push: branches: - "main"

jobs: test: name: Run tests & display coverage runs-on: ubuntu-latest permissions: # Gives the action the necessary permissions for publishing new # comments in pull requests. pull-requests: write # Gives the action the necessary permissions for pushing data to the # python-coverage-comment-action branch, and for editing existing # comments (to avoid publishing multiple comments in the same PR) contents: write steps: - uses: actions/checkout@v4

  - name: Install everything, run the tests, produce the .coverage file
    run: make test # This is the part where you put your own test command

  - name: Coverage comment
    id: coverage_comment
    uses: py-cov-action/python-coverage-comment-action@v3
    with:
      GITHUB_TOKEN: ${{ github.token }}

  - name: Store Pull Request comment to be posted
    uses: actions/upload-artifact@v4
    if: steps.coverage_comment.outputs.COMMENT_FILE_WRITTEN == 'true'
    with:
      # If you use a different name, update COMMENT_ARTIFACT_NAME accordingly
      name: python-coverage-comment-action
      # If you use a different name, update COMMENT_FILENAME accordingly
      path: python-coverage-comment-action.txt

.github/workflows/coverage.yml

name: Post coverage comment

on: workflow_run: workflows: ["CI"] types: - completed

jobs: test: name: Run tests & display coverage runs-on: ubuntu-latest if: github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion == 'success' permissions: # Gives the action the necessary permissions for publishing new # comments in pull requests. pull-requests: write # Gives the action the necessary permissions for editing existing # comments (to avoid publishing multiple comments in the same PR) contents: write # Gives the action the necessary permissions for looking up the # workflow that launched this workflow, and download the related # artifact that contains the comment to be published actions: read steps: # DO NOT run actions/checkout here, for security reasons # For details, refer to https://securitylab.github.com/research/github-actions-preventing-pwn-requests/ - name: Post comment uses: py-cov-action/python-coverage-comment-action@v3 with: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} GITHUB_PR_RUN_ID: ${{ github.event.workflow_run.id }} # Update those if you changed the default values: # COMMENT_ARTIFACT_NAME: python-coverage-comment-action # COMMENT_FILENAME: python-coverage-comment-action.txt

Basic usage without external contributors

If you don't expect external contributors, you don't need all the shenanigans with the artifacts and the 2nd workflow. This is likely to be the most straightforward way to configure it for private repositories. It might look like this:

.github/workflows/ci.yml

name: CI

on: pull_request: push: branches: - "main"

jobs: test: name: Run tests & display coverage runs-on: ubuntu-latest permissions: # Gives the action the necessary permissions for publishing new # comments in pull requests. pull-requests: write # Gives the action the necessary permissions for pushing data to the # python-coverage-comment-action branch, and for editing existing # comments (to avoid publishing multiple comments in the same PR) contents: write steps: - uses: actions/checkout@v4

  - name: Install everything, run the tests, produce the .coverage file
    run: make test # This is the part where you put your own test command

  - name: Coverage comment
    uses: py-cov-action/python-coverage-comment-action@v3
    with:
      GITHUB_TOKEN: ${{ github.token }}

Merging multiple coverage reports

In case you have a job matrix and you want the report to be on the global coverage, you can configure your ci.yml like this (coverage.yml remains the same)

name: CI

on: pull_request: push: branches: - "master" tags: - "*"

jobs: build: strategy: matrix: include: - python_version: "3.7" - python_version: "3.8" - python_version: "3.9" - python_version: "3.10"

name: "Python ${{ matrix.python_version }}"
runs-on: ubuntu-latest

steps:
  - uses: actions/checkout@v4

  - name: Set up Python
    id: setup-python
    uses: actions/setup-python@v4
    with:
      python-version: ${{ matrix.python_version }}

  - name: Install everything, run the tests, produce a .coverage.xxx file
    run: make test # This is the part where you put your own test command
    env:
      COVERAGE_FILE: ".coverage.${{ matrix.python_version }}"
      # The file name prefix must be ".coverage." for "coverage combine"
      # enabled by "MERGE_COVERAGE_FILES: true" to work. A "subprocess"
      # error with the message "No data to combine" will be triggered if
      # this prefix is not used.

  - name: Store coverage file
    uses: actions/upload-artifact@v4
    with:
      name: coverage-${{ matrix.python_version }}
      path: .coverage.${{ matrix.python_version }}
      # By default hidden files/folders (i.e. starting with .) are ignored.
      # You may prefer (for security reasons) not setting this and instead
      # set COVERAGE_FILE above to not start with a `.`, but you cannot
      # use "MERGE_COVERAGE_FILES: true" later on and need to manually
      # combine the coverage file using "pipx run coverage combine"
      include-hidden-files: true

coverage: name: Coverage runs-on: ubuntu-latest needs: build permissions: pull-requests: write contents: write steps: - uses: actions/checkout@v4

  - uses: actions/download-artifact@v4
    id: download
    with:
      pattern: coverage-*
      merge-multiple: true

  - name: Coverage comment
    id: coverage_comment
    uses: py-cov-action/python-coverage-comment-action@v3
    with:
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      MERGE_COVERAGE_FILES: true

  - name: Store Pull Request comment to be posted
    uses: actions/upload-artifact@v4
    if: steps.coverage_comment.outputs.COMMENT_FILE_WRITTEN == 'true'
    with:
      name: python-coverage-comment-action
      path: python-coverage-comment-action.txt

All options

Commenting on the PR on the push event

This action's PR comments with coverage reports is designed to work when running on the pull_request events. That being said, if your CI is running on feature branches on the push events and not on the pull_request events, we partly support a mode where the action can comment on the PR when running on the push events instead. This is most likely only useful for setups not accepting external PRs and you will not have the best user experience. If that's something you need to do, please have a look at this issue.

Overriding the template

By default, comments are generated from aJinja template that you can readhere.

If you want to change this template, you can set COMMENT_TEMPLATE. This is an advanced usage, so you're likely to run into more road bumps.

You will need to follow some rules for your template to be valid:

Examples

In the first example, we change the emoji that illustrates coverage going down from⬇️ to 😭:

{% extends "base" %} {% block emoji_coverage_down %}😭{% endblock emoji_coverage_down %}

In this second example, we replace the whole comment by something much shorter with the coverage (percentage) of the whole project from the PR build:

"Coverage: {{ coverage.info.percent_covered | pct }}{{ marker }}"

Monorepo setting

In case you want to use the action multiple times with different parts of your source (so you have multiple codebases into a single repo), you'll need to use SUBPROJECT_ID with a different value for each launch. You may still use the same step for storing all files as artifacts. You'll end up with a different comment for each launch. Feel free to use the COMMENT_TEMPLATE if you want each comment to clearly state what it relates to.

.github/workflows/ci.yml

name: CI

on: pull_request: push: branches: - "main"

jobs: test: name: Run tests & display coverage runs-on: ubuntu-latest permissions: pull-requests: write contents: write steps: - uses: actions/checkout@v3

  - name: Test project 1
    run: make -C project_1 test

  - name: Test project 2
    run: make -C project_2 test

  - name: Coverage comment (project 1)
    id: coverage_comment_1
    uses: py-cov-action/python-coverage-comment-action@v3
    with:
      COVERAGE_PATH: project_1
      SUBPROJECT_ID: project-1
      GITHUB_TOKEN: ${{ github.token }}

  - name: Coverage comment (project 2)
    id: coverage_comment_2
    uses: py-cov-action/python-coverage-comment-action@v3
    with:
      COVERAGE_PATH: project_2/src
      SUBPROJECT_ID: project-2
      GITHUB_TOKEN: ${{ github.token }}

  - name: Store Pull Request comment to be posted
    uses: actions/upload-artifact@v4
    if: steps.coverage_comment_1.outputs.COMMENT_FILE_WRITTEN == 'true' || steps.coverage_comment_2.outputs.COMMENT_FILE_WRITTEN == 'true'
    with:
      name: python-coverage-comment-action
      # Note the star
      path: python-coverage-comment-action*.txt

.github/workflows/coverage.yml

name: Post coverage comment

on: workflow_run: workflows: ["CI"] types: - completed

jobs: test: name: Run tests & display coverage runs-on: ubuntu-latest if: github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion == 'success' permissions: pull-requests: write contents: write actions: read steps: - name: Post comment uses: py-cov-action/python-coverage-comment-action@v3 with: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} GITHUB_PR_RUN_ID: ${{ github.event.workflow_run.id }} SUBPROJECT_ID: project-1 COVERAGE_PATH: project_1

  - name: Post comment
    uses: py-cov-action/python-coverage-comment-action@v3
    with:
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      GITHUB_PR_RUN_ID: ${{ github.event.workflow_run.id }}
      SUBPROJECT_ID: project-2
      COVERAGE_PATH: project_2/src

Other topics

Pinning

On the examples above, the version was set to the tag v3. Pinning to a major version will give you the latest release on this version. (Note that we release every time after a PR is merged). Pinning to a specific version (v3.1 for example) would make the action more reproducible, though you'd have to update it regularly (e.g. using Dependabot). You can also pin a commit hash if you want to be 100% sure of what you run, given that tags are mutable. Finally, You can also decide to pin to main, if you're OK with the action maybe breaking when (if) we release a v4.

Note on the state of this action

This action is tested with 100% coverage. That said, coverage isn't all, and there may be a lot of remaining issues :)

We accept Pull Requests (for bug fixes and previously-discussed features), and bug reports. For feature requests, this might depend on how much time we have on our hands at the moment, and how well you manage to sell it but don't get your hopes too high.

Generic coverage

Initially, the first iteration of this action was using the more genericcoverage.xml (Cobertura) in order to be language independent. It was later discovered that this format is very badly specified, as are mostly all coverage formats. For this reason, we switched to the much more specialized .coveragefile that is only produced for Python projects (also, the action was rewritten from the ground up). Because this would likely completely break compatibility, a brand new action (this action) was created.

You can find the (unmaintained) language-generic versionhere.

Why do we need relative_files = true ?

Yes, I agree, this is annoying! The reason is that by default, coverage writes the full path to the file in the .coverage file, but the path is most likely different between the moment where your coverage is generated (in your workflow) and the moment where the report is computed (in the action, which runs inside a docker).

I swear I saw something about a wiki somewhere?

A previous version of this action did things with the wiki. This is not the case anymore.

.coverage file generated on a Windows file system

If your project's coverage was built on Windows, you may get an error like:

CoverageWarning: Couldn't parse 'yourproject\__init__.py': No source for code: 'yourproject\__init__.py'. (couldnt-parse)

This is likely due to coverage being confused with the coverage being computed with \ but read with /. You can most probably fix it with the following in your coverage configuration:

[paths]
source =
    */project/module
    *\project\module

Private repositories

This action is supposedly compatible with private repository. Just make sure to use the svg badge directly, and not the shields.io URL.

Github Enterprise (GHE) Support

This action should be compatible with GitHub Enterprise. Just make sure to set the GITHUB_BASE_URL input to your GHE URL.

Upgrading from v2 to v3

When upgrading, we change the location and format where the coverage data is kept. Pull request that have not been re-based may be displaying slightly wrong information.

New comment format starting with 3.19

Starting with 3.19, the format for the Pull Request changed to a table with badges. We've been iterating a lot on the new format. It's perfectly ok if you preferred the old format. In that case, see #335 for instructions on how to emulate the old format usingCOMMENT_TEMPLATE.