Add summary for xfails with -rxX option by sturmf · Pull Request #11574 · pytest-dev/pytest (original) (raw)

@nicoddemus Regarding " traceback of xfail/xpass tests, which are not handled here, correct?"

• xfail - tracebacks are handled in summary_xfailures()
• xpass - there is no traceback, but the output is captured

The one thing I see missing is the assert output is not reported in the summary at the end, but the reason for xfail is
I propose reporting the assertion line, then adding a reason if there is one, like this:

XFAIL test name - assertion - reason
as in XFAIL test_xfail.py::test_xfail_reason - assert 1 == 2 - reason 1

In context with Fail and Xpass:

...
=============================== short test summary info ===============================
FAILED test_xfail.py::test_fail - assert 1 == 2
XFAIL test_xfail.py::test_xfail - assert 1 == 2
XFAIL test_xfail.py::test_xfail_reason - assert 1 == 2 - reason 1
XPASS test_xfail.py::test_xpass - reason 2
================== 1 failed, 1 passed, 2 xfailed, 1 xpassed in 0.05s ==================

@nicoddemus Should the reason be before the assert? Should a separator be more obvious, or is a dash fine?

@sturmf I have a modification for this, are you ok with me pushing to the branch?

The remaining work seems to be:

• add test cases and modify existing test cases as necessary
• refactor (I think this is optional, and could be done with a followup)