XFail and XPass output does not get printed to console · Issue #10618 · pytest-dev/pytest (original) (raw)

I don't see any output on the console when I run tests marked XFail or XPass. In fact, these tests only show up in the summaries, not in the PASSES nor the FAILURES section, where I would expect to see them. This would seem to be a fairly serious oversight, since testers certainly want to see the output of their tests that have been marked as such. Am I missing something?

Now, if I specify 'live logs' on the command line, I get my expected output, but only live; not captured. Given that a lot of pytest users probably don't know about this option, wouldn't it make sense to print the stdout/stderr/stdlog to console by default when processing XFail and XPass cases?

Thanks, Jeff

Example console outputs:

$ pytest -rAR -k test_single
========================================= test session starts =========================================
platform darwin -- Python 3.9.9, pytest-7.2.0, pluggy-1.0.0
rootdir: /Users/jwr003/coding/pytest-tui, configfile: pytest.ini
plugins: rerunfailures-10.3, metrics-0.4, assume-2.4.3, html-3.2.0, typhoon-xray-0.4.7,
json-report-1.5.0, Faker-13.15.0, adaptavist-5.8.0, session2file-0.1.11, allure-pytest-2.12.0,
jira-xray-0.8.2, reportportal-5.1.3, pylama-8.4.1, reportlog-0.1.2, metadata-2.0.2, tui-1.6.0

collected 78 items / 76 deselected / 2 selected

demo-tests/test_single_xpass_xfail.py xX                                                        [100%]

======================================= short test summary info =======================================
XFAIL demo-tests/test_single_xpass_xfail.py::test0_xfail
XPASS demo-tests/test_single_xpass_xfail.py::test0_xpass
============================ 76 deselected, 1 xfailed, 1 xpassed in 1.27s =============================

$ pytest --log-cli-level=DEBUG -rAR -k test_single
========================================= test session starts =========================================
platform darwin -- Python 3.9.9, pytest-7.2.0, pluggy-1.0.0
rootdir: /Users/jwr003/coding/pytest-tui, configfile: pytest.ini
plugins: rerunfailures-10.3, metrics-0.4, assume-2.4.3, html-3.2.0, typhoon-xray-0.4.7,
json-report-1.5.0, Faker-13.15.0, adaptavist-5.8.0, session2file-0.1.11, allure-pytest-2.12.0,
jira-xray-0.8.2, reportportal-5.1.3, pylama-8.4.1, reportlog-0.1.2, metadata-2.0.2, tui-1.6.0

collected 78 items / 76 deselected / 2 selected

demo-tests/test_single_xpass_xfail.py::test0_xfail
-------------------------------------------- live log call --------------------------------------------
CRITICAL root:test_single_xpass_xfail.py:10 CRITICAL
ERROR    root:test_single_xpass_xfail.py:11 ERROR
WARNING  root:test_single_xpass_xfail.py:12 WARNING
INFO     root:test_single_xpass_xfail.py:13 INFO
DEBUG    root:test_single_xpass_xfail.py:14 DEBUG
XFAIL                                                                                           [ 50%]
demo-tests/test_single_xpass_xfail.py::test0_xpass
-------------------------------------------- live log call --------------------------------------------
CRITICAL root:test_single_xpass_xfail.py:21 CRITICAL
ERROR    root:test_single_xpass_xfail.py:22 ERROR
WARNING  root:test_single_xpass_xfail.py:23 WARNING
INFO     root:test_single_xpass_xfail.py:24 INFO
DEBUG    root:test_single_xpass_xfail.py:25 DEBUG
XPASS                                                                                           [100%]

======================================= short test summary info =======================================
XFAIL demo-tests/test_single_xpass_xfail.py::test0_xfail
XPASS demo-tests/test_single_xpass_xfail.py::test0_xpass
============================ 76 deselected, 1 xfailed, 1 xpassed in 1.02s =============================

Test file:

import logging
import pytest

logger = logging.getLogger()


@pytest.mark.xfail()
def test0_xfail():
    print("Test 0 XFail")
    logger.critical("CRITICAL")
    logger.error("ERROR")
    logger.warning("WARNING")
    logger.info("INFO")
    logger.debug("DEBUG")
    assert False


@pytest.mark.xfail()
def test0_xpass():
    print("Test 0 XPass")
    logger.critical("CRITICAL")
    logger.error("ERROR")
    logger.warning("WARNING")
    logger.info("INFO")
    logger.debug("DEBUG")
    assert True