Working with non-python tests - pytest documentation (original) (raw)

Toggle table of contents sidebar

A basic example for specifying tests in Yaml files¶

Here is an example conftest.py (extracted from Ali Afshar’s special purpose pytest-yamlwsgi plugin). This conftest.py will collect test*.yaml files and will execute the yaml-formatted content as custom tests:

content of conftest.py

from future import annotations

import pytest

def pytest_collect_file(parent, file_path): if file_path.suffix == ".yaml" and file_path.name.startswith("test"): return YamlFile.from_parent(parent, path=file_path)

class YamlFile(pytest.File): def collect(self): # We need a yaml parser, e.g. PyYAML. import yaml

    raw = yaml.safe_load(self.path.open(encoding="utf-8"))
    for name, spec in sorted(raw.items()):
        yield YamlItem.from_parent(self, name=name, spec=spec)

class YamlItem(pytest.Item): def init(self, *, spec, **kwargs): super().init(**kwargs) self.spec = spec

def runtest(self):
    for name, value in sorted(self.spec.items()):
        # Some custom test execution (dumb example follows).
        if name != value:
            raise YamlException(self, name, value)

def repr_failure(self, excinfo):
    """Called when self.runtest() raises an exception."""
    if isinstance(excinfo.value, YamlException):
        return "\n".join(
            [
                "usecase execution failed",
                "   spec failed: {1!r}: {2!r}".format(*excinfo.value.args),
                "   no further details known at this point.",
            ]
        )
    return super().repr_failure(excinfo)

def reportinfo(self):
    return self.path, 0, f"usecase: {self.name}"

class YamlException(Exception): """Custom exception for error reporting."""

You can create a simple example file:

test_simple.yaml

ok: sub1: sub1

hello: world: world some: other

and if you installed PyYAML or a compatible YAML-parser you can now execute the test specification:

nonpython $ pytest test_simple.yaml =========================== test session starts ============================ platform linux -- Python 3.x.y, pytest-8.x.y, pluggy-1.x.y rootdir: /home/sweet/project/nonpython collected 2 items

test_simple.yaml F. [100%]

================================= FAILURES ================================= ______________________________ usecase: hello ______________________________ usecase execution failed spec failed: 'some': 'other' no further details known at this point. ========================= short test summary info ========================== FAILED test_simple.yaml::hello ======================= 1 failed, 1 passed in 0.12s ========================

You get one dot for the passing sub1: sub1 check and one failure. Obviously in the above conftest.py you’ll want to implement a more interesting interpretation of the yaml-values. You can easily write your own domain specific testing language this way.

Note

repr_failure(excinfo) is called for representing test failures. If you create custom collection nodes you can return an error representation string of your choice. It will be reported as a (red) string.

reportinfo() is used for representing the test location and is also consulted when reporting in verbose mode:

nonpython $ pytest -v =========================== test session starts ============================ platform linux -- Python 3.x.y, pytest-8.x.y, pluggy-1.x.y -- $PYTHON_PREFIX/bin/python cachedir: .pytest_cache rootdir: /home/sweet/project/nonpython collecting ... collected 2 items

test_simple.yaml::hello FAILED [ 50%] test_simple.yaml::ok PASSED [100%]

While developing your custom test collection and execution it’s also interesting to just look at the collection tree:

nonpython $ pytest --collect-only =========================== test session starts ============================ platform linux -- Python 3.x.y, pytest-8.x.y, pluggy-1.x.y rootdir: /home/sweet/project/nonpython collected 2 items

======================== 2 tests collected in 0.12s ========================