Consider nonparametrized tests in reordering by sadra-barikbin · Pull Request #11236 · pytest-dev/pytest (original) (raw)

Thanks for the PR @sadra-barikbin.

I'm interested to know what is the motivation for this change.

I think (no first hand knowledge, maybe @RonnyPfannschmidt knows) that the restriction of reordering to only parametrized tests is intentional. Consider for example (using tmp_path_factory as just some higher-than-function-scope fixture):

def test_1(tmp_path_factory): pass

def test_2(): pass

def test_3(tmp_path_factory): pass

Without this PR, the ordering is 1, 2, 3. With this PR, the ordering is 1, 3, 2. I think that most users would prefer the existing ordering, and would be confused by the new ordering, which would even put 3 before 2 when it's in an entirely different file. The new ordering goes too far if you catch my meaning.

IMO the guiding principles should be:

1. The source code order is the "basic" expected ordering...
2. But, we want to avoid repeated setups/teardowns of higher-than-function-scope fixtures. This mostly only occurs when parametrizing higher-than-function-scope fixtures, and not in non-parametrized functions.

So the way to achieve 2 without hurting 1 too much is to restrict the reordering to parametrized args.