moving test selection to pytest markers by mattsb42-aws · Pull Request #41 · aws/aws-encryption-sdk-python (original) (raw)

This change moves our test selection logic from using environment variables to filter out tests run in a large batch to instead using pytest markers to select specific tests. In addition to being simpler to maintain, this both allows each test environment to run fewer tests and makes it much clearer in our CI which types of tests fail (ex: if credentials are not found, only integration, acceptance, and examples tests will fail).

This adds six markers:

1. local: superset of unit and functional (does not require network access)
2. unit: mark test as a unit test (does not require network access)
3. functional: mark test as a functional test (does not require network access)
4. integ: mark a test as an integration test (requires network access)
5. accept: mark a test as an acceptance test (requires network access)
6. examples: mark a test as an examples test (requires network access)

Validating that all tests are caught, the counts for tests run and skipped when filtering for each marker:

local = unit + functional
861   = 598  + 263

all  = local + integ + accept + examples
2151 = 861   + 26    + 1260   + 4