Dynamically Filtered Tests - MATLAB & Simulink (original) (raw)

Assumption failures produce filtered tests. In thematlab.unittest.TestResult class, such a test is markedIncomplete.

Since filtering test content through the use of assumptions does not produce test failures, it has the possibility of creating dead test code. Avoiding this requires monitoring of filtered tests.

Test Methods

If an assumption failure is encountered inside of aTestCase method with the Test attribute, the entire method is marked as filtered, but MATLAB® runs the subsequent Test methods.

The following class contains an assumption failure in one of the methods in theTest block.

classdef ExampleTest < matlab.unittest.TestCase methods(Test) function testA(testCase) testCase.verifyTrue(true) end function testB(testCase) testCase.assumeEqual(0,1) % remaining test code is not exercised end function testC(testCase) testCase.verifyFalse(true) end end end

Since the testB method contains an assumption failure, when you run the test, the testing framework filters that test and marks it as incomplete. After the assumption failure in testB, the testing framework proceeds and executes testC, which contains a verification failure.

ts = matlab.unittest.TestSuite.fromClass(?ExampleTest); res = ts.run;

Running ExampleTest .

ExampleTest/testB was filtered. Details

.

Verification failed in ExampleTest/testC.

---------------------
Framework Diagnostic:
---------------------
verifyFalse failed.
--> The value must evaluate to "false".

Actual logical:
       1

------------------
Stack Information:
------------------
In C:\work\ExampleTest.m (ExampleTest.testC) at 11

================================================================================ . Done ExampleTest

Failure Summary:

 Name               Failed  Incomplete  Reason(s)
================================================================
 ExampleTest/testB              X       Filtered by assumption.
----------------------------------------------------------------
 ExampleTest/testC    X                 Failed by verification.

If you examine the TestResult, you notice that there is a passed test, a failed test, and a test that did not complete due to an assumption failure.

res =

1×3 TestResult array with properties:

Name
Passed
Failed
Incomplete
Duration
Details

Totals: 1 Passed, 1 Failed, 1 Incomplete. 2.4807 seconds testing time.

The testing framework keeps track of incomplete tests so that you can monitor filtered tests for nonexercised test code. You can see information about these tests within the TestResult object.

ans =

TestResult with properties:

      Name: 'ExampleTest/testB'
    Passed: 0
    Failed: 0
Incomplete: 1
  Duration: 2.2578
   Details: [1×1 struct]

Totals: 0 Passed, 0 Failed, 1 Incomplete. 2.2578 seconds testing time.

To create a modified test suite from only the filtered tests, select incomplete tests from the original test suite.

tsFiltered = ts([res.Incomplete])

tsFiltered =

Test with properties: Name: 'ExampleTest/testB' ProcedureName: 'testB' TestClass: "ExampleTest" BaseFolder: 'C:\work' Parameterization: [0×0 matlab.unittest.parameters.EmptyParameter] SharedTestFixtures: [0×0 matlab.unittest.fixtures.EmptyFixture] Tags: {1×0 cell}

Tests Include: 0 Parameterizations, 0 Shared Test Fixture Classes, 0 Tags.

Method Setup and Teardown Code

If an assumption failure is encountered inside aTestCase method with the TestMethodSetup attribute, MATLAB filters the method which was to be run for that instance. If a test uses assumptions from within the TestMethodSetup block, consider instead using the assumptions in the TestClassSetup block, which likewise filters all Test methods in the class but is less verbose and more efficient.

One of the methods in the following TestMethodSetup block within ExampleTest.m contains an assumption failure.

methods(TestMethodSetup)
    function setupMethod1(testCase)
        testCase.assumeEqual(1,0)
        % remaining test code is not exercised
    end
    function setupMethod2(testCase)
        disp('* Running setupMethod2 *')
        testCase.assertEqual(1,1)
    end
end

Updated ExampleTest Class Definition

classdef ExampleTest < matlab.unittest.TestCase methods(TestMethodSetup) function setupMethod1(testCase) testCase.assumeEqual(1,0) % remaining test code is not exercised end function setupMethod2(testCase) disp('* Running setupMethod2 *') testCase.assertEqual(1,1) end end

methods(Test)
    function testA(testCase)
        testCase.verifyTrue(true)
    end
    function testB(testCase)
        testCase.assumeEqual(0,1)
        % remaining test code is not exercised
    end
    function testC(testCase)
        testCase.verifyFalse(true)
    end
end

end

When you run the test, you see that the framework completes executes all the methods in the TestMethodSetup block that do not contain the assumption failure, and it marks as incomplete all methods in theTest block.

ts = matlab.unittest.TestSuite.fromClass(?ExampleTest); res = ts.run;

Running ExampleTest

================================================================================ ExampleTest/testA was filtered. Details

Failure Summary:

 Name               Failed  Incomplete  Reason(s)
================================================================
 ExampleTest/testA              X       Filtered by assumption.
----------------------------------------------------------------
 ExampleTest/testB              X       Filtered by assumption.
----------------------------------------------------------------
 ExampleTest/testC              X       Filtered by assumption.

The Test methods did not change but all 3 are filtered due to an assumption failure in the TestMethodSetup block. The testing framework executes methods in the TestMethodSetup block without assumption failures, such as setupMethod2. As expected, the testing framework executes setupMethod2 3 times, once before eachTest method.

Class Setup and Teardown Code

If an assumption failure is encountered inside of aTestCase method with the TestClassSetup orTestClassTeardown attribute, MATLAB filters the entire TestCase class.

The methods in the following TestClassSetup block withinExampleTest.m contains an assumption failure.

methods(TestClassSetup)
    function setupClass(testCase)
        testCase.assumeEqual(1,0)
        % remaining test code is not exercised
    end
end

Updated ExampleTest Class Definition

classdef ExampleTest < matlab.unittest.TestCase methods(TestClassSetup) function setupClass(testCase) testCase.assumeEqual(1,0) % remaining test code is not exercised end end

methods(TestMethodSetup)
    function setupMethod1(testCase)
        testCase.assumeEqual(1,0)
        % remaining test code is not exercised
    end
    function setupMethod2(testCase)
        disp('* Running setupMethod2 *')
        testCase.assertEqual(1,1)
    end
end

methods(Test)
    function testA(testCase)
        testCase.verifyTrue(true)
    end
    function testB(testCase)
        testCase.assumeEqual(0,1)
        % remaining test code is not exercised
    end
    function testC(testCase)
        testCase.verifyFalse(true)
    end
end

end

When you run the test, you see that the framework does not execute any of the methods in the TestMethodSetup or Test.

ts = matlab.unittest.TestSuite.fromClass(?ExampleTest); res = ts.run;

Running ExampleTest

================================================================================ All tests in ExampleTest were filtered. Details

Done ExampleTest

Failure Summary:

 Name               Failed  Incomplete  Reason(s)
================================================================
 ExampleTest/testA              X       Filtered by assumption.
----------------------------------------------------------------
 ExampleTest/testB              X       Filtered by assumption.
----------------------------------------------------------------
 ExampleTest/testC              X       Filtered by assumption.

The Test and TestMethodSetup methods did not change but everything is filtered due to an assumption failure in theTestClassSetup block.

See Also

matlab.unittest.qualifications.Assumable | matlab.unittest.TestCase | matlab.unittest.TestResult