Write Plugin to Add Data to Test Results - MATLAB & Simulink (original) (raw)

This example shows how to create a plugin that adds data toTestResult objects. The plugin appends the actual and expected values in an assertion to the Details property of theTestResult object. To extend theTestRunner, the plugin overrides select methods of thematlab.unittest.plugins.TestRunnerPlugin class.

Create Plugin Class

In a file in your current folder, create the custom plugin classDetailsRecordingPlugin, which inherits from theTestRunnerPlugin class. For the complete code forDetailsRecordingPlugin, see DetailsRecordingPlugin Class Definition Summary.

To store the actual and expected values in TestResult objects, define two constant properties, ActField andExpField, within a properties block. Set the value of ActField to the name of the field of theDetails structure that contains the actual value. Set the value of ExpField to the name of the field that contains the expected value.

properties (Constant, Access = private)
    ActField = 'ActualValue';
    ExpField = 'ExpectedValue';
end

Add Fields to Details Property

To add new fields to the Details property of allTestResult objects belonging to the test session, override the runSession method ofTestRunnerPlugin in a methods block with protected access. runSession adds two empty fields to the Details structure ofTestResult objects and invokes the superclass method to trigger the entire test run.

methods (Access = protected)
    function runSession(plugin,pluginData)
        resultDetails = pluginData.ResultDetails;
        resultDetails.append(plugin.ActField,{})
        resultDetails.append(plugin.ExpField,{})
        runSession@matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData);
    end
end

To add the fields, the implementation of runSession contains calls to the append method of thematlab.unittest.plugins.plugindata.ResultDetails class. Each call adds an empty field to the Details structure.

Extend Creation of Shared Test Fixtures and TestCase Instances

Add listeners for the AssertionPassed andAssertionFailed events by extending the methods used by the testing framework to create the test content. The test content includesTestCase instances for each Test element, class-level TestCase instances for theTestClassSetup and TestClassTeardown method blocks, and Fixture instances used when aTestCase class has theSharedTestFixtures attribute.

Invoke the corresponding superclass method when you override the creation methods. The listeners that you add to the returned Fixture or TestCase instances cause thereactToAssertion helper method to execute whenever an assertion is performed. To add assertion data to test results, pass the result modifier instance along with the assertion event listener data to the helper method.

Add these creation methods to a methods block withprotected access.

methods (Access = protected)
    function fixture = createSharedTestFixture(plugin, pluginData)
        fixture = createSharedTestFixture@...
            matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);
        resultDetails = pluginData.ResultDetails;
        fixture.addlistener('AssertionPassed',...
            @(~,evd)plugin.reactToAssertion(evd,resultDetails));
        fixture.addlistener('AssertionFailed',...
            @(~,evd)plugin.reactToAssertion(evd,resultDetails));
    end

    function testCase = createTestClassInstance(plugin,pluginData)
        testCase = createTestClassInstance@...
            matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData);
        resultDetails = pluginData.ResultDetails;
        testCase.addlistener('AssertionPassed',...
            @(~,evd)plugin.reactToAssertion(evd,resultDetails));
        testCase.addlistener('AssertionFailed',...
            @(~,evd)plugin.reactToAssertion(evd,resultDetails));
    end

    function testCase = createTestMethodInstance(plugin,pluginData)
        testCase = createTestMethodInstance@...
            matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData);
        resultDetails = pluginData.ResultDetails;
        testCase.addlistener('AssertionPassed',...
            @(~,evd)plugin.reactToAssertion(evd,resultDetails));
        testCase.addlistener('AssertionFailed',...
            @(~,evd)plugin.reactToAssertion(evd,resultDetails));
    end
end

Define Helper Method

In a methods block with private access, define the helper method reactToAssertion. This method uses the QualificationEventData instance to extract the actual and expected values in assertions based on the IsEqualTo constraint, converts the extracted values to cell arrays, and appends the cell arrays to the fields of the corresponding TestResult object.

methods (Access = private)
    function reactToAssertion(plugin,evd,resultDetails)
        if ~isa(evd.Constraint,'matlab.unittest.constraints.IsEqualTo')
            return
        end
        resultDetails.append(plugin.ActField,{evd.ActualValue})
        resultDetails.append(plugin.ExpField,{evd.Constraint.Expected})
    end
end

DetailsRecordingPlugin Class Definition Summary

This code provides the complete contents ofDetailsRecordingPlugin.

classdef DetailsRecordingPlugin < matlab.unittest.plugins.TestRunnerPlugin properties (Constant, Access = private) ActField = 'ActualValue'; ExpField = 'ExpectedValue'; end

methods (Access = protected)
    function runSession(plugin,pluginData)
        resultDetails = pluginData.ResultDetails;
        resultDetails.append(plugin.ActField,{})
        resultDetails.append(plugin.ExpField,{})
        runSession@matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData);
    end
    
    function fixture = createSharedTestFixture(plugin, pluginData)
        fixture = createSharedTestFixture@...
            matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);
        resultDetails = pluginData.ResultDetails;
        fixture.addlistener('AssertionPassed',...
            @(~,evd)plugin.reactToAssertion(evd,resultDetails));
        fixture.addlistener('AssertionFailed',...
            @(~,evd)plugin.reactToAssertion(evd,resultDetails));
    end
    
    function testCase = createTestClassInstance(plugin,pluginData)
        testCase = createTestClassInstance@...
            matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData);
        resultDetails = pluginData.ResultDetails;
        testCase.addlistener('AssertionPassed',...
            @(~,evd)plugin.reactToAssertion(evd,resultDetails));
        testCase.addlistener('AssertionFailed',...
            @(~,evd)plugin.reactToAssertion(evd,resultDetails));
    end
    
    function testCase = createTestMethodInstance(plugin,pluginData)
        testCase = createTestMethodInstance@...
            matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData);
        resultDetails = pluginData.ResultDetails;
        testCase.addlistener('AssertionPassed',...
            @(~,evd)plugin.reactToAssertion(evd,resultDetails));
        testCase.addlistener('AssertionFailed',...
            @(~,evd)plugin.reactToAssertion(evd,resultDetails));
    end
end

methods (Access = private)
    function reactToAssertion(plugin,evd,resultDetails)
        if ~isa(evd.Constraint,'matlab.unittest.constraints.IsEqualTo')
            return
        end
        resultDetails.append(plugin.ActField,{evd.ActualValue})
        resultDetails.append(plugin.ExpField,{evd.Constraint.Expected})
    end
end

end

Create Example Test Class

In your current folder, create a file named ExampleTest.m containing the following parameterized test class. The class results in a test suite with 25 elements, each corresponding to an experiment performed using a different seed for the random number generator. In each experiment, the testing framework creates a 1-by-100 vector of normally distributed random numbers and asserts that the magnitude of the difference between the actual and expected sample means is within 0.1.

classdef ExampleTest < matlab.unittest.TestCase properties SampleSize = 100; end

properties (TestParameter) 
    seed = num2cell(randi(10^6,1,25));
end
    
methods(Test)
    function testMean(testCase,seed)
        import matlab.unittest.constraints.IsEqualTo
        import matlab.unittest.constraints.AbsoluteTolerance
        rng(seed)
        testCase.assertThat(mean(randn(1,testCase.SampleSize)),...
            IsEqualTo(0,'Within',AbsoluteTolerance(0.1)));
    end
end

end

Add Plugin to TestRunner and Run Tests

At the command prompt, create a test suite from theExampleTest class.

import matlab.unittest.TestSuite import matlab.unittest.TestRunner

suite = TestSuite.fromClass(?ExampleTest);

Create a TestRunner instance with no plugins. This code creates a silent runner and gives you control over the installed plugins.

runner = TestRunner.withNoPlugins;

Add DetailsRecordingPlugin to the runner and run the tests.

runner.addPlugin(DetailsRecordingPlugin) result = runner.run(suite)

result =

1×25 TestResult array with properties:

Name
Passed
Failed
Incomplete
Duration
Details

Totals: 18 Passed, 7 Failed (rerun), 7 Incomplete. 0.12529 seconds testing time.

To retrieve more information about the behavior of random number generation, create a structure array from the Details structures of the test results.

details = [result.Details]

details =

1×25 struct array with fields:

ActualValue
ExpectedValue

Create an array containing the difference between the actual and expected values in each test and then display the error values in a bar graph. The seven bars with a length greater than 0.1 correspond to the failed tests.

errorInMean = cell2mat([details.ExpectedValue]) - cell2mat([details.ActualValue]);

bar(errorInMean) xlabel('Experiment') ylabel('Error')

Bar graph depicting error versus experiment