testrunner - Create test runner - MATLAB (original) (raw)

Create test runner

Since R2021a

Syntax

Description

runner = testrunner creates a default test runner, which is equivalent to the runner that the testing framework configures by default when you call theruntests function.

The testrunner function returns a matlab.unittest.TestRunner object. You can call the methods on the returned object to run and operate on your test suite and to customize running tests. For example, to run a suite of tests, use run(runner,suite).

example

runner = testrunner("minimal") creates a minimal runner with no plugins installed. The returned test runner is the simplest runner possible and produces no text output. Use this syntax when you want to have complete control over which plugins to add to the runner.

example

runner = testrunner("textoutput") creates a runner that is configured for text output. The output produced includes test progress as well as diagnostics in the event of test failures.

This syntax creates runners that tend to run tests more quickly, because the testing framework does not record diagnostics on test results produced by a nondefault runner. For more information, see Programmatically Access Test Diagnostics.

example

runner = testrunner(___,[Name=Value](#namevaluepairarguments)) specifies options using one or more name-value arguments in addition to any of the input argument combinations in previous syntaxes. For example, runner = testrunner("textoutput",OutputDetail="Verbose") creates a test runner that displays test output detail at the maximum verbosity level. (since R2024b)

Examples

collapse all

Run a suite of tests with a default runner and access the results.

Create a function-based test sampleTest.m in your current folder. The file contains two tests that pass and one test that intentionally fails.

function tests = sampleTest tests = functiontests(localfunctions); end

function testA(testCase) % Test passes verifyEqual(testCase,2+3,5) end

function testB(testCase) % Test fails verifyGreaterThan(testCase,13,42) end

function testC(testCase) % Test passes verifySubstring(testCase,"Hello World!","llo") end

Create a test suite from the tests in sampleTest.m. Then, create a default runner and run the tests.

suite = testsuite('SampleTest'); runner = testrunner; results = run(runner,suite);

Running sampleTest .

Verification failed in sampleTest/testB. --------------------- Framework Diagnostic: --------------------- verifyGreaterThan failed. --> The value must be greater than the minimum value.
Actual Value: 13 Minimum Value (Exclusive): 42 ------------------ Stack Information: ------------------ In C:\work\sampleTest.m (testB) at 10

.. Done sampleTest

Failure Summary:

 Name              Failed  Incomplete  Reason(s)
===============================================================
 sampleTest/testB    X                 Failed by verification.

Display the results from the second test.

ans =

TestResult with properties:

      Name: 'sampleTest/testB'
    Passed: 0
    Failed: 1
Incomplete: 0
  Duration: 0.3962
   Details: [1×1 struct]

Totals: 0 Passed, 1 Failed, 0 Incomplete. 0.39619 seconds testing time.

When you run tests with a default runner, the testing framework uses a DiagnosticsRecordingPlugin instance to record diagnostics on test results. Access the recorded diagnostics for the second test using the DiagnosticRecord field in the Details property on the TestResult object.

records = results(2).Details.DiagnosticRecord

records =

QualificationDiagnosticRecord with properties:

                      Event: 'VerificationFailed'
                 EventScope: TestMethod
              EventLocation: 'sampleTest/testB'
      TestDiagnosticResults: [1×0 matlab.automation.diagnostics.DiagnosticResult]
 FrameworkDiagnosticResults: [1×1 matlab.automation.diagnostics.DiagnosticResult]
AdditionalDiagnosticResults: [1×0 matlab.automation.diagnostics.DiagnosticResult]
                      Stack: [1×1 struct]
                     Report: 'Verification failed in sampleTest/testB.↵    ---------------------↵    Framework Diagnostic:↵    ---------------------↵    verifyGreaterThan failed.↵    --> The value must be greater than the minimum value.↵    ↵    Actual Value:↵        13↵    Minimum Value (Exclusive):↵        42↵    ------------------↵    Stack Information:↵    ------------------↵    In C:\work\sampleTest.m (testB) at 10'

Generate JUnit-style test results by creating a minimal runner and then adding an XMLPlugin instance to the runner.

Create a function-based test sampleTest.m in your current folder. The file contains two tests that pass and one test that intentionally fails.

function tests = sampleTest tests = functiontests(localfunctions); end

function testA(testCase) % Test passes verifyEqual(testCase,2+3,5) end

function testB(testCase) % Test fails verifyGreaterThan(testCase,13,42) end

function testC(testCase) % Test passes verifySubstring(testCase,'hello, world','llo') end

Create a test suite from the tests in sampleTest.m.

suite = testsuite('sampleTest');

Create a test runner with no plugins. This code creates a silent runner that produces no output. You can now install whatever plugins you like.

runner = testrunner('minimal');

Create an XMLPlugin instance that writes JUnit-style XML output to the file myTestResults.xml.

import matlab.unittest.plugins.XMLPlugin xmlFile = 'myTestResults.xml'; p = XMLPlugin.producingJUnitFormat(xmlFile);

Add the plugin to the test runner and run the tests.

addPlugin(runner,p) results = run(runner,suite);

Display the results from the second test.

ans = TestResult with properties:

      Name: 'sampleTest/testB'
    Passed: 0
    Failed: 1
Incomplete: 0
  Duration: 0.0723
   Details: [1×1 struct]

Totals: 0 Passed, 1 Failed (rerun), 0 Incomplete. 0.0723 seconds testing time.

Check for diagnostics recorded on test results. If you were using a default runner, there would be a DiagnosticRecord field at this location. But because you are using a nondefault runner, the framework does not create such a field.

records = results(2).Details

records = struct with no fields.

Now, view the contents of the generated artifact.

Actual Value:
    13
Minimum Value (Exclusive):
    42
------------------
Stack Information:
------------------
In C:\TEMP\Examples\matlab-ex97531283\sampleTest.m (testB) at 10</failure>
</testcase>
<testcase classname="sampleTest" name="testC" time="0.0029273"/>

Run a suite of tests with a runner that is configured for text output, and then access the results.

Create a function-based test sampleTest.m in your current folder. The file contains two tests that pass and one test that intentionally fails.

function tests = sampleTest tests = functiontests(localfunctions); end

function testA(testCase) % Test passes verifyEqual(testCase,2+3,5) end

function testB(testCase) % Test fails verifyGreaterThan(testCase,13,42) end

function testC(testCase) % Test passes verifySubstring(testCase,'hello, world','llo') end

Create a test suite from the tests in sampleTest.m.

suite = testsuite('sampleTest');

Create a runner that produces text output and use it to run the tests.

runner = testrunner('textoutput'); results = run(runner,suite);

Running sampleTest .

Verification failed in sampleTest/testB. --------------------- Framework Diagnostic: --------------------- verifyGreaterThan failed. --> The value must be greater than the minimum value.
Actual Value: 13 Minimum Value (Exclusive): 42 ------------------ Stack Information: ------------------ In C:\TEMP\Examples\matlab-ex48684143\sampleTest.m (testB) at 10

.. Done sampleTest

Failure Summary:

 Name              Failed  Incomplete  Reason(s)
===============================================================
 sampleTest/testB    X                 Failed by verification.

Display the results from the second test.

ans = TestResult with properties:

      Name: 'sampleTest/testB'
    Passed: 0
    Failed: 1
Incomplete: 0
  Duration: 1.9894
   Details: [1×1 struct]

Totals: 0 Passed, 1 Failed (rerun), 0 Incomplete. 1.9894 seconds testing time.

Check for diagnostics recorded on test results. If you were using a default runner, there would be a DiagnosticRecord field at this location. But because you are using a nondefault runner, the framework does not create such a field.

records = results(2).Details

records = struct with no fields.

Name-Value Arguments

collapse all

Specify optional pairs of arguments asName1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: runner = testrunner("textoutput",OutputDetail="Verbose",LoggingLevel="Detailed")

Display level of test output, specified as an integer scalar from0 through 4, a matlab.automation.Verbosity enumeration object, or a text representation of the enumeration.

By default, the test runner displays failing and logged events at thematlab.automation.Verbosity.Detailed level (level 3) and test run progress at the matlab.automation.Verbosity.Concise level (level 2).

Example: runner = testrunner("textoutput",OutputDetail="Verbose") creates a test runner that displays test output at the matlab.automation.Verbosity.Verbose level.

Verbosity level of logged diagnostics, specified as an integer scalar from0 through 4, a matlab.automation.Verbosity enumeration object, or a text representation of the enumeration. The test runner includes diagnostics logged at the specified level and below.

By default, the test runner includes diagnostics logged at thematlab.automation.Verbosity.Terse level (level 1). To exclude logged diagnostics, specify LoggingLevel asmatlab.automation.Verbosity.None (level 0).

Logged diagnostics are diagnostics that you supply to the unit testing framework with the log (TestCase) and log (Fixture) methods.

Example: runner = testrunner("textoutput",LoggingLevel="Detailed") creates a test runner that includes diagnostics logged at thematlab.automation.Verbosity.Detailed level and below.

Tips

• You can use the testrunner function instead of the static methods of the matlab.unittest.TestRunner class. This table shows the testrunner function call equivalent to each static method of the matlab.unittest.TestRunner class.

  testrunner Function	matlab.unittest.TestRunner Class
  r = testrunner	r = matlab.unittest.TestRunner.withDefaultPlugins
  r = testrunner("minimal")	r = matlab.unittest.TestRunner.withNoPlugins
  r = testrunner("textoutput")	r = matlab.unittest.TestRunner.withTextOutput

Version History

Introduced in R2021a

expand all

To control the verbosity of the test runner, use the OutputDetail and LoggingLevel name-value arguments.