testrunner - Create test runner - MATLAB (original) (raw)
Create test runner
Since R2021a
Syntax
Description
runner = testrunner
creates a default test runner, which is equivalent to the runner that the testing framework configures by default when you call theruntests function.
The testrunner
function returns a matlab.unittest.TestRunner object. You can call the methods on the returned object to run and operate on your test suite and to customize running tests. For example, to run a suite of tests, use run(runner,suite)
.
runner = testrunner("minimal")
creates a minimal runner with no plugins installed. The returned test runner is the simplest runner possible and produces no text output. Use this syntax when you want to have complete control over which plugins to add to the runner.
runner = testrunner("textoutput")
creates a runner that is configured for text output. The output produced includes test progress as well as diagnostics in the event of test failures.
This syntax creates runners that tend to run tests more quickly, because the testing framework does not record diagnostics on test results produced by a nondefault runner. For more information, see Programmatically Access Test Diagnostics.
runner = testrunner(___,[Name=Value](#namevaluepairarguments))
specifies options using one or more name-value arguments in addition to any of the input argument combinations in previous syntaxes. For example, runner = testrunner("textoutput",OutputDetail="Verbose")
creates a test runner that displays test output detail at the maximum verbosity level. (since R2024b)
Examples
Run a suite of tests with a default runner and access the results.
Create a function-based test sampleTest.m
in your current folder. The file contains two tests that pass and one test that intentionally fails.
function tests = sampleTest tests = functiontests(localfunctions); end
function testA(testCase) % Test passes verifyEqual(testCase,2+3,5) end
function testB(testCase) % Test fails verifyGreaterThan(testCase,13,42) end
function testC(testCase) % Test passes verifySubstring(testCase,"Hello World!","llo") end
Create a test suite from the tests in sampleTest.m
. Then, create a default runner and run the tests.
suite = testsuite('SampleTest'); runner = testrunner; results = run(runner,suite);
Running sampleTest .
Verification failed in sampleTest/testB.
---------------------
Framework Diagnostic:
---------------------
verifyGreaterThan failed.
--> The value must be greater than the minimum value.
Actual Value:
13
Minimum Value (Exclusive):
42
------------------
Stack Information:
------------------
In C:\work\sampleTest.m (testB) at 10
.. Done sampleTest
Failure Summary:
Name Failed Incomplete Reason(s)
===============================================================
sampleTest/testB X Failed by verification.
Display the results from the second test.
ans =
TestResult with properties:
Name: 'sampleTest/testB'
Passed: 0
Failed: 1
Incomplete: 0
Duration: 0.3962
Details: [1×1 struct]
Totals: 0 Passed, 1 Failed, 0 Incomplete. 0.39619 seconds testing time.
When you run tests with a default runner, the testing framework uses a DiagnosticsRecordingPlugin
instance to record diagnostics on test results. Access the recorded diagnostics for the second test using the DiagnosticRecord
field in the Details
property on the TestResult
object.
records = results(2).Details.DiagnosticRecord
records =
QualificationDiagnosticRecord with properties:
Event: 'VerificationFailed'
EventScope: TestMethod
EventLocation: 'sampleTest/testB'
TestDiagnosticResults: [1×0 matlab.automation.diagnostics.DiagnosticResult]
FrameworkDiagnosticResults: [1×1 matlab.automation.diagnostics.DiagnosticResult]
AdditionalDiagnosticResults: [1×0 matlab.automation.diagnostics.DiagnosticResult]
Stack: [1×1 struct]
Report: 'Verification failed in sampleTest/testB.↵ ---------------------↵ Framework Diagnostic:↵ ---------------------↵ verifyGreaterThan failed.↵ --> The value must be greater than the minimum value.↵ ↵ Actual Value:↵ 13↵ Minimum Value (Exclusive):↵ 42↵ ------------------↵ Stack Information:↵ ------------------↵ In C:\work\sampleTest.m (testB) at 10'
Generate JUnit-style test results by creating a minimal runner and then adding an XMLPlugin
instance to the runner.
Create a function-based test sampleTest.m
in your current folder. The file contains two tests that pass and one test that intentionally fails.
function tests = sampleTest tests = functiontests(localfunctions); end
function testA(testCase) % Test passes verifyEqual(testCase,2+3,5) end
function testB(testCase) % Test fails verifyGreaterThan(testCase,13,42) end
function testC(testCase) % Test passes verifySubstring(testCase,'hello, world','llo') end
Create a test suite from the tests in sampleTest.m
.
suite = testsuite('sampleTest');
Create a test runner with no plugins. This code creates a silent runner that produces no output. You can now install whatever plugins you like.
runner = testrunner('minimal');
Create an XMLPlugin
instance that writes JUnit-style XML output to the file myTestResults.xml
.
import matlab.unittest.plugins.XMLPlugin xmlFile = 'myTestResults.xml'; p = XMLPlugin.producingJUnitFormat(xmlFile);
Add the plugin to the test runner and run the tests.
addPlugin(runner,p) results = run(runner,suite);
Display the results from the second test.
ans = TestResult with properties:
Name: 'sampleTest/testB'
Passed: 0
Failed: 1
Incomplete: 0
Duration: 0.0723
Details: [1×1 struct]
Totals: 0 Passed, 1 Failed (rerun), 0 Incomplete. 0.0723 seconds testing time.
Check for diagnostics recorded on test results. If you were using a default runner, there would be a DiagnosticRecord
field at this location. But because you are using a nondefault runner, the framework does not create such a field.
records = results(2).Details
records = struct with no fields.
Now, view the contents of the generated artifact.
Verification failed in sampleTest/testB. --------------------- Framework Diagnostic: --------------------- verifyGreaterThan failed. --> The value must be greater than the minimum value.Actual Value:
13
Minimum Value (Exclusive):
42
------------------
Stack Information:
------------------
In C:\TEMP\Examples\matlab-ex97531283\sampleTest.m (testB) at 10</failure>
</testcase>
<testcase classname="sampleTest" name="testC" time="0.0029273"/>
Run a suite of tests with a runner that is configured for text output, and then access the results.
Create a function-based test sampleTest.m
in your current folder. The file contains two tests that pass and one test that intentionally fails.
function tests = sampleTest tests = functiontests(localfunctions); end
function testA(testCase) % Test passes verifyEqual(testCase,2+3,5) end
function testB(testCase) % Test fails verifyGreaterThan(testCase,13,42) end
function testC(testCase) % Test passes verifySubstring(testCase,'hello, world','llo') end
Create a test suite from the tests in sampleTest.m
.
suite = testsuite('sampleTest');
Create a runner that produces text output and use it to run the tests.
runner = testrunner('textoutput'); results = run(runner,suite);
Running sampleTest .
Verification failed in sampleTest/testB.
---------------------
Framework Diagnostic:
---------------------
verifyGreaterThan failed.
--> The value must be greater than the minimum value.
Actual Value:
13
Minimum Value (Exclusive):
42
------------------
Stack Information:
------------------
In C:\TEMP\Examples\matlab-ex48684143\sampleTest.m (testB) at 10
.. Done sampleTest
Failure Summary:
Name Failed Incomplete Reason(s)
===============================================================
sampleTest/testB X Failed by verification.
Display the results from the second test.
ans = TestResult with properties:
Name: 'sampleTest/testB'
Passed: 0
Failed: 1
Incomplete: 0
Duration: 1.9894
Details: [1×1 struct]
Totals: 0 Passed, 1 Failed (rerun), 0 Incomplete. 1.9894 seconds testing time.
Check for diagnostics recorded on test results. If you were using a default runner, there would be a DiagnosticRecord
field at this location. But because you are using a nondefault runner, the framework does not create such a field.
records = results(2).Details
records = struct with no fields.
Name-Value Arguments
Specify optional pairs of arguments asName1=Value1,...,NameN=ValueN
, where Name
is the argument name and Value
is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.
Example: runner = testrunner("textoutput",OutputDetail="Verbose",LoggingLevel="Detailed")
Display level of test output, specified as an integer scalar from0
through 4
, a matlab.automation.Verbosity enumeration object, or a text representation of the enumeration.
Numeric Representation | Enumeration Member Name | Verbosity Description |
---|---|---|
0 | None | No information |
1 | Terse | Minimal information |
2 | Concise | Moderate amount of information |
3 | Detailed | Some supplemental information |
4 | Verbose | Lots of supplemental information |
By default, the test runner displays failing and logged events at thematlab.automation.Verbosity.Detailed
level (level 3) and test run progress at the matlab.automation.Verbosity.Concise
level (level 2).
Example: runner = testrunner("textoutput",OutputDetail="Verbose")
creates a test runner that displays test output at the matlab.automation.Verbosity.Verbose
level.
Verbosity level of logged diagnostics, specified as an integer scalar from0
through 4
, a matlab.automation.Verbosity enumeration object, or a text representation of the enumeration. The test runner includes diagnostics logged at the specified level and below.
Numeric Representation | Enumeration Member Name | Verbosity Description |
---|---|---|
0 | None | No information |
1 | Terse | Minimal information |
2 | Concise | Moderate amount of information |
3 | Detailed | Some supplemental information |
4 | Verbose | Lots of supplemental information |
By default, the test runner includes diagnostics logged at thematlab.automation.Verbosity.Terse
level (level 1). To exclude logged diagnostics, specify LoggingLevel
asmatlab.automation.Verbosity.None
(level 0).
Logged diagnostics are diagnostics that you supply to the unit testing framework with the log (TestCase) and log (Fixture) methods.
Example: runner = testrunner("textoutput",LoggingLevel="Detailed")
creates a test runner that includes diagnostics logged at thematlab.automation.Verbosity.Detailed
level and below.
Tips
- You can use the
testrunner
function instead of the static methods of the matlab.unittest.TestRunner class. This table shows thetestrunner
function call equivalent to each static method of thematlab.unittest.TestRunner
class.testrunner Function matlab.unittest.TestRunner Class r = testrunner r = matlab.unittest.TestRunner.withDefaultPlugins r = testrunner("minimal") r = matlab.unittest.TestRunner.withNoPlugins r = testrunner("textoutput") r = matlab.unittest.TestRunner.withTextOutput
Version History
Introduced in R2021a
To control the verbosity of the test runner, use the OutputDetail
and LoggingLevel
name-value arguments.