runtests - Run set of tests - MATLAB (original) (raw)

Syntax

Description

[testResults](#mw%5Fccbac464-285a-4a54-a87f-d7eb3d3e811d) = runtests runs all the tests in your current folder and returns the test results.

example

[testResults](#mw%5Fccbac464-285a-4a54-a87f-d7eb3d3e811d) = runtests([tests](#btzwrop-tests)) runs the specified tests.

example

[testResults](#mw%5Fccbac464-285a-4a54-a87f-d7eb3d3e811d) = runtests(___,[Name=Value](#namevaluepairarguments)) specifies options using one or more name-value arguments in addition to any of the input argument combinations in previous syntaxes. For example, testResults = runtests(IncludeSubfolders=true) runs all the tests in the current folder and any of its subfolders.

example

[[testResults](#mw%5Fccbac464-285a-4a54-a87f-d7eb3d3e811d),[coverageResults](#mw%5F7afac245-79ef-42a5-8e37-d974bd882a5b)] = runtests(___) also returns the results of the code coverage analysis when you specify theReportCoverageFor name-value argument. (since R2023b)

example

Examples

collapse all

Create a folder myExample in your current working folder, and change into that folder.

In the myExample folder, create a test script, typeTest.m.

%% Test double class exp = 'double'; act = ones; assert(isa(act,exp))

%% Test single class exp = 'single'; act = ones('single'); assert(isa(act,exp))

%% Test uint16 class exp = 'uint16'; act = ones('uint16'); assert(isa(act,exp))

In the myExample folder, create a test script, sizeValueTest.m.

%% Test size exp = [7 13]; act = ones([7 13]); assert(isequal(size(act),exp))

%% Test values act = ones(42); assert(unique(act) == 1)

Run all tests in the current folder.

Running sizeValueTest .. Done sizeValueTest


Running typeTest ... Done typeTest


ans =

1x5 TestResult array with properties:

Name
Passed
Failed
Incomplete
Duration
Details

Totals: 5 Passed, 0 Failed, 0 Incomplete. 0.038077 seconds testing time.

MATLAB® ran 5 tests. There are 2 passing tests from sizeValueTest and 3 passing tests from typeTest.

Create the test file shown below, and save it as runtestsExampleTest.m on your MATLAB path.

function tests = runtestsExampleTest tests = functiontests(localfunctions);

function testFunctionOne(testCase)

Run the tests.

results = runtests('runtestsExampleTest.m');

Running runtestsExampleTest . Done runtestsExampleTest


If it does not exist, create the runtestsExampleTest.m test file from the previous example.

Create a subfolder, tmpTest, and, in that folder, create this runtestsExampleSubfolderTest.m test file.

function tests = runtestsExampleSubfolderTest tests = functiontests(localfunctions);

function testFunctionTwo(testCase)

Run the tests from the folder above tmpTest by specifying IncludeSubfolders astrue. The runtests function runs the tests in both the current folder and the subfolder.

results = runtests(pwd,IncludeSubfolders=true);

Running runtestsExampleTest . Done runtestsExampleTest


Running runtestsExampleSubFolderTest . Done runtestsExampleSubFolderTest


If you do not specify the IncludeSubfolders name-value argument, runtests does not run the test in the subfolder.

Running runtestsExampleTest . Done runtestsExampleTest


When your current folder is a project root folder or when you pass the path to a project root folder to the runtests function, runtests runs all test files contained in the specified project that are labeled with the Test classification.

This example assumes that a project folder atC:\projects\project1 contains test files that are labeled with the Test classification. Change your current folder to the project root folder and run the tests in the project.

cd 'C:\projects\project1' runtests

Alternatively, you can run the tests by openingproject1. Close the project when you are finished.

proj = openProject('C:\projects\project1'); runtests close(proj)

As another alternative, run the tests in the project by passing the full path to the project root folder to runtests.

runtests('C:\projects\project1')

Create this function-based test in a file namedrunInParallelTest.m in your current folder.

function tests = runInParallelTest tests = functiontests(localfunctions);

function testA(testCase) verifyEqual(testCase,5,5);

function testB(testCase) verifyTrue(testCase,logical(1));

function testC(testCase) verifySubstring(testCase,'SomeLongText','Long');

function testD(testCase) verifySize(testCase,ones(2,5,3),[2 5 3]);

function testE(testCase) verifyGreaterThan(testCase,3,2);

function testF(testCase) verifyEmpty(testCase,{},'Cell array is not empty.');

function testG(testCase) verifyMatches(testCase,'Some Text','Some [Tt]ext');

Run the tests in parallel. Running tests in parallel requires Parallel Computing Toolbox™. The testing framework might vary the order and number of groups or which tests it includes in each group.

results = runtests("runInParallelTest.m",UseParallel=true);

Split tests into 7 groups and running them on 4 workers.

Finished Group 2

Running runInParallelTest . Done runInParallelTest



Finished Group 3

Running runInParallelTest . Done runInParallelTest



Finished Group 1

Running runInParallelTest . Done runInParallelTest



Finished Group 4

Running runInParallelTest . Done runInParallelTest



Finished Group 6

Running runInParallelTest . Done runInParallelTest



Finished Group 5

Running runInParallelTest . Done runInParallelTest



Finished Group 7

Running runInParallelTest . Done runInParallelTest


In your working folder, create testZeros.m. This class contains four test methods.

classdef testZeros < matlab.unittest.TestCase properties (TestParameter) type = {'single','double','uint16'} outSize = struct('s2d',[3 3],'s3d',[2 5 4]) end

methods (Test)
    function testClass(testCase,type,outSize)
        testCase.verifyClass(zeros(outSize,type),type)
    end
    
    function testSize(testCase,outSize)
        testCase.verifySize(zeros(outSize),outSize)
    end
    
    function testDefaultClass(testCase)
        testCase.verifyClass(zeros,'double')
    end
    
    function testDefaultSize(testCase)
        testCase.verifySize(zeros,[1 1])
    end
    
    function testDefaultValue(testCase)
        testCase.verifyEqual(zeros,0)
    end
end

end

The full test suite has 11 test elements: 6 from the testClass method, 2 from the testSize method, and 1 each from the testDefaultClass, testDefaultSize, and testDefaultValue methods.

At the command prompt, run all the parameterizations for the testSize method.

runtests('testZeros/testSize')

Running testZeros .. Done testZeros


ans =

1×2 TestResult array with properties:

Name
Passed
Failed
Incomplete
Duration
Details

Totals: 2 Passed, 0 Failed, 0 Incomplete. 0.023971 seconds testing time.

The runtests function executed the two parameterized tests from the testSize method. Alternatively, you can specify the test procedure name with runtests('testZeros','ProcedureName','testSize').

Run the test elements that use the outSize parameter property.

runtests('testZeros','ParameterProperty','outSize')

Running testZeros ........ Done testZeros


ans =

1×8 TestResult array with properties:

Name
Passed
Failed
Incomplete
Duration
Details

Totals: 8 Passed, 0 Failed, 0 Incomplete. 0.036078 seconds testing time.

The runtests function executed eight tests that use the outSize parameter property: six from the testClass method and two from the testSize method.

Run the test elements that use the single parameter name.

runtests('testZeros','ParameterName','single')

Running testZeros .. Done testZeros


ans =

1×2 TestResult array with properties:

Name
Passed
Failed
Incomplete
Duration
Details

Totals: 2 Passed, 0 Failed, 0 Incomplete. 0.007455 seconds testing time.

The runtests function executed the two tests from the testClass method that use the outSize parameter name.

Since R2023b

Run your tests and collect code coverage results by using the runtests function.

In a file named quadraticSolver.m in your current folder, create the quadraticSolver function. The function takes as inputs the coefficients of a quadratic polynomial and returns the roots of that polynomial. If the coefficients are specified as nonnumeric values, the function throws an error.

function r = quadraticSolver(a,b,c) % quadraticSolver returns solutions to the % quadratic equation ax^2 + bx + c = 0.

if ~isa(a,"numeric") || ~isa(b,"numeric") || ~isa(c,"numeric") error("quadraticSolver:InputMustBeNumeric", ... "Coefficients must be numeric.") end

r(1) = (-b + sqrt(b^2 - 4ac)) / (2a); r(2) = (-b - sqrt(b^2 - 4ac)) / (2a);

end

To test the quadraticSolver function, create the SolverTest class in a file named SolverTest.m in your current folder. Define three Test methods that test the function against real solutions, imaginary solutions, and nonnumeric inputs.

classdef SolverTest < matlab.unittest.TestCase methods (Test) function realSolution(testCase) actSolution = quadraticSolver(1,-3,2); expSolution = [2 1]; testCase.verifyEqual(actSolution,expSolution) end function imaginarySolution(testCase) actSolution = quadraticSolver(1,2,10); expSolution = [-1+3i -1-3i]; testCase.verifyEqual(actSolution,expSolution) end function nonnumericInput(testCase) testCase.verifyError(@()quadraticSolver(1,"-3",2), ... "quadraticSolver:InputMustBeNumeric") end end end

Run the tests in the SolverTest class and also perform a code coverage analysis by specifying the ReportCoverageFor name-value argument. To programmatically access the coverage results in addition to generating a code coverage report, invoke the runtests function with two output arguments. After the tests run, the first output contains the test results and the second output contains the coverage results.

[testResults,coverageResults] = runtests("SolverTest", ... "ReportCoverageFor","quadraticSolver.m")

Running SolverTest ... Done SolverTest


MATLAB code coverage report has been saved to: C:\Users\hrastega\AppData\Local\Temp\tp50794fb5_477b_45bc_9837_59491d3d2e4b\index.html

testResults =

1×3 TestResult array with properties:

Name
Passed
Failed
Incomplete
Duration
Details

Totals: 3 Passed, 0 Failed, 0 Incomplete. 0.021243 seconds testing time.

coverageResults =

Result with properties:

    Filename: "C:\work\quadraticSolver.m"
CreationDate: 30-Oct-2023 11:01:29

Coverage summary (use generateHTMLReport to generate an HTML report): Function: 1/1 (100%) Statement: 4/4 (100%)

Use coverageSummary to retrieve information from the coverage results.

Input Arguments

collapse all

Tests, specified as a string array, character vector, or cell array of character vectors. Use this argument to specify your test content. For example, you can specify a test file, a test class, a folder that contains test files, a namespace that contains test classes, or a project folder that contains test files.

Example: runtests("myTestFile.m")

Example: runtests(["myTestFile/test1" "myTestFile/test3"])

Example: runtests("myNamespace.MyTestClass")

Example: runtests(pwd)

Example: runtests({'myNamespace.MyTestClass','myTestFile.m',pwd,'myNamespace.innerNamespace'})

Example: runtests("C:\projects\project1")

Name-Value Arguments

expand all

Specify optional pairs of arguments asName1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: results = runtests(tests,Name="productA_*") runsTest elements with a name that starts with"productA_".

Test Identification

expand all

Option to run tests in subfolders, specified as a numeric or logical0 (false) or1 (true). By default, the framework runs tests in the specified folders, but not in their subfolders.

Option to run tests in inner namespaces, specified as a numeric or logical 0 (false) or1 (true). By default, the framework runs tests in the specified namespaces, but not in their inner namespaces.

Option to include tests from referenced projects, specified as a numeric or logical0 (false) or 1 (true). For more information on referenced projects, see Componentize Large Projects.

Action to take against an invalid test file in a folder or namespace that the function is processing, specified as one of these values:

An invalid test file is a test file that the framework cannot run. Examples include a test file that contains syntax errors, a function-based test file that is missing local functions, and a file with a Test method that is passed an undefined parameterization property.

Test Filtering

expand all

Name of the base folder that contains the test file, specified as a string array, character vector, or cell array of character vectors. This argument filters the test suite. For the testing framework to include a test in the filtered suite, the Test element must be contained in one of the base folders specified byBaseFolder. If none of theTest elements match a base folder, an empty test suite is returned. Use the wildcard character (*) to match any number of characters. Use the question mark character (?) to match a single character.

For test files defined in namespaces, the base folder is the parent of the top-level namespace folder.

Names of the files and folders that contain source code, specified as a string vector, character vector, or cell vector of character vectors. This argument filters the test suite by including only the tests that depend on the specified source code. If none of the tests depend on the source code, an empty test suite is returned.

The specified value must represent at least one existing file. If you specify a folder, the framework extracts the paths to the files within the folder.

You must have a MATLAB® Test™ license to use DependsOn. For more information about selecting tests by source code dependency, see matlabtest.selectors.DependsOn (MATLAB Test).

Example: DependsOn=["myFile.m" "myFolder"]

Example: DependsOn=["folderA" "C:\work\folderB"]

Name of the test, specified as a string array, character vector, or cell array of character vectors. This argument filters the test suite. For the testing framework to include a test in the filtered suite, the Name property of theTest element must match one of the names specified byName. If none of the Test elements have a matching name, an empty test suite is returned. Use the wildcard character (*) to match any number of characters. Use the question mark character (?) to match a single character.

For a given test file, the name of a test uniquely identifies the smallest runnable portion of the test content. The test name includes the namespace name, filename (excluding the extension), procedure name, and information about parameterization.

Name of a test class property that defines a parameter used by the test, specified as a string array, character vector, or cell array of character vectors. This argument filters the test suite. For the testing framework to include a test in the filtered suite, theParameterization property of the Test element must contain at least one of the property names specified byParameterProperty. If none of the Test elements have a matching property name, an empty test suite is returned. Use the wildcard character (*) to match any number of characters. Use the question mark character (?) to match a single character.

Name of a parameter used by the test, specified as a string array, character vector, or cell array of character vectors. MATLAB generates parameter names based on the test class property that defines the parameters. For example:

The ParameterName argument filters the test suite. For the testing framework to include a test in the filtered suite, theParameterization property of theTest element must contain at least one of the parameter names specified by ParameterName. If none of the Test elements have a matching parameter name, an empty test suite is returned. Use the wildcard character (*) to match any number of characters. Use the question mark character (?) to match a single character.

Name of the class that the test class derives from, specified as a string array, character vector, or cell array of character vectors. This argument filters the test suite. For the testing framework to include a test in the filtered suite, theTestClass property of the Test element must point to a test class that derives from one of the classes specified bySuperclass. If none of the Test elements match a class, an empty test suite is returned.

Name of a tag used by the test, specified as a string array, character vector, or cell array of character vectors. This argument filters the test suite. For the testing framework to include a test in the filtered suite, the Tags property of the Test element must contain at least one of the tag names specified by Tag. If none of the Test elements have a matching tag name, an empty test suite is returned. Use the wildcard character (*) to match any number of characters. Use the question mark character (?) to match a single character.

Test Run Customization

expand all

Option to apply strict checks when running tests, specified as a numeric or logical 0 (false) or1 (true). For example, the framework generates a qualification failure if a test issues a warning.

Option to run tests in parallel, specified as a numeric or logical0 (false) or1 (true).

By default, runtests runs tests in serial. If you specify UseParallel as true, then runtests divides the test suite into groups and runs the groups in parallel if:

Otherwise, runtests runs tests in serial regardless of the value ofUseParallel.

Testing in parallel might not be compatible with other options. For example, testing occurs in serial if UseParallel and Debug are both specified astrue. When running test in parallel, the testing framework might vary the order and number of groups or which tests it includes in each group.

Option to apply debugging capabilities when running tests, specified as a numeric or logical 0 (false) or 1 (true). For example, if a test failure occurs, the framework pauses test execution to enter debug mode.

Display level of event details, specified as an integer scalar from0 through 4, a matlab.automation.Verbosity enumeration object, or a text representation of the enumeration.

Numeric Representation Enumeration Member Name Verbosity Description
0 None No information
1 Terse Minimal information
2 Concise Moderate amount of information
3 Detailed Some supplemental information
4 Verbose Lots of supplemental information

The runtests function displays failing and logged events with the amount of detail specified byOutputDetail. By default,runtests displays failing and logged events at the matlab.automation.Verbosity.Detailed level (level 3) and test run progress at thematlab.automation.Verbosity.Concise level (level 2).

Verbosity level of logged diagnostics included for the test run, specified as an integer scalar from 0 through4, a matlab.automation.Verbosity enumeration object, or a text representation of the enumeration. Theruntests function includes diagnostics that are logged at this level and below.

Numeric Representation Enumeration Member Name Verbosity Description
0 None No information
1 Terse Minimal information
2 Concise Moderate amount of information
3 Detailed Some supplemental information
4 Verbose Lots of supplemental information

By default, runtests includes diagnostics logged at the matlab.automation.Verbosity.Terse level (level 1). To exclude logged diagnostics, specifyLoggingLevel asmatlab.automation.Verbosity.None (level 0).

Logged diagnostics are diagnostics that you supply to the testing framework with a call to the log (TestCase) or log (Fixture) method.

Artifact Generation

expand all

Names of source files and folders for code coverage analysis, specified as a string array, character vector, or cell array of character vectors. If you specify relative paths, the paths must be in the current folder. Otherwise, you must specify full paths. You can specify the paths to folders or to files that have a.m, .mlx, or.mlapp extension.

When you specify this argument, the function runs your tests and generates an HTML code coverage report for the specified source code. The report shows the parts of the source code that were executed by the tests. If you specify the coverageResults output argument (since R2023b) in addition to ReportCoverageFor, then the function provides programmatic access to coverage results in addition to generating the HTML code coverage report.

Example: ReportCoverageFor=pwd

Example: ReportCoverageFor=["myFile.m" "myFolder"]

Example: ReportCoverageFor=["folderA" "C:\work\folderB"]

Option to create or update baseline data in a MAT file being used in a test with specific qualification methods, specified as a numeric or logical 0 (false) or1 (true). You must have aMATLAB Test or Simulink® Test license to useGenerateBaselines.

If you specify GenerateBaselines astrue, you must use at least one of these qualification methods in your tests.

MATLAB Test Simulink Test
verifyEqualsBaseline verifySignalsMatch
assumeEqualsBaseline assumeSignalsMatch
assertEqualsBaseline assertSignalsMatch
fatalAssertEqualsBaseline fatalAssertSignalsMatch

For an example, see Using MATLAB-Based Simulink Tests in the Test Manager (Simulink Test).

Output Arguments

collapse all

Test results, returned as a row vector of matlab.unittest.TestResult objects. Each element of the vector provides information about one of the tests that ran.

Since R2023b

Coverage results, returned as a column vector of matlab.coverage.Result objects. Each element of the vector provides information about one of the files in your source code that was covered by the tests.

Use this argument with the ReportCoverageFor name-value argument to programmatically access the results of code coverage analysis for your source code.

Tips

Extended Capabilities

Version History

Introduced in R2013b

expand all

When you select function-based or class-based tests using theDependsOn name-value argument (requires MATLAB Test), the function more accurately selects tests that depend on the specified source code. If the function can determine which individual tests in the test file depend on the source code, then it selects only the dependent tests and excludes the rest. Otherwise, the function includes all the tests in the test file.

In previous releases, the function includes all the tests in a test file if the file depends on the specified source code, without attempting to exclude tests that are not dependent on the source code.

The IncludeSubpackages name-value argument is now named IncludeInnerNamespaces. The behavior remains the same, and existing instances of IncludeSubpackages in your code continue to work as expected. There are no plans to remove support for existing references to IncludeSubpackages.

If you have a MATLAB Test license, you can specify any type of source file using theDependsOn name-value argument. In previous releases, you can specify files only with a .m, .p,.mlx, .mlapp, .mat, or.slx extension.

To programmatically access the results of code coverage analysis for your source code, specify a second output argument when you use theReportCoverageFor name-value argument. For an example, seeRetrieve Code Coverage Information.

You can filter a test suite by test file dependency on specified source code. Use theDependsOn name-value argument (requires MATLAB Test) to specify the source files and folders.

If you have Requirements Toolbox™ and MATLAB Test installed, you can use the runtests function to run tests that verify requirement sets. To run tests, specify one or more requirement set files as a string scalar or string vector. For example,results = runtests("myRequirementSet.slreqx") runs the tests that verify the specified requirement set.

To specify whether the testing framework issues a warning or throws an error when it encounters an invalid test file in a folder or namespace, use theInvalidFileFoundAction name-value argument.

When you assign a nonempty cell array to a parameterization property, the testing framework generates parameter names from the elements of the cell array by taking into account their values, types, and dimensions. In previous releases, if the property value is a cell array of character vectors, the framework generates parameter names from the values in the cell array. Otherwise, the framework specifies parameter names as value1, value2, …, valueN.

If your code uses parameter names to create or filter test suites, replace the old parameter names with the descriptive parameter names. For example, update suite = testsuite(pwd,"ParameterName","value1") by replacing value1 with a descriptive parameter name.

The IncludeSubfolders name-value argument treats folders and namespaces the same way. For example,runtests(pwd,IncludeSubfolders=true) runs all the tests in the current folder and any of its subfolders, including namespace folders. In previous releases, IncludeSubfolders ignores namespace folders.

The runtests function ignores any files in a MATLAB project that do not define test procedures. For example, if an abstract TestCase class definition file is labeled with theTest classification, the function ignores it. In previous releases, MATLAB produces an error if runtests is called on a project that uses the Test classification for any files other than concrete test files.

If MATLAB runs without the Java® Virtual Machine (JVM®) software, runtests cannot run the tests in a MATLAB project. The reason is that the project cannot be opened without the JVM software. In previous releases, when MATLAB runs without the JVM software, runtests creates a suite from the test files in the project and runs the suite.

You can run tests on a thread-based pool (requires Parallel Computing Toolbox) by starting a parallel pool of thread workers and then calling theruntests function with the UseParallel name-value argument.

Tests to run with runtests on a thread-based pool are subject to these restrictions:

You can run tests in parallel on clusters and clouds (requires MATLAB Parallel Server and Parallel Computing Toolbox). To run tests on a remote parallel pool, call theruntests function with the UseParallel name-value argument.

You can create standalone applications that support running tests in parallel (requires MATLAB Compiler™ and Parallel Computing Toolbox). Use the directive %#function parallel.Pool in your code so that MATLAB Compiler can locate and package all of the components required for running tests in parallel. For more information, see Compile MATLAB Unit Tests.

To create or update a MAT file being used in a test with specific qualification methods, use the GenerateBaselines name-value argument. You must have Simulink Test installed to use GenerateBaselines.

When your current folder is a project root folder or when you pass the path to a project root folder to the runtests function, the function runs all test files contained in the specified project that are labeled with theTest classification.

To run the tests from referenced projects, use theIncludeReferencedProjects name-value argument.

To specify the source code to include in the code coverage report, use theReportCoverageFor name-value argument.