addMetrics - Compute additional classification performance metrics - MATLAB (original) (raw)

Compute additional classification performance metrics

Since R2022b

Syntax

Description

rocmetrics computes the false positive rates (FPR), true positive rates (TPR), and additional metrics specified by the AdditionalMetrics name-value argument. After creating a rocmetrics object, you can compute additional classification performance metrics by using theaddMetrics function.

[UpdatedROCObj](#mw%5Fa416fd71-1e4d-4a5f-b1fa-3cb7d6f616ea) = addMetrics([rocObj](#mw%5F2d878b4b-0ef2-41e4-a47a-d5a586a0393d%5Fsep%5Fshared-rocObj),[metrics](#mw%5F149a8d41-4d7f-4847-a5bf-1ea52de4d492)) computes additional classification performance metrics specified inmetrics using the classification model information stored in therocmetrics objectrocObj.

UpdatedROCObj contains all the information inrocObj plus additional performance metrics computed byaddMetrics. The function attaches the additional computed metrics (metrics) as new variables in the table of the Metrics property.

If you compute confidence intervals when you create rocObj, theaddMetrics function computes the confidence intervals for the additional metrics. The new variables in theMetrics property contain a three-column matrix in which the first column corresponds to the metric values, and the second and third columns correspond to the lower and upper bounds, respectively. Using confidence intervals requires Statistics and Machine Learning Toolbox™.

example

Examples

collapse all

Compute the performance metrics (FPR, TPR, and expected cost) for a multiclass classification problem when you create a rocmetrics object. Compute additional metrics, the positive predictive value (PPV) and the negative predictive value (NPV), and add them to the object.

Load a sample of true labels and the prediction scores for a classification problem. For this example, there are five classes: daisy, dandelion, roses, sunflowers, and tulips. The class names are stored in classNames. The scores are the softmax prediction scores generated using the predict function. scores is an N-by-K array where N is the number of observations and K is the number of classes. The column order of scores follows the class order stored in classNames.

load('flowersDataResponses.mat')

scores = flowersData.scores; trueLabels = flowersData.trueLabels;

classNames = flowersData.classNames;

Create a rocmetrics object by using the true labels and the classification scores. Specify the column order of scores using classNames. By default, rocmetrics computes the FPR and TPR. Specify AdditionalMetrics="ExpectedCost" to compute the expected cost as well.

rocObj = rocmetrics(trueLabels,scores,classNames, ... AdditionalMetrics="ExpectedCost");

The table in the Metrics property of rocObj contains performance metric values for each of the classes, vertically concatenated according to the class order. Find and display the top rows for the second class in the table.

idx = rocObj.Metrics.ClassName == classNames(2); head(rocObj.Metrics(idx,:))

ClassName    Threshold    FalsePositiveRate    TruePositiveRate    ExpectedCost
_________    _________    _________________    ________________    ____________

dandelion        1                0                      0           0.045287  
dandelion        1                0                0.23889           0.034469  
dandelion        1                0                0.26111           0.033462  
dandelion        1                0                0.27222           0.032959  
dandelion        1                0                0.28889           0.032204  
dandelion        1                0                0.29444           0.031953  
dandelion        1                0                    0.3           0.031701  
dandelion        1                0                0.31111           0.031198

The table in Metrics contains the variables for the class names, threshold, false positive rate, true positive rate, and expected cost (the additional metric).

After creating a rocmetrics object, you can compute additional metrics using the classification model information stored in the object. Compute the PPV and NPV by using the addMetrics function. To overwrite the input argument rocObj, assign the output of addMetrics to the input.

rocObj = addMetrics(rocObj,["PositivePredictiveValue","NegativePredictiveValue"]);

Display the Metrics property for the top rows.

head(rocObj.Metrics(idx,:))

ClassName    Threshold    FalsePositiveRate    TruePositiveRate    ExpectedCost    PositivePredictiveValue    NegativePredictiveValue
_________    _________    _________________    ________________    ____________    _______________________    _______________________

dandelion        1                0                      0           0.045287                NaN                       0.7551        
dandelion        1                0                0.23889           0.034469                  1                      0.80202        
dandelion        1                0                0.26111           0.033462                  1                      0.80669        
dandelion        1                0                0.27222           0.032959                  1                      0.80904        
dandelion        1                0                0.28889           0.032204                  1                      0.81259        
dandelion        1                0                0.29444           0.031953                  1                      0.81378        
dandelion        1                0                    0.3           0.031701                  1                      0.81498        
dandelion        1                0                0.31111           0.031198                  1                      0.81738

The table in Metrics now includes the PositivePredictiveValue and NegativePredictiveValue variables in the last two columns, in the order you specified. Note that the positive predictive value (PPV = TP/(TP+FP)) is NaN for the reject-all threshold (largest threshold), and the negative predictive value (NPV = TN/(TN+FN)) is NaN for the accept-all threshold (lowest threshold). TP, FP, TN, and FN represent the number of true positives, false positives, true negatives, and false negatives, respectively.

Input Arguments

collapse all

Object evaluating classification performance, specified as a rocmetrics object.

Additional model performance metrics to compute, specified as a character vector or string scalar of the built-in metric name, string array of names, function handle (@metricName), or cell array of names or function handles. Arocmetrics object always computes the false positive rates (FPR) and the true positive rates (TPR) to obtain a ROC curve. Therefore, you do not have to specify to compute FPR and TPR.

Built-in metrics — Specify one of the following built-in metric names by using a character vector or string scalar. You can specify more than one by using a string array.

Name	Description
"TruePositives" or"tp"	Number of true positives (TP)
"FalseNegatives" or"fn"	Number of false negatives (FN)
"FalsePositives" or"fp"	Number of false positives (FP)
"TrueNegatives" or"tn"	Number of true negatives (TN)
"SumOfTrueAndFalsePositives" or"tp+fp"	Sum of TP and FP
"RateOfPositivePredictions" or"rpp"	Rate of positive predictions (RPP),(TP+FP)/(TP+FN+FP+TN)
"RateOfNegativePredictions" or"rnp"	Rate of negative predictions (RNP),(TN+FN)/(TP+FN+FP+TN)
"Accuracy" or"accu"	Accuracy, (TP+TN)/(TP+FN+FP+TN)
"FalseNegativeRate","fnr", or"miss"	False negative rate (FNR), or miss rate,FN/(TP+FN)
"TrueNegativeRate","tnr", or"spec"	True negative rate (TNR), or specificity,TN/(TN+FP)
"PositivePredictiveValue","ppv", "prec", or"precision"	Positive predictive value (PPV), or precision,TP/(TP+FP)
"NegativePredictiveValue" or"npv"	Negative predictive value (NPV),TN/(TN+FN)
"ExpectedCost" or"ecost"	Expected cost,(TPcost(P\|P)+FNcost(N
"f1score"	F1 score, 2TP/(2TP+FP+FN)
You can obtain all of the previous metrics by specifying "all". You cannot specify "all" in conjunction with any other metric.
The software computes the scale vector using the prior class probabilities (Prior) and the number of classes in Labels, and then scales the performance metrics according to this scale vector. For details, see Performance Metrics.

Custom metric — Specify a custom metric by using a function handle. A custom function that returns a performance metric must have this form:
metric = customMetric(C,scale,cost)
- The output argument metric is a scalar value.
- A custom metric is a function of the confusion matrix (C), scale vector (scale), and cost matrix (cost). The software finds these input values for each one-versus-all binary problem. For details, see Performance Metrics.
  * C is a 2-by-2 confusion matrix consisting of [TP,FN;FP,TN].
  * scale is a2-by-1 scale vector.
  * cost is a2-by-2 misclassification cost matrix.

The software does not support cross-validation for a custom metric. Instead, you can specify to use bootstrap when you create arocmetrics object.

Note that the positive predictive value (PPV) isNaN for the reject-all threshold for which TP = FP = 0, and the negative predictive value (NPV) is NaN for the accept-all threshold for which TN = FN = 0. For more details, see Thresholds, Fixed Metric, and Fixed Metric Values.

Example: ["Accuracy","PositivePredictiveValue"]

Example: {"Accuracy",@m1,@m2} specifies the accuracy metric and the custom metrics m1 and m2 as additional metrics.addMetrics stores the custom metric values as variables namedCustomMetric1 and CustomMetric2 in theMetrics property.

Data Types: char | string | cell | function_handle

Output Arguments

collapse all

Object evaluating classification performance, returned as a rocmetrics object.

To overwrite the input argument rocObj, assign the output of addMetrics to rocObj:

rocObj = addMetrics(rocObj,metrics);

Version History

Introduced in R2022b

expand all

addmetrics has new metrics:

"f1score", which computes the F1 score.
"precision", which is the same as "ppv" and "prec".
"all", which computes all supported metrics. You cannot use "all" in combination with any other metric.