evaluateObjectDetection - Evaluate object detection data set against ground truth - MATLAB (original) (raw)

Evaluate object detection data set against ground truth

Since R2023b

Syntax

Description

[metrics](#mw%5F0d5d1202-9009-4538-b22e-d1f21e8bbfbf) = evaluateObjectDetection([detectionResults](#mw%5Fc9f582df-d62c-4b91-b66b-b406197cc8c5),[groundTruthData](#mw%5Ff751181a-eb7d-4933-937a-7341c5979f86)) evaluates the quality of the object detection results detectionResults against the labeled ground truth groundTruthData and returns various metrics.

example

[metrics](#mw%5F0d5d1202-9009-4538-b22e-d1f21e8bbfbf) = evaluateObjectDetection([detectionResults](#mw%5Fc9f582df-d62c-4b91-b66b-b406197cc8c5),[groundTruthData](#mw%5Ff751181a-eb7d-4933-937a-7341c5979f86),[threshold](#mw%5F4f7b64a3-311d-4edd-8fca-5382288721dd)) specifies the overlap threshold for assigning an output bounding box to a ground truth bounding box.

[metrics](#mw%5F0d5d1202-9009-4538-b22e-d1f21e8bbfbf) = evaluateObjectDetection(___,[Name=Value](#namevaluepairarguments)) specifies one or more name-value arguments, in addition to any combination of input arguments from previous syntaxes, to configure the object detection results evaluation. For example, AdditionalMetrics="AOS" includes average orientation similarity metrics in the output.

Examples

collapse all

Load a table containing images and ground truth bounding box labels. The first column contains the images, and the remaining columns contain the labeled bounding boxes.

data = load("vehicleTrainingData.mat"); trainingData = data.vehicleTrainingData;

Set the value of the dataDir variable as the location where the vehicleTrainingData.mat file is located. Load the test data into a local vehicle data folder.

dataDir = fullfile(toolboxdir("vision"),"visiondata"); trainingData.imageFilename = fullfile(dataDir,trainingData.imageFilename);

Create an imageDatastore using the files from the table.

imds = imageDatastore(trainingData.imageFilename);

Create a boxLabelDatastore using the label columns from the table.

blds = boxLabelDatastore(trainingData(:,2:end));

Load Pretrained Object Detector

Load a pretrained YOLO v2 object detector trained to detect vehicles into the workspace.

vehicleDetector = load("yolov2VehicleDetector.mat"); detector = vehicleDetector.detector;

Evaluate and Plot Object Detection Metrics

Run the detector on the test images. Set the detection threshold to a low value to detect as many objects as possible. This helps you evaluate the detector precision across the full range of recall values.

results = detect(detector,imds,Threshold=0.01);

Use evaluateObjectDetection to compute metrics for evaluating the performance of an object detector.

metrics = evaluateObjectDetection(results,blds);

Return the precision and recall metrics for the vehicle class using the precisionRecall object function.

[recall,precision,scores] = precisionRecall(metrics); ap = averagePrecision(metrics);

Plot the precision-recall curve for the vehicle class, the only class in the data set. Compute the average precision (AP) using the averagePrecision object function.

figure plot(recall{1},precision{1}) grid on title("Average Precision = " + ap); xlabel("Recall"); ylabel("Precision");

Compute the summary of the object detection metrics for the data set using the summarize object function.

[summaryDataset,summaryClass] = summarize(metrics); summaryDataset

summaryDataset=1×3 table NumObjects mAPOverlapAvg mAP0.5 __________ _____________ _______

   336           0.99096       0.99096

Input Arguments

collapse all

Predicted object detection results, specified as a three-column table containing the bounding box, predicted label, and score for each detected object. The table describes the formats of the required elements.

The order of the elements does not matter. When the datastore returns a cell array with more than three elements, the evaluateObjectDetection function assumes that the first element with an _M_-by-4 or_M_-by-5 numeric array contains the bounding boxes, the first element with categorical data contains the label data, and the first element with_M_-by-1 numeric data contains the scores.

You can create the detection results table using the detect function associated with your object detector..

Labeled ground truth, specified as a datastore with two outputs or a table with two columns. This table describes the formats of the required elements.

The order of the elements does not matter. When the datastore returns a cell array with more than three elements, the evaluateObjectDetection function assumes that the first element with an _M_-by-4 or_M_-by-5 numeric array contains the bounding boxes, the first element with categorical data contains the label data, and the first element with_M_-by-1 numeric data contains the scores.

Overlap threshold for assigning a detection to a ground truth box, specified as a numeric scalar in the range [0, 1] or a numeric vector. The overlap ratio is the intersection over union (IoU) ratio of two bounding boxes, or the ratio of the bounding box overlap area to the combined area of the predicted boxes and ground truth (minus the overlap).

IoU = Area of Overlap / (Ground Truth Area + Predicted Box Area – Intersection Area)

When you specify a numeric vector, the evaluateObjectDetection function calculates metrics for each threshold.

Tip

To reduce the number of false negatives, you can lower the IoU threshold at the possible expense of increasing the number of false positives. To learn about the role of IoU in detection evaluation, see Overlap Threshold (Intersection Over Union).

Tip

To optimize model performance for your application, you can evaluate the object detection metrics across a range of overlap thresholds. To learn how to fine-tune object detector performance using multiple overlap thresholds, see Tips.

Name-Value Arguments

collapse all

Specify optional pairs of arguments asName1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: evaluateObjectDetection(__,AdditionalMetrics=["LAMR","AOS"]) additionally returns LAMR and AOS.

Evaluation progress display toggle, specified as a numeric or logical1 (true) or 0 (false). If you specify Verbose astrue, the function displays progress information in the Command Window. The displayed information includes a progress bar, elapsed time, estimated time remaining, and data set metrics.

Additional metrics, specified as "LAMR","AOS", or a string array. The function returns the specified metrics as additional columns in the ClassMetrics andImageMetrics tables of theobjectDetectionMetrics object, as shown in this table below.

To specify "AOS" as an additional metric, your input data must contain rotated bounding boxes.

Output Arguments

More About

collapse all

Use overlap (or IoU) threshold to determine whether a detection is a true positive or false positive.

A detection is a true positive (TP) when both of these conditions are met:

• The predicted class matches the class of a ground truth (indicated by the confidence score exceeding the confidence threshold).
• The predicted bounding box has an IoU greater than or equal to the specified overlap threshold.

A detection is a false positive (FP) when at least one of these conditions is met:

• The predicted class does not match the class of a ground truth.
• The predicted bounding box has an IoU less than the specified overlap threshold.

A missed detection is a false negative (FN) when one or more of these conditions is met:

• The confidence score of the detection is lower than the confidence score threshold, resulting in a missed detection.
• The detection has an IoU less than the specified threshold, resulting in the predicted bounding box does not sufficiently overlapping with the ground truth, leading to the detection being discarded.

Quantify the accuracy of an object detector on a particular dataset by evaluating the overlap between the predicted bounding boxes and the ground truth bounding boxes.

Tips

To evaluate model performance across different levels of localization accuracy and fine-tune an object detector, compute metrics at multiple overlap thresholds. Overlap thresholds vary typically from 0.5 to 0.95 during evaluation. For example, set the overlap (IoU) threshold value to 0.5 to accept moderate overlap accuracy, and set the value to 0.95 to ensure very high localization accuracy. There are many ways to use the overlap threshold to tune object detector performance.

• By evaluating common metrics, such as precision and recall, at different overlap thresholds, you can evaluate the tradeoff between detecting more objects (higher recall) and ensuring those detections are accurate (higher precision).
• By evaluating model performance at a lower IoU threshold, such as 0.5, and at higher thresholds, such as 0.95, you can determine how good the detector is at identifying objects (detection) at the expense of precise localization (bounding box accuracy).
• By computing metrics such as the mean average precision (mAP) over multiple overlap thresholds, you can produce a single performance figure that accounts for both detection and localization accuracy, enabling you to compare different detection models.

To learn how to compute metrics at a range of overlap thresholds to tune detector performance, see the "Precision and Recall for a Single Class" section of the Multiclass Object Detection Using YOLO v2 Deep Learning example.

Version History

Introduced in R2023b

expand all

The evaluateObjectDetection function enables you to simultaneously calculate the average precision, miss rate, and AOS (if applicable) to evaluate detector performance. Therefore, evaluateObjectDetection is recommended overevaluateDetectionPrecision, evaluateDetectionMissRate, and evaluateDetectionAOS, which will be removed in a future release. To update your code, ensure that your ground truth data is in the correct format. See the groundTruthData argument for more details.

evaluateObjectDetection improves onevaluateDetectionPrecision by enabling you to compute the object detection performance metrics for the entire data set over all classes and at specified overlap thresholds, use rotated bounding boxes, and calculate the confusion matrix. The corresponding objectDetectionMetrics object also enables you to evaluate object detection metrics across various object size ranges using the metricsByArea function.