Get Started with Image Segmentation - MATLAB & Simulink (original) (raw)
Image segmentation is a process in image processing and computer vision that involves dividing an image into multiple segments or regions. The primary goal of image segmentation is to identify objects and boundaries in images. Image segmentation plays a key role in various applications, including scientific imaging, traffic control systems, machine vision, face and text recognition, and image editing and computer graphics. You can use image segmentation for various tasks, including:
- Basic image processing, such as differentiating objects from the background or from each other.
- First step in tasks that requires precise delineation and pixel-level localization of objects in an image, such as object analysis and quantification.
- Subsequent processing step that creates boundaries or masks at the pixel level after objects have been identified, such as in instance segmentation tasks.
Which image segmentation technique you choose often depends on your specific application and the characteristics of the images to be segmented.
This table lists the techniques for image segmentation available in Image Processing Toolbox™, Computer Vision Toolbox™, and Medical Imaging Toolbox™. Each entry provides basic background information, tips for getting started, and a representative visualization.
| Technique | Get Started | Visualization of Sample Output |
|---|---|---|
Segment Anything Model (SAM) – Segment images automatically and interactively. More AboutThe SAM is a state-of-the-art, zero-shot image segmentation model that uses deep learning neural networks to accurately segment objects within images without requiring training. |
Instantaneously segment objects or the entire image in theImage Segmenter app.Segment entire image or a complex collection of many objects at once, with a distinct mask for each object, usingimsegsam.Interactively segment objects using points, masks, or ROIs that you provide by configuring a pretrained SAM as asegmentAnythingModel object.To learn more, see Get Started with Segment Anything Model for Image Segmentation. |  |
| Classical image segmentation techniques – Apply semi-automated and automated image segmentation algorithms, as well as processing methods such as thresholding. | Use one of the Segmentation Techniques functions or interactive apps, such as the Image Segmenter app and Color Thresholder app, to:Segment images or objects that have a very structured or textured pattern.Process binary images.Preprocess images for further steps, such as object analysis or deep learning tasks.To learn more, see the Classical Image Segmentation Techniques section. |  |
| Semantic segmentation techniques (Computer Vision Toolbox) – Train a deep learning neural network on a custom data set to segment images with complex scenes or domain-specific objects. More AboutSemantic segmentation classifies each pixel in an image into a predefined category. It does not differentiate between separate objects of the same class, but instead aims to understand the image at a pixel level and assign a class label (such as car, tree, or building) to each pixel.Use deep-learning-based semantic segmentation for data sets of complex image scenes where the overall composition of the scene is important, but the identity of individual objects is not. |
Interactively segment and label objects as ground truth using the Segment Anything Model (SAM) in the Image Labeler (Computer Vision Toolbox) app. For an example, see Automatically Label Ground Truth Using Segment Anything Model (Computer Vision Toolbox).Use the labeled ground truth data to train a deep learning network for semantic segmentation, such as unet (Computer Vision Toolbox) or deeplabv3plus (Computer Vision Toolbox), to categorize every pixel in an image using class annotations.After training a network, segment a test image usingsemanticseg (Computer Vision Toolbox).To learn more, see Get Started with Semantic Segmentation Using Deep Learning (Computer Vision Toolbox). |  |
| Instance segmentation techniques (Computer Vision Toolbox) – Train a deep learning instance segmentation model, or use a pretrained model to segment instances of objects in images. More AboutInstance segmentation not only classifies each pixel into a class but also distinguishes between different instances of the same class within the image. It combines the tasks of object detection (identifying and bounding objects) and semantic segmentation (classifying pixels), providing a unique identifier for each object instance. |
Distinguish between individual objects of the same category in a test image using a pretrained Mask R-CNN or SOLOv2 network, configured by the maskrcnn (Computer Vision Toolbox) and solov2 (Computer Vision Toolbox) objects and their object functions, respectively. Train a pretrained instance segmentation network, such asmaskrcnn (Computer Vision Toolbox) or solov2 (Computer Vision Toolbox), on a custom data set with labeled ground truth images, and segment a test image.To learn more, see Get Started with Instance Segmentation Using Deep Learning (Computer Vision Toolbox). |  |
| Medical image segmentation (Medical Imaging Toolbox) – Perform medical image segmentation using deep learning, a labeling app, or image processing algorithms. | Interactively segment objects or regions in medical images by specifying visual prompts to the Medical Segment Anything Model (MedSAM) using the medicalSegmentAnythingModel (Medical Imaging Toolbox) object.Label ground truth data in a 2-D medical image or 3-D volume by using the Medical Image Labeler (Medical Imaging Toolbox) app. You can use the labeled data to train semantic segmentation deep learning networks.Segment radiology images using fully automated and interactive deep learning models in the Medical Open Network for AI (MONAI) Label platform of the Medical Image Labeler (Medical Imaging Toolbox) app.For more information, see Analysis and Applications (Medical Imaging Toolbox). |  |
Classical Image Segmentation Techniques
Use the classical segmentation techniques available in Image Processing Toolbox when GPU hardware resources are limited and computing speed is critical. To get started with classical techniques using the Image Segmenter app for increased interactivity, see Getting Started with Image Segmenter. Use this table to select a technique based on the characteristics of your image and application.
| Image Characteristics | Recommended Classical Techniques | Example |
|---|---|---|
| Clear and well-defined boundaries | Threshold methods such as multithresh, otsuthresh, adaptthresh, andgraythresh can easily separate objects from the background based on intensity levels. |  For an example that uses this image, see Find Threshold and Segment Bright Rice Grains from Dark Background. |
| Images with distinct color regions | To create a binary mask for objects that have distinct colors that contrast significantly with the background or other objects, or objects of uniform color, use the Color Thresholder app. |  For an example that uses the Color Thresholder app, see Segment Image and Create Mask Using Color Thresholder. |
| Blurry or low-contrast boundaries | Use graph-based methods, such as the grabcut function, when the image has blurry object boundaries and a non-uniform intensity or noise distribution, at the expense of higher computational load.Use the watershed function to separate touching objects in a noisy image at the possible expense of over-segmenting the image.Use the activecontour function when there is clear definition of foreground objects despite blurry edges. |  For an example that uses this image, see Segment Foreground from Background in Image Using Grabcut. |
| Overlapping or touching objects | The watershed transform watershed can help separate objects that are touching by identifying the_watershed lines_ between different objects based on the gradient. |  For an example that uses this image, see Compute Watershed Transform and Display Resulting Label Matrix. |
| Objects with similar intensity to the background | You can perform clustering using the imsegkmeans, imsegkmeans3, and imsegisodata functions to segment an image into clusters based on color or intensity and spatial proximity. These methods can separate objects from the background even when their intensities are similar, but subtly distinct. Use k-means and ISODATA clustering functions to effectively partition an image or volume where distinct colors or intensities represent different segments. |  For an example that uses this image, see Segment 2-D Hyperspectral Image Using ISODATA Clustering. |
| Objects with texture or internal variation | Region-based methods, using the watershed or grayconnected functions, can help segment objects with internal variation, as they group pixels or subregions into larger regions based on predefined criteria, such as texture or color. |  For an example that uses this image, see Segment Image Using Flood-Fill Technique. |
| Large-scale images | For large images containing a lot of detail, you can significantly reduce computational load by using the superpixels region-based method of the superpixels orsuperpixels3 function. |  For an example that uses this image, see Plot Land Classification with Color Features and Superpixels. |
Visualize Segmentation Results
Visualize segmentation results to verify the accuracy of the segmentation (how well the segmented regions match the objects of interest), interpret your segmentation results, and identify potential postprocessing steps such as mask refinement. This table describes the options for visualizing segmentation results in Image Processing Toolbox and Computer Vision Toolbox, and the corresponding functions and representative visualization.
| Visualization Task | Function | Visualization Example |
|---|---|---|
| Display a binary mask or image. | imshow |  |
| Overlay a binary mask on an image. | imoverlay |  |
| Display boundaries of segmented regions over an image. | visboundaries |  |
| Overlay a label matrix on an image, coloring each segmented region differently based on its label. | labeloverlay |  |
| Combine two images, or an image and a mask, into a single visual output. | imfuse |  |
| Combine two images into a single, cohesive, composite image. You can specify to blend only the region of the foreground image specified by the mask. | imblend |  |
| Display a mask or a stack of masks overlaid on an image, and optionally specify a unique color for each mask. | insertObjectMask (Computer Vision Toolbox) |  |
MATLAB® supports a wide range of colormaps and enables you to create custom visualization functions. You can tailor a visualization to the specific requirements of your application, such as highlighting particular features or ensuring that visualizations are accessible to viewers with color vision deficiencies.
Evaluate Segmentation Results
Evaluate segmentation results by using quantitative metrics to measure how well a segmentation technique performs, and ensure that it meets the specific requirements of your application. This table describes the options for evaluating segmentation results using the functions in Image Processing Toolbox and Computer Vision Toolbox.
| Goal | Function | Evaluation Approach |
|---|---|---|
| Evaluate the overlap between the segmentation and a ground truth mask. | jaccard | Compute the Jaccard similarity coefficient, a measure of the intersection over union for the segmented result and the ground truth. |
| Evaluate the overlap between the segmentation and a ground truth mask, assessing the accuracy of the model in capturing the shape and size of segmented objects. | dice | Compute the Dice similarity coefficient, which is twice the area of overlap divided by the total number of pixels in both the ground truth and the segmented image. |
| Evaluate how accurately the boundaries of the segmented regions match the actual boundaries. | bfscore | Compute the Boundary F1 Score (BF score) between the segmented boundary and the ground truth boundary. |
| Evaluate the overlap between predicted and ground truth segmentations in semantic segmentation tasks that have imbalanced data sets with varying class frequencies. | generalizedDice (Computer Vision Toolbox) | Compute the weighted average of the Dice similarity coefficient across different classes, accounting for class imbalance. |
| Evaluate semantic segmentation results against the ground truth, classifying each pixel into one of several categories. | evaluateSemanticSegmentation (Computer Vision Toolbox) | Compare semantic segmentation results against the ground truth data by computing standard metrics, including pixel accuracy, mean IoU, and weighted IoU. |
| Evaluate instance segmentation results against the ground truth. | evaluateInstanceSegmentation (Computer Vision Toolbox) | Compare instance segmentation results against the ground truth data by computing standard metrics, including the confusion matrix, average precision, and precision recall. |
See Also
Apps
- Image Segmenter | Color Thresholder | Image Labeler (Computer Vision Toolbox)
Functions
- imsegsam | semanticseg (Computer Vision Toolbox)
Topics
- Getting Started with Image Segmenter
- Get Started with Segment Anything Model for Image Segmentation
- Segment Objects Using Segment Anything Model (SAM) in Image Segmenter
- Automatically Label Ground Truth Using Segment Anything Model (Computer Vision Toolbox)
- Specify ROI as Binary Mask
- Get Started with Image Preprocessing and Augmentation for Deep Learning (Computer Vision Toolbox)
- Get Started with Medical Segment Anything Model for Medical Image Segmentation (Medical Imaging Toolbox)