A Bayesian Approach to Digital Matting (original) (raw)
Yung-Yu Chuang1 Brian Curless1 David Salesin1,2 Richard Szeliski2
1University of Washington 2Microsoft Research
Abstract
This paper proposes a new Bayesian framework for solving the matting problem, i.e. extracting a foreground element from a background image by estimating an opacity for each pixel of the foreground element. Our approach models both the foreground and background color distributions with spatially-varying mixtures of Gaussians, and assumes a fractional blending of the foreground and background colors to produce the final output. It then uses a maximum-likelihood criterion to estimate the optimal opacity, foreground and background simultaneously. In addition to providing a principled approach to the matting problem, our algorithm effectively handles objects with intricate boundaries, such as hair strands and fur, and provides an improvement over existing techniques for these difficult cases.Citation (bibTex)
Yung-Yu Chuang, Brian Curless, David H. Salesin, and Richard Szeliski. A Bayesian Approach to Digital Matting. In Proceedings of IEEE Computer Vision and Pattern Recognition (CVPR 2001), Vol. II, 264-271, December 2001Paper
CVPR 2001 paper (3.6MB PDF) Addendum
- We forgot to mention one thing in the paper. Because foreground and background samples are also observations from the camera, they should have the same noise characteristics as the observation C. Hence, we added the same amount of camera variance \sigmac to the covariance matrices of foreground and background samples in Equation (7). We used eigen-analysis to find the orientation of the covariance matrix and added \sigmac2 in every axis. That is, we decomposed \SigmaF as U S VT. Let S=diag{s12,s22,s32}, we set S'=diag(s12+\sigmac2, s22+\sigmac2, s32+\sigmac2) and assign the new \Sigma_F as U S' VT. By doing so, we also avoided most of the degenerate cases, i.e., non-invertible matrices.
- For the window for collecting foreground and background samples, we set a minimal window size and a minimal number of samples. We start from a window with the minimal window sizw. If such a window does not give us enough samples, we gradually increase the window until the minimal number of samples is satistified. Note that, in this way, the windows for background and foreground might end up with different sizes. Results
Inputs, Masks and Composites
| | Blue-screen matting | Difference matting | Natural image matting | | |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Input | 
| 
| 
| 
|
| Segmentation | 
| 
| 
| 
|
| Composite(Bayesian) | 
| | 
| 
|
Lighthouse image and background images used in composite courtesy Philip Greenspun, http://philip.greenspun.com.
Woman image was obtained from Corel Knockout's tutorial, Copyright © 2001 Yung-Yu Chuang, Brian Curless, David Salesin, Richard Szeliski and its licensors Corel. All rights reserved.
Blue-screen Matting
| | Alpha Matte | Composite (black) | Inset | Composite | |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| Mishima | 
| 
| 
| 
|
| Bayesian | 
| 
| 
| 
|
| Ground truth | 
| 
| 
| 
|
"Synthetic" Natural Image Matting
| | Alpha Matte | Composite | Inset | |
| ----------------- | ---------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| DifferenceMatting | 
| 
| 
|
| Knockout | 
| 
| 
|
| Ruzon &Tomasi | 
| 
| 
|
| Bayesian | 
| 
| 
|
| GroundTruth | 
| 
| 
|
Natural Image Matting
| | Alpha Matte | Composite | Inset | Alpha Matte | Composite | Inset | |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| Knockout | 
| 
| 
| 
| 
| 
|
| Ruzon &Tomasi | 
| 
| 
| 
| 
| 
|
| Bayesian | 
| 
| 
| 
| 
| 
|
Additional results
| Input |  |
 |
 |
 |
 |
|---|---|---|---|---|---|
| Alpha |  |
 |
 |
 |
 |
| Composite |  |
 |
 |
 |
 |
The first two images courtesy Philip Greenspun, http://philip.greenspun.com.
Woman image was obtained from Corel Knockout's tutorial.
cyy -a-t- cs.washington.edu