Reconstruction of 3D faces by shape estimation and texture interpolation (original) (raw)

On Learning 3D Face Morphable Model from In-the-wild Images

IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019

As a classic statistical model of 3D facial shape and albedo, 3D Morphable Model (3DMM) is widely used in facial analysis, e.g., model fitting, image synthesis. Conventional 3DMM is learned from a set of 3D face scans with associated well-controlled 2D face images, and represented by two sets of PCA basis functions. Due to the type and amount of training data, as well as, the linear bases, the representation power of 3DMM can be limited. To address these problems, this paper proposes an innovative framework to learn a nonlinear 3DMM model from a large set of in-the-wild face images, without collecting 3D face scans. Specifically, given a face image as input, a network encoder estimates the projection, lighting, shape and albedo parameters. Two decoders serve as the nonlinear 3DMM to map from the shape and albedo parameters to the 3D shape and albedo, respectively. With the projection parameter, lighting, 3D shape, and albedo, a novel analytically-differentiable rendering layer is designed to reconstruct the original input face. The entire network is end-to-end trainable with only weak supervision. We demonstrate the superior representation power of our nonlinear 3DMM over its linear counterpart, and its contribution to face alignment, 3D reconstruction, and face editing.

Reconstructing 3D Face Shapes from Single 2D Images Using an Adaptive Deformation Model

Lecture Notes in Computer Science, 2013

The Representational Power (RP) of an example-based model is its capability to depict a new 3D face for a given 2D face image. In this contribution, a novel approach is proposed to increase the RP of the 3D reconstruction PCA-based model by deforming a set of examples in the training dataset. By adding these deformed samples together with the original training samples we gain more RP. A 3D PCA-based model is adapted for each new input face image by deforming 3D faces in the training data set. This adapted model is used to reconstruct the 3D face shape for the given input 2D near frontal face image. Our experimental results justify that the proposed adaptive model considerably improves the RP of the conventional PCA-based model.

3D Face Reconstruction from 2D Images

2008 Digital Image Computing: Techniques and Applications, 2008

This paper surveys the topic of 3D face reconstruction using 2D images from a computer science perspective. Various approaches have been proposed as solutions for this problem but most have their limitations and drawbacks. Shape from shading, Shape from silhouettes, Shape from motion and Analysis by synthesis using morphable models are currently regarded as the main methods of attaining the facial information for reconstruction of its 3D counterpart. Though this topic has gained a lot of importance and popularity, a fully accurate facial reconstruction mechanism has not yet being identified due to the complexity and ambiguity involved. This paper discusses about the general approaches of 3D face reconstruction and their drawbacks. It concludes with an analysis of several implementations and some speculations about the future of 3D face reconstruction.

Method Comparison of 3D Facial Reconstruction Coresponding to 2D Image

IOP Conference Series: Materials Science and Engineering

Recent years, facial reconstruction becomes a much-studied topic. In particular, the past decade has witnessed a renewed interest, generating a large number of research centers and proposing techniques to address them but most have their limitations and drawbacks. The main constrain of attaining an accuracy of the facial information for reconstruction of its 3D counterpart was increasing accuracy for 3D face geometry. Though this topic has gained a lot of concern and popularity, a fully accurate facial reconstruction mechanism has not yet been identified due to the complexity and ambiguity involved. This survey focuses on 3D face reconstruction by presenting comparison methods beetween Shape-from-Shading, the 3D Morphable Model and Structure from Movement based on 2D face analysis with higher accuracy. 3D face model density affects the provided information. The dense of the 3D facial model is; the more information it could provide. 3D facial reconstruction method currently requires a complicated process and high system costs.

3D Face Reconstruction from a Single Image Using a Single Reference Face Shape

IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000

Human faces are remarkably similar in global properties, including size, aspect ratio, and location of main features, but can vary considerably in details across individuals, gender, race, or due to facial expression. We propose a novel method for 3D shape recovery of faces that exploits the similarity of faces. Our method obtains as input a single image and uses a mere single 3D reference model of a different person's face. Classical reconstruction methods from single images, i.e., shape-from-shading, require knowledge of the reflectance properties and lighting as well as depth values for boundary conditions. Recent methods circumvent these requirements by representing input faces as combinations (of hundreds) of stored 3D models. We propose instead to use the input image as a guide to "mold" a single reference model to reach a reconstruction of the sought 3D shape. Our method assumes Lambertian reflectance and uses harmonic representations of lighting. It has been tested on images taken under controlled viewing conditions as well as on uncontrolled images downloaded from the Internet, demonstrating its accuracy and robustness under a variety of imaging conditions and overcoming significant differences in shape between the input and reference individuals including differences in facial expressions, gender, and race.

Feature-preserving detailed 3D face reconstruction from a single image

Proceedings of the 15th ACM SIGGRAPH European Conference on Visual Media Production - CVMP '18

Dense 3D face reconstruction plays a fundamental role in visual media production involving digital actors. We improve upon high fidelity reconstruction from a single 2D photo with a reconstruction framework that is robust to large variations in expressions, poses and illumination. We provide a global optimization step improving the alignment of 3D facial geometry to tracked 2D landmarks with 3D Laplacian deformation. Face detail is improved through, extending Shape from Shading reconstruction with fitted albedo prior masks, together with a fast proportionality constraint between depth and image gradients consistent with local self-occlusion behavior. Together these measures better preserve the crucial facial features that define an actor's identity, and we illustrate this through a variety of comparisons with related works.

3D Face Reconstruction from Monocular Video and its Applications In the Wild

2020

3D face reconstruction is a very popular field of computer vision due to its applications in social media, entertainment and health. However, ever since the introduction of 3D morphable models as facial priors, 3D face reconstruction has been dominated by reconstruction from single images due to its ease and proximity to 3D face alignment. Even so, single image reconstruction methods suffer from inconsistent reconstructions across time and view points. Hence a natural extension is to reconstruct 3D face shape from videos. Because of recent methods in single image reconstruction setting the standards for state-of-the-art reconstruction, we introduce a method to fuse single image reconstructions across multiple frames to create a more accurate reconstruction. Furthermore, the lack of structured video datasets that fully captures the face and provide 3D ground truth scans, led us to develop and release the 3DFAW-Video dataset and challenge. We also introduce a symmetric distance metric...

State-of-the-Art in 3D Face Reconstruction from a Single RGB Image

2021

Since diverse and complex emotions need to be expressed by different facial deformation and appearances, facial animation has become a serious and ongoing challenge for computer animation industry. Face reconstruction techniques based on 3D morphable face model and deep learning provide one effective solution to reuse existing databases and create believable animation of new characters from images or videos in seconds, which greatly reduce heavy manual operations and a lot of time. In this paper, we review the databases and state-of-the-art methods of 3D face reconstruction from a single RGB image. First, we classify 3D reconstruction methods into three categories and review each of them. These three categories are: Shape-from-Shading (SFS), 3D Morphable Face Model (3DMM), and Deep Learning (DL) based 3D face reconstruction. Next, we introduce existing 2D and 3D facial databases. After that, we review 10 methods of deep learning-based 3D face reconstruction and evaluate four representative ones among them. Finally, we draw conclusions of this paper and discuss future research directions.

3D Face Reconstruction From Single 2D Image Using Distinctive Features

3D face reconstruction is considered to be a useful computer vision tool, though it is difficult to build. This paper proposes a 3D face reconstruction method, which is easy to implement and computationally efficient. It takes a single 2D image as input, and gives 3D reconstructed images as output. Our method primarily consists of three main steps: feature extraction, depth calculation, and creation of a 3D image from the processed image using a Basel face model (BFM). First, the features of a single 2D image are extracted using a two-step process. Before distinctive-features extraction, a face must be detected to confirm whether one is present in the input image or not. For this purpose, facial features like eyes, nose, and mouth are extracted. Then, distinctive features are mined by using scale-invariant feature transform (SIFT), which will be used for 3D face reconstruction at a later stage. Second step comprises of depth calculation, to assign the image a third dimension. Multivariate Gaussian distribution helps to find the third dimension, which is further tuned using shading cues that are obtained by the shape from shading (SFS) technique. Thirdly, the data obtained from the above two steps will be used to create a 3D image using BFM. The proposed method does not rely on multiple images, lightening the computation burden. Experiments were carried out on different 2D images to validate the proposed method and compared its performance to those of the latest approaches. Experiment results demonstrate that the proposed method is time efficient and robust in nature, and it outperformed all of the tested methods in terms of detail recovery and accuracy. INDEX TERMS 3D face reconstruction, feature extraction, facial modeling, gaussian distribution.