Face Modelling and Tracking from Range Scans (original) (raw)

Automatic 3D Facial Model and Texture Reconstruction from Range Scans

2010

This paper presents a fully automatic approach to fitting a generic facial model to detailed range scans of human faces to reconstruct 3D facial models and textures with no manual intervention (such as specifying landmarks). A Scaling Iterative Closest Points (SICP) algorithm is introduced to compute the optimal rigid registrations between the generic model and the range scans with different sizes. And then a new template-fitting method, formulated in an optmization framework of minimizing the physically based elastic energy derived from thin shells, faithfully reconstructs the surfaces and the textures from the range scans and yields dense point correspondences across the reconstructed facial models. Finally, we demonstrate a facial expression transfer method to clone facial expressions from the generic model onto the reconstructed facial models by using the deformation transfer technique.

Reanimating real humans: automatic reconstruction of animated faces from range data

2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763), 2004

Advances in 3D scanning technology have enabled automatic capture of complex 3D models such as human faces with highly detailed surfaces. However, the range data can not be used easily for animatable face modeling due to absence of functional animation structure and dense surface data. This paper presents an automatic facial model adaptation algorithm for reconstruction of animatable individualized 3D facial models from range data. A generic model that represents both the face shape and anatomical structure serves as the starting point for the adaptation algorithm. The global adaptation transforms the generic model to align it with the scan data in the 3D space based the measurements between a set of 3D landmarks. The local adaptation then deforms the skin mesh of the generic model to fit all of its vertices to the scan surface. The underlying muscle structure is automatically adapted and facial texture is transferred. The reconstructed 3D face resembles the shape and color of real individual and can be animated immediately with the muscle parameters.

3D Face Mesh Modeling from Range Images for 3D Face Recognition

2007 IEEE International Conference on Image Processing, 2007

We present an algorithm for 3D face deformation and modeling using range data captured by a 3D scanner. Using only three facial feature points extracted from the range images and a 3D generic face model, the algorithm first aligns the 3D model to the entire range data of a given subject's face. Then each aligned triangle of the mesh model, with three vertices, is treated as a surface plane which is then fitted to the corresponding interior 3D range data, using least squares plane fitting. Via triangular vertices subdivisions, a higher resolution model is generated from the coordinates of the aligned and fitted model. Finally the model and its triangular surfaces are fitted once again resulting in a smoother mesh model that resembles and captures the surface characteristic of the face. Application of the final deformed model in 3D face recognition, using a publicly available database, shows promising results.

Fully Automatic Facial Deformation Transfer

Symmetry

Facial Animation is a serious and ongoing challenge for the Computer Graphic industry. Because diverse and complex emotions need to be expressed by different facial deformation and animation, copying facial deformations from existing character to another is widely needed in both industry and academia, to reduce time-consuming and repetitive manual work of modeling to create the 3D shape sequences for every new character. But transfer of realistic facial animations between two 3D models is limited and inconvenient for general use. Modern deformation transfer methods require correspondences mapping, in most cases, which are tedious to get. In this paper, we present a fast and automatic approach to transfer the deformations of the facial mesh models by obtaining the 3D point-wise correspondences in the automatic manner. The key idea is that we could estimate the correspondences with different facial meshes using the robust facial landmark detection method by projecting the 3D model to ...

Learning a model of facial shape and expression from 4D scans

Fig. 1. FLAME example. Top: Samples of the D3DFACS dataset. Middle: Model-only registration. Bottom: Expression transfer to Beeler et al. [2011] subject using model only. The field of 3D face modeling has a large gap between high-end and low-end methods. At the high end, the best facial animation is indistinguishable from real humans, but this comes at the cost of extensive manual labor. At the low end, face capture from consumer depth sensors relies on 3D face models that are not expressive enough to capture the variability in natural facial shape and expression. We seek a middle ground by learning a facial model from thousands of accurately aligned 3D scans. Our FLAME model (Faces Learned with an Articulated Model and Expressions) is designed to work with existing graphics software and be easy to fit to data. FLAME uses a linear shape space trained from 3800 scans of human heads. FLAME combines this linear shape space with an articulated jaw, neck, and eyeballs, pose-dependent corrective blendshapes, and additional global expression * Both authors contributed equally to the paper † This research was performed while TL and JR were at the MPI for Intelligent Systems. blendshapes. The pose and expression dependent articulations are learned from 4D face sequences in the D3DFACS dataset along with additional 4D sequences. We accurately register a template mesh to the scan sequences and make the D3DFACS registrations available for research purposes. In total the model is trained from over 33, 000 scans. FLAME is low-dimensional but more expressive than the FaceWarehouse model and the Basel Face Model. We compare FLAME to these models by fitting them to static 3D scans and 4D sequences using the same optimization method. FLAME is significantly more accurate and is available for research purposes (http://flame.is.tue.mpg.de).

3D Body Reconstruction from Photos Based on Range Scan

Technologies for E-Learning and Digital Entertainment, 2006

We present a data-driven shape model for reconstructing human body models from one or more 2D photos. One of the key tasks in reconstructing the 3D model from image data is shape recovery, a task done until now in utterly geometric way, in the domain of human body modeling. In contrast, we adopt a data-driven, parameterized deformable model that is acquired from a collection of range scans of real human body. The key idea is to complement the image-based reconstruction method by leveraging the quality shape and statistic information accumulated from multiple shapes of range-scanned people. In the presence of ambiguity either from the noise or missing views, our technique has a bias towards representing as much as possible the previously acquired 'knowledge' on the shape geometry. Texture coordinates are then generated by projecting the modified deformable model onto the front and back images. Our technique has shown to reconstruct successfully human body models from minimum number images, even from a single image input.

Automatic non-rigid registration of 3D dynamic data for facial expression synthesis and transfer

2008 IEEE Conference on Computer Vision and Pattern Recognition, 2008

Automatic non-rigid registration of 3D time-varying data is fundamental in many vision and graphics applications such as facial expression analysis, synthesis, and recognition. Despite many research advances in recent years, it still remains to be technically challenging, especially for 3D dynamic, densely-sampled facial data with a large number of degrees of freedom (necessarily used to represent rich and subtle facial expressions). In this paper, we present a new method for automatic non-rigid registration of 3D dynamic facial data using least-squares conformal maps, and based on this registration method, we also develop a new framework of facial expression synthesis and transfer. Nowadays more and more 3D dynamic, densely-sampled data become prevalent with the advancement of novel 3D scanning techniques. To analyze and utilize such huge 3D data, an efficient non-rigid registration algorithm is needed to establish one-to-one inter-frame correspondences. Towards this goal, a non-rigid registration algorithm of 3D dynamic facial data is developed by using least-squares conformal maps with additional feature correspondences detected by employing active appearance models (AAM). The proposed method with additional, interior feature constraints guarantees that the non-rigid data will be accurately registered. The least-squares conformal maps between two 3D surfaces are globally optimized with the least angle distortion and the resulting 2D maps are stable and one-to-one. Furthermore, by using this non-rigid registration method, we develop a new system of facial expression synthesis and transfer. Finally, we perform a series of experiments to evaluate our non-rigid registration method and demonstrate its efficacy and efficiency in the applications of facial expression synthesis and transfer.

Multi-scale capture of facial geometry and motion

2007

Abstract We present a novel multi-scale representation and acquisition method for the animation of high-resolution facial geometry and wrinkles. We first acquire a static scan of the face including reflectance data at the highest possible quality. We then augment a traditional marker-based facial motion-capture system by two synchronized video cameras to track expression wrinkles. The resulting model consists of high-resolution geometry, motion-capture data, and expression wrinkles in 2D parametric form.

On the Multi-View Fitting and Construction of Dense Deformable Face Models

Active Appearance Models (AAMs) are generative, parametric models that have been successfully used in the past to model deformable objects such as human faces. Fitting an AAM to an image consists of minimizing the error between the input image and the closest model instance; i.e. solving a nonlinear optimization problem. In this thesis we study three important topics related to deformable face models such as AAMs: (1) multi-view 3D face model fitting, (2) multi-view 3D face model construction, and (3) automatic dense deformable face model construction. The original AAMs formulation was 2D, but they have recently been extended to include a 3D shape model. A variety of single-view algorithms exist for fitting and constructing 3D AAMs but one area that has not been studied is multi-view algorithms. In the first part of this thesis we describe an algorithm for fitting a single AAM to multiple images, captured simultaneously by cameras with arbitrary locations, rotations, and response functions. This algorithm uses the scaled orthographic imaging model used by previous authors, and in the process of fitting computes, or calibrates, the scaled orthographic camera matrices. We also describe an extension of this algorithm to calibrate weak perspective (or full perspective) camera models for each of the cameras. In essence, we use the human face as a (nonrigid) calibration grid. We demonstrate that the performance of this algorithm is roughly comparable to a standard algorithm using a calibration grid. We then show how camera calibration improves the performance of AAM fitting. A variety of non-rigid structure-from-motion algorithms, both single-view and multiview, have been proposed that can be used to construct the corresponding 3D non-rigid shape models of a 2D AAM. In the second part of this thesis we show that constructing a 3D face model using non-rigid structure-from-motion suffers from the Bas-Relief ambiguity and may result in a "scaled" (stretched/compressed) model. We outline a robust non-rigid Mohit Gupta; for sitting through my practice talks and providing constructive criticisms. Many thanks to Ralph Gross, Goksel Dedeoglu and Fernando De La Torre for their help.