Real-time facial feature tracking from 2D+3D video streams (original) (raw)

Local or Global 3D Face and Facial Feature Tracker

2007

We present in this paper a solution for 3D face and facial feature tracking using canonical correlation analysis and a 3D geometric model. This model is controlled with 17 parameters (6 for the 3D pose, and 11 for facial animation), and is used to crop out reference 2D shape free texture maps from the incoming input frames. Model parameters are updated via image registration in the texture map space. For registration, we use CCA to learn and exploit the dependency between texture residuals and model parameter corrections. We compare tracking results using two kinds of texture maps: one local (image patches around selected vertices of the 3D model), and one global (the whole image patch under the 3D model). Experiments evaluating the effectiveness on the approaches are reported.

Structure and appearance features for robust 3D facial actions tracking

2009 IEEE International Conference on Multimedia and Expo, 2009

This paper presents a robust and accurate method for joint head pose and facial actions tracking, even under challenging conditions such as varying lighting, large head movements, and fast motion. This is made possible by the combination of two types of facial features. We use locations sampled from the facial texture whose appearance is initialized on the first frame and adapted over time, and also illumination-invariant patches located on characteristic points of the face such as the corners of the eyes or of the mouth. The first type of features contains rich information about the global appearance of the face and thus leads to an accurate tracking, while the second type guaranties robustness and stability by avoiding drift. We demonstrate our system on the Boston University Face Tracking benchmark, and show it outperforms state-of-the-art methods.

Linear Tracking of Pose and Facial Features

2007

We present an approach for simultaneous monocular 3D face pose and facial animation tracking. The pose and facial features are estimated from observed raw brightness shape-free 2D image patches. A parameterized 3D face model is adopted to crop out and to normalize the shape of patches from video frames. Starting from the face model aligned on an observed human face, we learn the relation between a set of perturbed parameters of the face model and the associated image patches using a Canonical Correlation Analysis. This knowledge, obtained from an observed patch in the current frame, is used to estimate the correction to be added to the pose of the face and to the animation parameters controlling the lips, eyebrows and eyes. Ground truth data is used to evaluate both the pose and facial animation tracking efficiency in long real video sequences.

Fitting and Tracking 3D/4D Facial Data Using a Temporal Deformable Shape Model

ICME, 2013

In this paper, we propose a novel method for detecting and tracking landmark facial features on purely geometric 3D and 4D range models. Our proposed method involves fitting a new multiframe constrained 3D temporal deformable shape model (TDSM) to range data sequences. We consider this a temporal based deformable model as we concatenate consecutive deformable shape models into a single model driven by the appearance of facial expressions. This allows us to simultaneously fit multiple models over a sequence of time with one TDSM. To our knowledge, it is the first work to address multiple shape models as a whole to track 3D dynamic range sequences without assistance of any texture information. The accuracy of the tracking results is evaluated by comparing the detected landmarks to the ground truth. The efficacy of the 3D feature detection and tracking over range model sequences has also been validated through an application in 3D geometric based face and expression analysis and expression sequence segmentation. We tested our method on the publicly available databases, BU-3DFE [15], BU-4DFE [16], and FRGC 2.0 . We also validated our approach on our newly developed 3D dynamic spontaneous expression database .

Robust facial action recognition from real-time 3D streams

This paper presents a completely automated facial action and facial expression recognition system using 2D+3D images recorded in real-time by a structured light sensor. It is based on local feature tracking and rule-based classification of geometric, appearance and surface curvature measurements. Good performance is achieved under relatively non-controlled conditions. Measurement name Measurement M 1 Inner eyebrow displacement d 5,22 , d 9,27 M 2 Outer eyebrow displacement d 7,17 , d 15,26 M 3 Inner eyebrow corners dist. d 5,9 M 4 Eyebrow from nose root dist. d 5,35 , d 9,45 M 5 Eye opening d 20,24 , d 29,33 M 6 Eye shape d 20,24 /d 18,22 M 7 Nose length (d 35,36 + d 45,44 )/2 M 8 Nose width d 36,44 M 9

An Active Model for Facial Feature Tracking

EURASIP Journal on Advances in Signal Processing, 2002

We present a system for finding and tracking a face and extract global and local animation parameters from a video sequence. The system uses an initial colour processing step for finding a rough estimate of the position, size, and inplane rotation of the face, followed by a refinement step drived by an active model. The latter step refines the previous estimate, and also extracts local animation parameters. The system is able to track the face and some facial features in near real-time, and can compress the result to a bitstream compliant to MPEG-4 face and body animation.

3D Constrained Local Model for rigid and non-rigid facial tracking

2012

We present 3D Constrained Local Model (CLM-Z) for robust facial feature tracking under varying pose. Our approach integrates both depth and intensity information in a common framework. We show the benefit of our CLM-Z method in both accuracy and convergence rates over regular CLM formulation through experiments on publicly available datasets. Additionally, we demonstrate a way to combine a rigid head pose tracker with CLM-Z that benefits rigid head tracking. We show better performance than the current state-of- ...

A Robust Facial Feature Point Tracker using Graphical Models

2007 5th International Symposium on Image and Signal Processing and Analysis, 2007

In recent years, facial feature point tracking becomes a research area that is used in human-computer interaction (HCI), facial expression analysis, etc. In this paper, a statistical method for facial feature point tracking is proposed. Feature point tracking is a challenging topic in scenarios involving arbitrary head movements and uncertain data because of noise and/or occlusions. As a natural human action, people move their heads or occlude their faces with their hands or fingers. With this motivation, a graphical model that uses temporal information about feature point movements as well as the spatial relationships between such points, which is updated in time to deal with different head pose variations, is built. Based on this model, an algorithm that achieves feature point tracking through a video observation sequence is implemented. Also, an occlusion detector is proposed to automatically detect occluded points. The proposed method is applied on 2D gray scale video sequences consisting head movements and occlusions and the superiority of this approach over existing techniques is demonstrated.

Face pose estimation and tracking using automatic 3D model construction

2008

This paper presents a method for robustly tracking and estimating the face pose of a person in both indoor and outdoor environments. The method is invariant to identity and that does not require previous training. A face model is automatically initialized and constructed on-line, when the face is frontal to the stereo camera system. To build the model, a fixed point distribution is superposed over the frontal face, and several appropriate points close to those locations are chosen for tracking. Using the stereo correspondence of the two cameras, the 3D coordinates of these points are extracted, and the 3D model is created. RANSAC and POSIT are used for tracking and 3D pose calculation at each frame. The approach runs in real time, and has been tested on sequences recorded in the laboratory and in a moving car.

Face tracking and pose estimation with automatic three-dimensional model construction

IET COMPUTER VISION, 2009

A method for robustly tracking and estimating the face pose of a person using stereo vision is presented. The method is invariant to identity and does not require previous training. A face model is automatically initialised and constructed online: a fixed point distribution is superposed over the face when it is frontal to the cameras, and several appropriate points close to those locations are chosen for tracking. Using the stereo correspondence of the cameras, the three-dimensional (3D) coordinates of these points are extracted, and the 3D model is created. The 2D projections of the model points are tracked separately on the left and right images using SMAT. RANSAC and POSIT are used for 3D pose estimation. Head rotations up to +458 are correctly estimated. The approach runs in real time. The purpose of this method is to serve as the basis of a driver monitoring system, and has been tested on sequences recorded in a moving car.