Face Recognition and 3D Face Modeling
Technical Overview
Overview
Microsoft Research has developed a framework for face recognition. The system takes as input a
simple frontal photograph of a face taken with normal illumination and neutral expression. It then
creates a 3D face model. Different, realistic virtual faces with variant pose, illumination and
expression are synthesized and used for face recognition. Test results showed that face recognition
accuracy for non frontal poses is better than conventional approaches. The best performance
improvements are achieved in half-profile views.
In a first step frontal face detection and alignment algorithms are utilized to locate a frontal face and
the facial feature points within an image, such as the contour points of the face, left and right eyes,
mouth and nose. Then, the 3D face shape is reconstructed according to the feature points and a 3D
face database. After that, the face model is texture-mapped by projecting the input 2D image onto the
3D face shape. Based on this 3D face model, virtual samples with variant pose, illumination and
expression are synthesized to represent the 2D face image space. Finally, face recognition is
conducted in this enlarged face subspace. The output images with variant pose, illumination and
expression greatly improve face recognition results.
The images shown below represent sample output of the 3D face modeling tool. The first and third
rows are real photographs of various poses taken from a test database provided by Carnegie Mellon
University. The second and fourth rows are the corresponding poses 3D Face Modeling generated.
© 2006 Microsoft Corporation. All rights reserved.
The information contained in this document relates to Microsoft Research pre-release/prototype products
and technologies, which may be substantially modified before their final commercial release by third party
IP venture companies. Accordingly, the information may not accurately describe or reflect the products
and technologies when first commercially released. This document is provided for informational purposes
only and Microsoft makes no warranties, express or implied, with respect to this document or the
information contained in it. Reproduction and redistribution of this document requires the express
permission of Microsoft Corporation.
The images shown below illustrate how the system creates facial expressions. Again, the first row is
the expressions taken from a test database provided by Carnegie Mellon University. The second row
shows the 3D Face Modeling generated expressions corresponding to the first line.
© 2006 Microsoft Corporation. All rights reserved.
The information contained in this document relates to Microsoft Research pre-release/prototype products
and technologies, which may be substantially modified before their final commercial release by third party
IP venture companies. Accordingly, the information may not accurately describe or reflect the products
and technologies when first commercially released. This document is provided for informational purposes
only and Microsoft makes no warranties, express or implied, with respect to this document or the
information contained in it. Reproduction and redistribution of this document requires the express
permission of Microsoft Corporation.
Components
The Face Recognition and 3D Face Modeling framework consists of two parts:
1. 2D-to-3D face reconstruction:
Starting from a frontal face image of a person with normal illumination and neutral
expression the software finds 83 key feature points such as face contour points, eye centers
and nose contour that characterize the face. In another step the key feature points are used
to compute the 3D face model. In a final step the face texture is extracted from the input
image and mapped to the 3D face model.
2. Face recognition:
To increase the accuracy of the face recognition algorithm, many face images are created by
rotating the 3D face model. Test results showed that face recognition accuracy for non
frontal poses is better than conventional approaches. The best performance improvements
are achieved in half-profile views.
Core Scenarios
1.
2.
3.
4.
3D face reconstruction and 3D face modeling can be applied to video and online gaming
(avatars, personalization).
This system could be applied to the following surveillance scenarios:
i. In casinos to detect people who have been barred from gambling.
ii. In correctional facilities to screen visitors.
iii. In airports to identify aircrews or to detect criminals.
An intelligent photo sorting application could use face recognition technology to
automatically tag and sort photos. A user would tag a photo with a particular person once
and the application could automatically tag all other photos featuring that person.
Other areas where face recognition is useful include
i. Document control (photos in passports or national ID cards)
ii. Computer security (user access verification)
iii. Time and attendance (entry/exit verification)
© 2006 Microsoft Corporation. All rights reserved.
The information contained in this document relates to Microsoft Research pre-release/prototype products
and technologies, which may be substantially modified before their final commercial release by third party
IP venture companies. Accordingly, the information may not accurately describe or reflect the products
and technologies when first commercially released. This document is provided for informational purposes
only and Microsoft makes no warranties, express or implied, with respect to this document or the
information contained in it. Reproduction and redistribution of this document requires the express
permission of Microsoft Corporation.
iv. Transactional authentication (credit cards, ATMs, point-of-sale)
Documentation
The following publication by Microsoft Research is related to Face Recognition and 3D Face
Modeling:

Yuxiao Hu, Dalong Jiang, Shuicheng Yan, Lei Zhang, Hongjiang Zhang, Automatic 3D
Reconstruction for Face Recognition, in Proc. 6th IEEE International Conference on Automatic
Face and Gesture Recognition (FG), Seoul, Korea, 2004
Related Work
1.
2.
Face Recognition and 3D Face Modeling is based on face detection technology also available
for licensing through the Microsoft IP Ventures program. Please click here for more
information.
Another application of the face alignment methods in Face Recognition and 3D Face
Modeling is implemented in a technology called Personalized Facial Sketch also available for
licensing. Please click here for more information.
Differentiating Features
1.
2.
3.
Only one single photo of a frontal face is required as input for Face Recognition. The
burdensome work of adding faces to the database is significantly less compared to other face
recognition solutions available in the market.
The synthesized face samples provide the capability to conduct recognition under difficult
conditions including variant pose, illumination and expression.
The 2D-to-3D integrated face reconstruction approach is fully automatic and faster than
other 3D reconstruction approaches.
© 2006 Microsoft Corporation. All rights reserved.
The information contained in this document relates to Microsoft Research pre-release/prototype products
and technologies, which may be substantially modified before their final commercial release by third party
IP venture companies. Accordingly, the information may not accurately describe or reflect the products
and technologies when first commercially released. This document is provided for informational purposes
only and Microsoft makes no warranties, express or implied, with respect to this document or the
information contained in it. Reproduction and redistribution of this document requires the express
permission of Microsoft Corporation.
Demo/Prototype
Development Status
Face Recognition and 3D Face Modeling is currently a research prototype. The following screenshot
shows the user interface for 3D Face Reconstruction and Synthesis:
Technical Specifications


Implemented in C++
Recognition speed: ~4 sec/face image (512 x 512 pixels) on a P4, 1.3 GHz PC with 256 MB
RAM
© 2006 Microsoft Corporation. All rights reserved.
The information contained in this document relates to Microsoft Research pre-release/prototype products
and technologies, which may be substantially modified before their final commercial release by third party
IP venture companies. Accordingly, the information may not accurately describe or reflect the products
and technologies when first commercially released. This document is provided for informational purposes
only and Microsoft makes no warranties, express or implied, with respect to this document or the
information contained in it. Reproduction and redistribution of this document requires the express
permission of Microsoft Corporation.