The University of Auckland | Computer Science | New Zealand
PRESENTATION
3D Human Face Reconstruction
and Expression Modelling
Alexander Woodward
2009
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Outline
Aim
System overview
Related work
3D face reconstruction
Expression modelling
Contributions and future work
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Overview
Aim: Integrated system for 3D face reconstruction and
expression modelling
 Vision based not graphics based
 Low cost and self-contained
Results can be applied to:
Biometrics and security
Biomedical visualisation
Computer and video games
Film
Teleconferencing
Human computer interaction
3
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
System overview
3D reconstruction
Static
Dynamic
Active & passive binocular stereo
3D video scanner
Active structured lighting
Active photometric stereo
3D data
Expression modelling
Marker based motion capture
Muscle inverse kinematics
Video based
Sequences from 3D video
scanner
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Related work
Complete systems for face reconstruction and
animation are uncommon
High hardware requirements
Data acquisition, motion capture and animation systems are
often provided as disparate packages or only as a service,
cf. a stand-alone solution
At least 9 prominent projects aimed toward complete
systems
Excluding in-house solutions
Large body of work in 3D face research
 3D reconstruction, expressions, motion capture
13 April 2015
Department of Computer Science
5
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Related work
Borshukov et al (2003 – 2007)
Playable Universal Capture approach
 3D scanner, marker based tracking, optical flow, video texture
Ma et al (2007, 2008)
Capture face reflectance
 3D scanner, photometric stereo, motion capture
 Light stage – 156 LED lights over an icosahedron
Image Metrics Inc. & U Sth Carolina Graphics Lab (2008)
Digital Emily project
 Light stage captures geometry and reflectance
 33 expressions captured; creates an animation rig
 Performance data mapped to the 3D face
6
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
3D reconstruction
3D reconstruction
Static
Dynamic
Active & passive binocular stereo
3D video scanner
Active structured lighting
Active photometric stereo
3D data
Expression modelling
Marker based motion capture
Muscle inverse kinematics
Video based
Sequences from 3D video
scanner
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
3D reconstruction requirements
Off-the-shelf hardware, no special properties
Cameras, PC, projector
Low acquisition time – faces move, esp. children
Controlled lighting
Vision based
New algorithms
Useful for any type of object
8
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Static 3D reconstruction
3D reconstruction
Static
Dynamic
Active & passive binocular stereo
3D video scanner
Active structured lighting
Active photometric stereo
3D data
Expression modelling
Marker based motion capture
Muscle inverse kinematics
Video based
Sequences from 3D video
scanner
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Static 3D reconstruction
 Evaluated approaches:
1. Active & passive binocular stereo
2. Active structured lighting
3. Active photometric stereo
 Evaluate effectiveness
 Accuracy, time complexity
 Determine best approach for dynamic
3D reconstruction system
 12 algorithms
 Database of 15 faces
 Alternative test set
 Focus on stereo algorithms
 Ground truth data: 3D scanner
 Compared to Middlebury, algorithms rank differently for faces
 Projected patterns improve and level out performance
10
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Active binocular stereo
Strip colour pattern: much higher accuracy
SAD correlation algorithm:
Strip pattern
SAD - without
pattern
SAD - with
strip pattern
80%
92%
Pattern colour should contrast strongly on skin
11
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Statistical results
BP
73%
GC
77%
FCV
69%
+ Grad. + Strip
77%
89%
+ Grad. + Strip
83%
92%
Four Path Shapelet
54%
71%
CM
88%
SAD
80%
+ Grad. + Strip
89%
92%
+ Grad. + Strip
85%
92%
DPM
+ Grad.
+ Strip
79%
84%
92%
SDPS + Grad.
89%
90%
+ Strip
93%
Gray code
97%
Ground truth
12
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Dynamic 3D reconstruction
3D reconstruction
Static
Dynamic
Active & passive binocular stereo
3D video scanner
Active structured lighting
Active photometric stereo
3D data
Expression modelling
Marker based motion capture
Muscle inverse kinematics
Video based
Sequences from 3D video
scanner
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Dynamic 3D reconstruction
Reconstruction at video rates →3D video!
 From static reconstruction best results:
 ‘One shot’ active illumination + Symmetric Dynamic Programming (SDPS)
 Project pattern every other frame to get a clean texture
(2)
(3)
(1)
Monochrome stereo pair of video cameras +
3rd colour web camera obtains colour texture.
14
2009
The University of Auckland | Computer Science | New Zealand
PRESENTATION
Colour texture generation
+
Colour image
(reprojected into same
reference frame)
→
Monochrome
image
Final texture
Low resolution colour information combined with high
resolution luminance information
 Next step: colour video cameras
15
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Videos
16
The University of Auckland | Computer Science | New Zealand
Patternless reconstruction
PRESENTATION
2009
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Marker based expression modelling
3D reconstruction
Static
Dynamic
Active & passive binocular stereo
3D video scanner
Active structured lighting
Active photometric stereo
3D data
Expression modelling
Marker based motion capture
Muscle inverse kinematics
Video based
Sequences from 3D video
scanner
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Marker based expression modelling
Data driven:
 Stereo web-cameras, face
markers.
 Head motion - rigid
 Expressions - non-rigid
Tracked 3D points
 Unique 3D face model
mapping
Virtual muscle animation
 17 active muscles
 Muscle inverse kinematics (IK) –
Jacobian Transpose
19
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Example videos
Happiness – easiest to reproduce
Surprise
20
2009
The University of Auckland | Computer Science | New Zealand
PRESENTATION
Anger – needs teeth!
Disgust – pursing of mouth & closing of eyes not represented
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Video based expression modelling
3D Reconstruction
Static
Dynamic
Active & passive binocular stereo
3D video scanner
Active structured lighting
Active photometric stereo
3D data
Expression modelling
Marker based motion capture
Muscle inverse kinematics
Video based
Sequences from 3D video
scanner
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
3D video based expression
modelling
Image blending
Novel face expressions from
multiple video sequences
Interactive
Low preparation
Not data driven
Dense depth data – cf. marker
system
 Video based → realistic 3D
movement and texture
Reconstruction data directly
used for expression modelling
Sub-region masks
11 Control
control points
points
23
2009
Synthetic expression results
The University of Auckland | Computer Science | New Zealand
PRESENTATION
Sadness: lower face region, anger: right eye region, surprise: left eye region
Happiness: lower face region, surprise: left and right eye regions
24
2009
Synthetic expression results
The University of Auckland | Computer Science | New Zealand
PRESENTATION
Fear: lower face region, happiness: right eye region, anger: left eye region
Disgust: lower face region, anger: left and right eye regions
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Contributions
3D face reconstruction and expression modelling
system
 Unique tool-set
Low-cost, off-the-shelf
Vision based
To 3D face reconstruction:
 Extensive reconstruction comparison
 Face database
 Dynamic reconstruction system for 3D video: SDPS + pattern
To expression modelling:
 Marker based performance capture system
Muscle based IK animation system, unique mapping approach
 Video based expression system – realistic, less flexible
26
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Future work and perspective
Many areas for future research
Refine hardware - better reconstructions ( low-cost? )
Markerless motion capture - face ( feature ) tracking
 Statistical analysis on video data
 Active appearance model (AAM)
New animation system (out of scope)
Full body → complete character
Synergy of computer vision and computer graphics!
Physical models for animation
Computer vision tools
 Especially 3D video & markerless motion capture
The University of Auckland | Computer Science | New Zealand
PRESENTATION
Questions?
2009
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Timeline of Experiments
Ekman - 1987
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Universal expressions
Sadness
Anger
Fear
Disgust
Happiness
Surprise
Recognisable in every culture! Used as exemplar expressions to
judge my results
31
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Types of binocular stereo algorithm
 Local vs global optimisation
 WTA
 SAD, SSD
 Chen-Medioni –
 local method with explicit surface constraints
 Seed propagation approach
 Dynamic programming – 1D optimisation
 SDPS – markov chain
 DPM
 Cubic algorithms – 2D optimisation
 Markov random field
 Energy minimisation
 Graph-cut (KZ1, RoyCox), Belief Propagation,
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Types of photometric stereo algorithm
Experiment focused on integration methods
Assumes C² continuity – i.e. a smooth second derivative
Local optimisation – based on curve integrals
Four path integration
Shapelet
 Explicit summation of basis functions
Global optimisation
FCV – Frankot Chellappa Variant
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Structured lighting techniques
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Body modelling and animation
Body: generic skinned animation
Skeletal hierarchy, fully articulated
• The bones of the hand
• Each bone of the
• The body model with underlying
skeleton
13 April 2015
skeleton has a region of
influence, denoted in green
Department of Computer Science
• Movement of the forearm
35
The University of Auckland | Computer Science | New Zealand
PRESENTATION
Interactive personalised avatar creator
Input photograph
RBF mapping
36
2009
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Results
13 April 2015
Department of Computer Science
37
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Results
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Results
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
3D video based expression system
overview
Acquire sequences of individual expressions using
dynamic 3D face reconstruction system.
 Expression sequences start from a neutral state.
 Test subject’s head remains in the same position for every sequence
 A reference texture and depth map are taken from the neutral
expression and used as the base for all image regions
11 control points are manually annotated on video
sequences.
 Future work to automate this process.
Six sub-regions manually defined on the face.
A sub-region’s texture and depth updated by dragging
a control point residing in it and its currently chosen
expression
sequence.
13 April
2015
Department of Computer Science
40
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
System conclusions
Sinusoidal interpolation instead of a linear one. This
roughly models the biphasic nature of skin
Realistic animations are created as motion is derived
from 3D video sequences of real-life test subjects.
 A user can create unnatural but interesting looking expressions that
can convey a comical feel
Texture maps sourced from video sequences solves
the loss of detail in the marker based approach
 However, apart from the control points that were manually specified,
no points on the face surface are tracked
Results could be refined by improving the quality of 3D
video reconstruction.
13 April 2015
Department of Computer Science
41
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Test subject placement
Subject can be placed with knowledge of required view
area, sensor size, and camera lens:
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Projector synchronisation
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
RBF mapping approach
Radial Basis Functions
User specified point correspondences on generic
model and 3D face data
Specify divergences between data
For each dimension (in 3D)
Find RBF approximation of (1D) displacements within the 3D
space of specified points.
Using this RBF approximation all 3D points from the
generic model can be mapped to the 3D face data
proportions
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Marker tracking
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Rigid and non-rigid motion
Anchor markers:
Rigid orientation:
Remove rigid motion by using transpose of orientation and
centre of gravity of anchors
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Muscle inverse kinematics
Forward kinematics is the calculation of a new position g of
an end effector by specifying updates to parameters of a
kinematic chain
Inverse kinematics is the calculation of parameters for a
kinematic chain to meet a desired goal position g, when
starting from an initial position e.
Kinematic chain consists of joints
Each joint has DOF’s – its animitable parameters,
E.g. 3-DOF for position, 1-DOF for orientation around one axis
(position of joint implied through kinematic chain transformation)
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Jacobian Transpose approach
FK:
IK:
e = current end
effector position
d = change in end effector
position
First order estimate in positional
change:
Change in parameters:
Jacobian Transpose estimate:
g = goal end
effector position
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Estimate assured to move closer to
the goal g:
Always moving in a direction less
than 90 degrees from d
13 April 2015
Department of Computer Science
49
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Interface between Raw Data and
Generic Model
 User specifies a ‘minimal’ set of correspondences between
raw and generic data
 Radial Basis Functions (RBF) used as the interpolant
Model with
animation system
Depth map
Correspondences
made and mapped
via RBF with a final
nearest point map
and texture
projection
Results in a custom
face with animation
system in place
•Feature extraction as a goal
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Face Animation Model
 Research primarily based on Terzopoulos, Waters, Parke collective work in
the field
 Physically based model for skin tissue
 Mass-Spring system
 Epidermal – Fascial – Skull levels of tissue
 Forces are applied to the tissue to simulate muscle contractions
 Springs bring elasticity, allow forces to propagate-> stretches and pulls!
 Abstract muscle definitions
 Decoupled from model
 Warped via RBF also
 Two types
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Face Animation System Forces
Model the behaviour of the tissue
 Reactionary over the evolution of applied muscle forces:
Skull Penetration Constraint
Spring Forces
g j  c j (l j - lj )s j , gi  g j
cj
- biphasic spring constant
l j
- rest length of spring
lj
- current length of spring
s j  (x j  xi ) / l j
- spring direction vector

 fin  n i n i when f i n  n i  0
si  
otherwise
0


Muscle Forces
Applied to fascia nodes based
on the abstract muscle
definitions………..(explained
later)
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Face System Forces
Volume Preservation Force

q  k1 V  V
e
i
e
e
n
e
i

 k2 p  p
e
i
e
i

nei
- epidermal normal for volume element ‘e’
p ei , pie
- current and rest nodal positions with
respect to center of mass of element ‘e’
k1 , k 2
- force scaling constants
These forces allows for tissue form restitution
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Linear Muscle
Linear Muscle:
Applies forces to nodes inside it’s angular range
Influence is weighted by angle and radius from muscle vector
Displacement formula:
p  p  akr
Where
a  1
pv1
pv1
cos( )
cos()
and
D

cos(
1

); for p inside sector( v1p n p m p1 )

R
2

s
r
cos( D  Rs  ); for p inside sector(p np r p sp m )

R f  Rs 2
‘k’ = muscle contraction increment.
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Ellipsoid Muscle
Ellipsoid Muscle:
Acts like a string bag
Application of force weighted by radius only
Defined by major and 2 minor axes
Can generate puckering effects
Displacement formula:
p  p  kr
pv1
pv1
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Physical Simulation
 Layered Tissue Model is a physically based one
 Euler integration is used to run the simulation
Equations of motion

1 e
~t
~ t  ~s t  h
a 
f i   i v ti  ~
git  q
i
i
i
mi
t
i

Acceleration
v ti t  v ti  tati
Velocity
xti t  xti  tvti t
Nodal position
Velocity dependent
damping co-efficient.
Controls the rate of
dissipation of kinetic
energy which eventually
brings the facial mesh to
rest.
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
6 pre-built expressions
Happiness
Sadness
Surprise
Fear
Disgust
Anger
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
General conclusions and future
work (old version)
Investigated low-end and cost effective equipment to
create self-contained tools that can run entirely on any
end user system.
A unique solution has been proposed for 3D face
reconstruction and expression modelling with appropriate
hardware
Synchronised audio capture of speech sequences
would greatly add to the realism.
The attachment of the face model to a body would
complete the system, giving a fully realised virtual
human.
13 April 2015
Department of Computer Science
59
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Conclusions drawn from the static reconstruction
experiment formed the basis of a dynamic 3D face
reconstruction system
3D face reconstructions have no notion of higher order
surface structure and are just a collection of points.
 This structure was addressed in the second part of this thesis which
investigated face expression modelling.
Marker motion was combined with a muscle inverse
kinematics framework to drive the facial animation
system.
 A static face texture impacts on the visual result, as illumination cues
such as wrinkles and shadowing over the face are lost.
13 April 2015
Department of Computer Science
60
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
To supplement the work on 3D faces, a body model
was created
Interactive 3D video expression creation system which
ties together 3D face reconstruction and expression
modelling.
Main problems faced were dealing with hardware
constraints
 But focus on low-cost and off-the-shelf solutions
Focused on the computer vision aspects of facial
reconstruction and expressions as opposed to
computer graphics
13 April 2015
Department of Computer Science
61
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
A combined marker and dense 3D reconstruction
system could be developed, to incorporate further
information for a muscle inverse kinematics system
Highly detailed face animation is best served by taking
advantage of real world data in the form of digital
images and computer vision processing
Advanced physical models of faces meets the tools
and approaches investigated within this thesis
13 April 2015
Department of Computer Science
62
13 April 2015
Department of Computer Science
63
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Details in important areas of the face that are not
currently modelled include the eyelids, lips, teeth and
inner mouth.
Loss of texture detail in the forehead where the
wrinkles are lost
Fine tuning of the preset muscle locations and
parameters when mapping a new face model was
sometimes needed to improve results or correct
muscles
13 April 2015
Department of Computer Science
64
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
System analysis
Capable of reproducing facial expressions from marker
motion.
Low cost hardware, easy retargetting to other models.
Many differences between test-subjects
Expression articulation, muscle control, gross face
movement
Difficulty in performing when no emotional tie involved.
 Easy to understand the need for directors in performance capture
situations.
 Some user direction was needed to describing to a test subject how
an expression should be created
13 April 2015
Department of Computer Science
65
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Issue: potentially multiple solutions for a vertex
position when influenced by multiple muscles
Illumination conditions affect coloured marker
detection
Reflectance properties of the skin surface are
important visual cues.
Missing in this system
Addressed in next chapter.
13 April 2015
Department of Computer Science
66
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Body modelling and animation
system
Skinned animation system was chosen for real-time
capability and ease of creating new body poses
A posable skeleton is associated with a body model (skin
surface description), usually in the form of geometric data
 Forward and inverse kinematics used for animation
 Skin surface under new pose is determined based on skeletal bone
local coordinate systems and blending between adjacent bones.
Future work: combine with the face reconstructions
and animation systems.
13 April 2015
Department of Computer Science
67
2009
Static reconstruction experiment
The University of Auckland | Computer Science | New Zealand
PRESENTATION
 Evaluated three computer vision approaches to 3-D face reconstruction.
 Binocular stereo: passive.
 Structured lighting: active.
 Photometric stereo: active.
 Two main aims:
 Determine their effectiveness for 3D facial reconstruction.
 Accuracy, time complexity.
 Provide a new and alternative test set for evaluating algorithms.
 Database of faces.
 We focus on stereo vision algorithms.
 Integrated lab environment designed.
 12 algorithms tested in total.
 Results compared to ground truth data obtained from a commercial 3D scanner.
 Summary:
 Active illumination techniques are most accurate.
 Stereo algorithm rankings were different from that expected.
 ‘One shot’ active illumination coupled with a traditional stereo algorithm a strong
choice.
13 April 2015
Department of Computer Science
68
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Photogrammetry Laboratory
Optical ‘Range’.
Integrated.
Multiple systems view a
common scene.
 Stereo bench.
 Sideways for face capture!
Example Data:
 Projector for structured
lighting.
 Light sources for
photometric stereo.
 Commercial 3D scanner.
 Solutionix Rexcan 400.
Depth map
13 April 2015
Department of Computer Science
Perspective visualisation
69
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Calibration
System calibration:
Estimates intrinsic and extrinsic
camera parameters
 I.e. camera projection matrices
 For cameras:
A calibration cube - 63 markings
defines a world co-ordinate system
Tsai calibration
 For the lights:
A calibration sphere - estimates
directions to lights
Simple analytic derivation,
inaccurate
 Could also calibrate the projector using
Tsai’s algorithm
13 April 2015
Department of Computer Science
70
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
World to image co-ordinates
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
 Rectification:
 The camera calibration matrices were used to rectify images.
The resultant image pairs meet the epipolar constraint.
 Data processing:
 Data must be compared in a common co-ordinate frame.
 Alignment done using a semi-automatic process involving 3D object rigid
transformations.
 Small number of manual correspondences made.
 Data projected into disparity space.
13 April 2015
Department of Computer Science
72
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Database of 15
people created
Data acquired
from all systems
Rexcan ground
truth
Test-bed for new
algorithms
73
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Binocular Stereo
 Approach 1: Binocular stereo (stereo
vision).
System Geometry (side view)
 Passive.
 Active research area in our department.
 Textureless regions cause problems.
 Remedy via active illumination.
 Test a set of local and global algorithms.
Tested algorithms:
Sum of Absolute Differences (SAD)
 Use two Canon digital SLRs –
6 Mpixels
 1536 x 1024
resolution.
Dynamic Programming Method (DPM)
Symmetric Dynamic Programming Stereo (SDPS)
Graph Cut (GC)
Belief–Propagation (BP)
Chen and Medioni (CM) – seed based algorithm
13 April 2015
Department of Computer Science
74
2009
Structured Lighting
The University of Auckland | Computer Science | New Zealand
PRESENTATION
 Approach 2: Structured Lighting.
System Geometry (side view)
 Active approach. Depth inferred in the
same manner as stereo.
 Augment stereo system with a colour
projector.
 Add structure to scene -> break
homogeneity.
 Projects 800 x 600 pixel image.
Acer PL111 LCD Projector.
6 of the Gray code projections:
 Interested in ‘one shot’ patterns over
Gray code.
Tested algorithms:
Time-multiplexed structured lighting using Gray code
Direct coding - ‘one shot’ colour gradation pattern.
Direct coding - ‘one shot’ colour strip pattern.
13 April 2015
Department of Computer Science
•Add texture to face.
•Used with standard stereo
algorithms.
75
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Photometric Stereo
 Approach 3: Photometric Stereo (PSM).
 Face viewed under 3 different known lighting
conditions.
 Depth by integrating recovered surface orientation
map.
 Albedo independent approach used.
System Geometry (top-down view)
 Three 150W light sources.
 Analysed gradient field integration
techniques.
Tested algorithms:
Frankot-Chellappa Variant (FCV)
Fourier based integration.
Four-Scan method
Local integration paths.
Shapelets
Summation of correlated basis functions.
13 April 2015
Department of Computer Science
76
2009
A Collection of Reconstructions
The University of Auckland | Computer Science | New Zealand
PRESENTATION
 Example depth maps:
Ground
truth
Gray
code
FCV
SAD
Structured lighting
SDPS
GC
CM
Binocular Stereo
Photometric Stereo
13 April 2015
Department of Computer Science
77
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Photometric Stereo Results
Reconstruction accuracy:
 17 test subjects.
Percentage of errors less than 2 disparity units
Method
P <=2,%
69
54
71
97
Gold standard result for accuracy
13 April 2015
Department of Computer Science
78
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Passive Stereo Results
Reconstruction accuracy:
Method
P<=2,%
89
79
77
GC
80
73
88
97
13 April 2015
Department of Computer Science
79
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Stereo + Gradation Pattern
Reconstruction accuracy:
Method
P<=2,%
90
84
83
GC
85
77
89
97
13 April 2015
Department of Computer Science
80
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Stereo + Strip Pattern
Reconstruction accuracy:
Method
P<=2,%
93
92
92
GC
93
89
92
97
13 April 2015
Department of Computer Science
81
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Improvement to Stereo from Active
Illumination
Addition of the Strip colour pattern.
SAD stereo algorithm:
Strip pattern
Depth map
SAD - without
pattern
SAD - with
strip pattern
P<=2 = 80%
P<=2 = 93%
Pattern colour should avoid skin tones.
13 April 2015
Department of Computer Science
82
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
Error map
example
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Gray code approach most accurate.
Slower acquisition time.
Look to alternative ‘one shot’ approaches.
Photometric stereo least accurate.
Our test set has high resolution images and large
disparity ranges.
O(n3) stereo algorithms – GC, BP – inappropriate.
 Long processing time.
Parameter setting difficult.
Our results differ from the Middlebury rankings:
http://cat.middlebury.edu/stereo/
13 April 2015
Department of Computer Science
84
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
All results contain errors.
 Need post processing to clean up data.
 Even for the commercial 3D scanner.
Faces have many unique properties posing a challenge
for 3D reconstruction
Human sensitivity to errors in reconstruction - we see faces
all the time.
For computer vision:




13 April 2015
Specularities.
Anistropic reflectance of hair.
Sub-surface scattering.
Large homogenous regions.
Department of Computer Science
85
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
Static reconstruction conclusion
and analysis
 Framework and test-bench for active and passive 3-D
acquisition systems designed.
 Three computer vision approaches tested.
 12 algorithms altogether.
 Analysed accuracy of algorithms for 3D face reconstruction.
 Data compared to scanner benchmark.
 Provided new alternative test set to Middlebury for testing
stereo algorithms
 High resolution images of faces.
 Passive stereo combined with active illumination a promising
approach.
 Want a one shot approach for faces (moving object).
 SDPS + Strip pattern.
 Leads to real-time spatio-temporal acquisition.
 Acquire 3D face performance.
13 April 2015
Department of Computer Science
86
13 April 2015
Department of Computer Science
87
The University of Auckland | Computer Science | New Zealand
PRESENTATION
2009
2009
PRESENTATION
The University of Auckland | Computer Science | New Zealand
A generic face model with an abstract muscle animation
system was designed during my Master’s thesis.
 Refined for PhD thesis.
Can be personalised with 3D data and texture information from the
static reconstruction experiment using a custom RBF mapping
procedure.
• Example of muscle contraction
• Generic morphable face with linear
and ellipsoid muscles
13 April 2015
Department of Computer Science
• A biomechanical tissue model
88