Humans:
The Final Frontier
source: http://www.gimartex.es/myfiles/Ballet-dancer_01.jpg
Adrien Treuille
Overview
• State of the art.
• Body models.
• Animation
• Vote.
• Questions
source: 3dscience.com
Overview
• State of the art.
• Body models.
• Animation
• Vote.
• Questions
source: 3dscience.com
•
State of the Art
Steady but slow progress
towards digital humans.
•
State of the Art
Steady but slow progress
towards digital humans.
•
(As usual, rendering ahead
of animation.)
•
State of the Art
Steady but slow progress
towards digital humans.
•
•
(As usual, rendering ahead
of animation.)
State of the art for animation
production.
•
State of the Art
Steady but slow progress
towards digital humans.
•
(As usual, rendering ahead
of animation.)
•
State of the art for animation
production.
•
State of the art for games.
•
State of the Art
Steady but slow progress
towards digital humans.
•
(As usual, rendering ahead
of animation.)
•
State of the art for animation
production.
•
•
State of the art for games.
Uncanny Valley.
•
State of the Art
Steady but slow progress
towards digital humans.
•
(As usual, rendering ahead
of animation.)
•
State of the art for animation
production.
•
•
•
State of the art for games.
Uncanny Valley.
Facial Animation.
•
State of the Art
Steady but slow progress
towards digital humans.
•
(As usual, rendering ahead
of animation.)
•
State of the art for animation
production.
•
•
•
•
State of the art for games.
Uncanny Valley.
Facial Animation.
Most human animation is
data driven.
(Like saying that
graphics is solved
by the camera.)
Overview
• State of the art.
• Body models.
• Animation
• Vote.
• Questions
source: 3dscience.com
Overview
• State of the art.
• Body models.
• Animation
• Vote.
• Questions
source: 3dscience.com
Body Representation
source: http://www.3eop.com/data/3d/images/08_05_26_anatomy_study_male.jpg
How to represent a human body on a computer?
Body Representation
• Kinematic Skeleton
source: https://buffy.eecs.berkeley.edu/PHP/resabs/resabs.php?
f_year=2005&f_submit=advgrp&f_advid=10917651
Body Representation
• Kinematic Skeleton
• Anatomical
source: http://physbam.stanford.edu/~fedkiw/
To appear in SIGGRAPH 2005.
Body Representation
A
B
C
Figure 5: Using M ESH IK to pose a bar: (A) Two example poses superimposed o
constrained to stay in place while a single vertex on the right side is manipulated.
Note that M ESH IK generalizes beyond the two examples and can create arbitrary b
A
B
C
D generalizes poorly. (D)
E In this top down view,F moving the constra
interpolates and
no examples were provided in this direction. (E)–(F) Providing one additional ex
generalize
to bends
that direction
as The
well left
as incap
the space
between.
Figure 5: Using M ESH IK to pose a bar: (A) Two example poses superimposed on
top ofineach
other. (B)
of theinunbent
bar is
• Kinematic Skeleton
• Anatomical
• Pure Mesh
constrained to stay in place while a single vertex on the right side is manipulated. Three edits using our nonlinear feature space are shown.
Note that M ESH IK generalizes beyond the two examples and can create arbitrary bends in the plane. (C) In contrast, the linear feature space
interpolates and generalizes poorly. (D) In this top down view, moving the constrained vertex perpendicular to the bend causes a shear since
no examples were provided in this direction. (E)–(F) Providing one additional example in the perpendicular direction allows M ESH IK to
generalize to bends in that direction as well as in the space in between.
A
B
ç
source: http://physbam.stanford.edu/~fedkiw/
A
B
Figure 6: Top row: Ten lion example poses. Bottom row: A sequence of posing
front leg is pulled forward and the lion continuously deforms as the constraint is
the front leg can be edited in isolation. (D) A similar operation is performed to adj
example.
ç
C
D
Figure 6: Top row: Ten lion example poses. Bottom row: A sequence of posing operations. (A) Two handle vertices are chosen. (B) The
source:(C)
http://people.csail.mit.edu/sumner/research/meshik/
front leg is pulled forward and the lion continuously deforms as the constraint is moved.
The red region is selected and frozen so that
the front leg can be edited in isolation. (D) A similar operation is performed to adjust the tail. The final pose is different from any individual
example.
A
B
C
D
Figure 7: Posing simulated flag. Top row: Fourteen examples of a flag blowin
To appear in SIGGRAPH 2005.
Body Representation
A
B
C
D
Figure 5: Using M ESH IK to pose a bar: (A) Two example poses superimposed on top
constrained to stay in place while a single vertex on the right side is manipulated. Thre
Note that M ESH IK generalizes beyond the two examples and can create arbitrary bends
A
B
C
D generalizes poorly. (D)
E In this top down view,F moving the constrained v
interpolates and
no examples were provided in this direction. (E)–(F) Providing one additional exampl
generalizeon
to bends
that direction
as The
well left
as incap
the space
between.
Figure 5: Using M ESH IK to pose a bar: (A) Two example poses superimposed
top ofineach
other. (B)
of theinunbent
bar is
• Kinematic Skeleton
• Anatomical
• Pure Mesh
• What are the
constrained to stay in place while a single vertex on the right side is manipulated. Three edits using our nonlinear feature space are shown.
Note that M ESH IK generalizes beyond the two examples and can create arbitrary bends in the plane. (C) In contrast, the linear feature space
interpolates and generalizes poorly. (D) In this top down view, moving the constrained vertex perpendicular to the bend causes a shear since
no examples were provided in this direction. (E)–(F) Providing one additional example in the perpendicular direction allows M ESH IK to
generalize to bends in that direction as well as in the space in between.
advantages
and
A
disadvantages?
B
ç
A
B
C
Figure 6: Top row: Ten lion example poses. Bottom row: A sequence of posing opera
front leg is pulled forward and the lion continuously deforms as the constraint is move
the front leg can be edited in isolation. (D) A similar operation is performed to adjust th
example.
ç
C
D
Figure 6: Top row: Ten lion example poses. Bottom row: A sequence of posing operations. (A) Two handle vertices are chosen. (B) The
front leg is pulled forward and the lion continuously deforms as the constraint is moved. (C) The red region is selected and frozen so that
the front leg can be edited in isolation. (D) A similar operation is performed to adjust the tail. The final pose is different from any individual
example.
A
B
C
D
To appear in SIGGRAPH 2005.
Body Representation
A
B
C
D
Figure 5: Using M ESH IK to pose a bar: (A) Two example poses superimposed on top
constrained to stay in place while a single vertex on the right side is manipulated. Thre
Note that M ESH IK generalizes beyond the two examples and can create arbitrary bends
A
B
C
D generalizes poorly. (D)
E In this top down view,F moving the constrained v
interpolates and
no examples were provided in this direction. (E)–(F) Providing one additional exampl
generalizeon
to bends
that direction
as The
well left
as incap
the space
between.
Figure 5: Using M ESH IK to pose a bar: (A) Two example poses superimposed
top ofineach
other. (B)
of theinunbent
bar is
• Kinematic Skeleton
• Anatomical
• Pure Mesh
• What are the
constrained to stay in place while a single vertex on the right side is manipulated. Three edits using our nonlinear feature space are shown.
Note that M ESH IK generalizes beyond the two examples and can create arbitrary bends in the plane. (C) In contrast, the linear feature space
interpolates and generalizes poorly. (D) In this top down view, moving the constrained vertex perpendicular to the bend causes a shear since
no examples were provided in this direction. (E)–(F) Providing one additional example in the perpendicular direction allows M ESH IK to
generalize to bends in that direction as well as in the space in between.
advantages
and
A
disadvantages?
B
ç
A
B
C
Figure 6: Top row: Ten lion example poses. Bottom row: A sequence of posing opera
front leg is pulled forward and the lion continuously deforms as the constraint is move
the front leg can be edited in isolation. (D) A similar operation is performed to adjust th
example.
ç
C
D
Figure 6: Top row: Ten lion example poses. Bottom row: A sequence of posing operations. (A) Two handle vertices are chosen. (B) The
front leg is pulled forward and the lion continuously deforms as the constraint is moved. (C) The red region is selected and frozen so that
the front leg can be edited in isolation. (D) A similar operation is performed to adjust the tail. The final pose is different from any individual
example.
A
B
C
D
Skeleton Representation
f2 f3
f1
f4
f5
ω5
root: ω0
Ω is the vector of
internal joint
angles, i.e.
shoulders, hips,
etc.
ω0 = [x0 , θ0 ] ∈ R
6
ωi = fi,Ω (ωi−1 )
Motion Capture
• Attach markers to a humans body.
• Calibrate a skeleton which makes those
markers “make sense.”
• Cameras capture 2D markers positions.
• Estimate 3D marker positions.
• Inverse kinematics: convert marker
positions to skeleton...
• How?
Marker Energy Function
ωi = fi,Ω (ωi−1 )
m̂j = τi (ωi )mj
E=
!
j
!
||m̂j
− m̂j ||
2
dE
dΩ
Derivatives
derivative
calculation
goes
towards
root
ωi = fi,Ω (ωi−1 )
!
dE
!
T dm̂j
=2
(m̂j − m̂j )
dΩ
dΩ
j
vector matrix
"
!
dm̂j
∂ m̂j ∂ωi
∂ωi ∂ωi−1
∂ωi ∂ωi−1 ∂ωi−2
=
+
+
+ ···
dΩ
∂ωi
∂Ω
∂ωi−1 ∂Ω
∂ωi−1 ∂ωi−2 ∂Ω
matrix
matrix multiplies
Inverse Kinematics Summary
• Telescoping composition of functions
from root.
• Compute derivatives in the opposite
direction!
To appear in SIGGRAPH 2005.
Body Representation
A
B
C
D
Figure 5: Using M ESH IK to pose a bar: (A) Two example poses superimposed on top
constrained to stay in place while a single vertex on the right side is manipulated. Thre
Note that M ESH IK generalizes beyond the two examples and can create arbitrary bends
A
B
C
D generalizes poorly. (D)
E In this top down view,F moving the constrained v
interpolates and
no examples were provided in this direction. (E)–(F) Providing one additional exampl
generalizeon
to bends
that direction
as The
well left
as incap
the space
between.
Figure 5: Using M ESH IK to pose a bar: (A) Two example poses superimposed
top ofineach
other. (B)
of theinunbent
bar is
• Kinematic Skeleton
• Anatomical
• Pure Mesh
• What are the
constrained to stay in place while a single vertex on the right side is manipulated. Three edits using our nonlinear feature space are shown.
Note that M ESH IK generalizes beyond the two examples and can create arbitrary bends in the plane. (C) In contrast, the linear feature space
interpolates and generalizes poorly. (D) In this top down view, moving the constrained vertex perpendicular to the bend causes a shear since
no examples were provided in this direction. (E)–(F) Providing one additional example in the perpendicular direction allows M ESH IK to
generalize to bends in that direction as well as in the space in between.
advantages
and
A
disadvantages?
B
ç
A
B
C
Figure 6: Top row: Ten lion example poses. Bottom row: A sequence of posing opera
front leg is pulled forward and the lion continuously deforms as the constraint is move
the front leg can be edited in isolation. (D) A similar operation is performed to adjust th
example.
ç
C
D
Figure 6: Top row: Ten lion example poses. Bottom row: A sequence of posing operations. (A) Two handle vertices are chosen. (B) The
front leg is pulled forward and the lion continuously deforms as the constraint is moved. (C) The red region is selected and frozen so that
the front leg can be edited in isolation. (D) A similar operation is performed to adjust the tail. The final pose is different from any individual
example.
A
B
C
D
To appear in SIGGRAPH 2005.
Body Representation
A
B
C
D
Figure 5: Using M ESH IK to pose a bar: (A) Two example poses superimposed on top
constrained to stay in place while a single vertex on the right side is manipulated. Thre
Note that M ESH IK generalizes beyond the two examples and can create arbitrary bends
A
B
C
D generalizes poorly. (D)
E In this top down view,F moving the constrained v
interpolates and
no examples were provided in this direction. (E)–(F) Providing one additional exampl
generalizeon
to bends
that direction
as The
well left
as incap
the space
between.
Figure 5: Using M ESH IK to pose a bar: (A) Two example poses superimposed
top ofineach
other. (B)
of theinunbent
bar is
• Kinematic Skeleton
• Anatomical
• Pure Mesh
• What are the
constrained to stay in place while a single vertex on the right side is manipulated. Three edits using our nonlinear feature space are shown.
Note that M ESH IK generalizes beyond the two examples and can create arbitrary bends in the plane. (C) In contrast, the linear feature space
interpolates and generalizes poorly. (D) In this top down view, moving the constrained vertex perpendicular to the bend causes a shear since
no examples were provided in this direction. (E)–(F) Providing one additional example in the perpendicular direction allows M ESH IK to
generalize to bends in that direction as well as in the space in between.
advantages
and
A
disadvantages?
B
ç
A
B
C
Figure 6: Top row: Ten lion example poses. Bottom row: A sequence of posing opera
front leg is pulled forward and the lion continuously deforms as the constraint is move
the front leg can be edited in isolation. (D) A similar operation is performed to adjust th
example.
ç
C
D
Figure 6: Top row: Ten lion example poses. Bottom row: A sequence of posing operations. (A) Two handle vertices are chosen. (B) The
front leg is pulled forward and the lion continuously deforms as the constraint is moved. (C) The red region is selected and frozen so that
the front leg can be edited in isolation. (D) A similar operation is performed to adjust the tail. The final pose is different from any individual
example.
A
B
C
D
xi,4
0
xi,3
0
xi,2
(a)
(b)
Dense Marker Capture
(b)
Figure 3: (a) a marker (red dot) and its one-ring neighbors (blue
dots); (b) a local reference frame on the marker.
n0i
p: (a) Twelve cameras surrounding a small
meras (shown in red) were aimed up rather
downward facing markers; (b) 350 small
subject’s body.
Motion Systems 2006]. To increase accucameras close to the subject with a small
imately 2 m by 2 m by 2.5 m high). Two
rather than down to capture markers facing
(a)
(b)
ure 2(a) shows the camera configuration.
t 0i
x0i
0
bi
0
xi,6
0
xi,1
0
xi,5
left arm
0
xi,3
0
xi,2
(a)
(a)
0
xi,4
(b)
Figure 3: (a) a marker (red dot) and its one-ring neighbors (blue
dots); (b) a local
(b)reference frame on the(c)marker.
ely
350 reflective
markers
the subject’s
: Capture
setup: (a)
Twelveoncameras
surrounding Figure
a small4: The reference pose: (a) markers; (b) marker surface; (c)
ubtle
of the
skin effectively,
frame at each marker.
egion.movement
Two cameras
(shown
in red) werewe
aimed uplocal
rather
ameter
3.0 mm)downward
with a hemispherical
shape(b) 350 small
wn
to capture
facing markers;
of the markers
from the
body. Although
attached
to the subject’s
body.
ave an even distribution of the markers on
a mesh from the marker positions in the reference pose. Neighborpproximate grid on the subject’s body, and
ing markers are identified based on the geodesic distance along the
hat grid.
The average
twoincreaseskin.
This process was done manually for our experiments. We call
econd
[Vicon
Motion distance
Systems between
2006]. To
accuas
4-5 cm. the
We cameras
supplemented
thethe
grid
pat- with the
resulting mesh the marker surface (Figure 4(a) and (b)). The
positioned
close to
subject
a small
rkers in(approximately
areas where more
indices
region
2 mresolution
by 2 m bywould
2.5 m high).
Two of the vertices on the surface are assigned to the markers as
of theaimed
elbowupand
the point
on thetolower
partmarkerslabels.
is done only once for
were
rather
than down
capture
facingThe construction(a)of the marker surface
(b)
(c)
or ground.
example).
Figure
shows
marker
each subject.
he
Figure
2(a)2(b)
shows
the the
camera
configuration.
left arm
On the markerFigure
surface,
we denote
thepose:
marker
index(b)
i as
mi , surface; (c)
ed approximately 350 reflective markers on the subject’s
4: The
reference
(a)with
markers;
marker
positions
the markers
with of
thethe
VICON
1 ≤ we
i ≤ N, where
is the
number
of markers, and its one-ring
oD capture
theofsubtle
movement
skin effectively,
local N
frame
at each
marker.
Motion
Systems
2006].3.0
In mm)
conventional
moneighbor
mall
markers
(diameter
with a hemispherical
shape as mi,k , 1 ≤ k ≤ di , where di is the valence of marker i.
tion is
by of
a skeleton
modelfrom
and the
a rigid
mize
theaided
offset
the markers
body. Although
We denote the position of the i-th marker by x0i and the position of
ver,
becausetoofhave
the an
significant
deformations
necessary
even distribution
of the markers
on
a mesh from
theinmarker
positions
in the
reference
pose. Neighborits one-ring
neighbors
as x0i,k
the reference
pose.
Using
a similar
notdrew
make
assumption.
assump,ldwe
anthis
approximate
gridThis
on the
subject’s body,
and to that
ing of
markers
areand
identified
based on[2005],
the geodesic
distance along the
technique
Lipman
his
colleagues
we
define
r motion
the The
face,average
but ourdistance
whole between two
he
markerscapture
on thatofgrid.
skin.
process
done manually
We call
the local frame
for This
marker
i in was
the reference
pose for
withour
itsexperiments.
origin
Laser Range Scanning
Filling in Missing Data
(c)
(d)
source: Allen, Curless, Popović. The space of human body shapes: reconstruction and
parameterization from range scans.
How could this be accomplished?
What can you do with a
huge set of human meshes
in vertex correspondence?
source: Allen, Curless, Popović. The space of human body shapes: reconstruction and parameterization from range scans.
PCA Shape Analysis
Displacements of
Example Poses
=
PCA
Basis
Coefficient Matrix
source: Anguelov, Srinivasan, Koller, Thrun, Rodgers, Davis. SCAPE: Shape Completion and Animation of People.
Multilinear Analysis
To appear in SIGGRAPH 2005.
mode 3
identity
d3
d2
d1
mode 2
expression
T
mode 1
(a)
(b)
(c)
(d)
vertices
Figure 2: In (a) we show a 3rd -order (3-mode) tensor T whose
modes have d1 , d2 , and d3 elements respectively. Depending on
Figure 3: Data tensor for a bilinear model that varies with idenhow we look at the data within the tensor, we can identify three
tity and expression; the first mode contains vertices, while the secVlasic, Brand, Pfister, Popović. Face Transfer with Multilinear Models.
mode spaces. By viewing the data assource:
vectors
parallel to the first
ond and third modes correspond to expression and identity respecmode (b), we define mode-1 space as the span of those vectors.
tively. The data is arranged so that each slice along the second mode
Similarly, mode-2 space is spanned by vectors parallel to the second
contains the same expression (in different identities) and each slice
mode (c), and mode-3 space by vectors in the third mode (d).
along the third mode contains the same identity (in different expressions). In our trilinear experiments we have added a fourth mode,
where scans in each slice share the same viseme.
Mode-n Product. The most obvious way of manipulating mode
spaces is via linear transformation, officially referred to as the
4.1 Face Data
mode-n product. It is defined between a tensor T and a matrix
M for a specific mode n, and is written as a multiplication with a
We demonstrate our proof-of-concept system on two separate face
subscript: T ×n M. This notation indicates a linear transformamodels: a bilinear model, and a trilinear model. Both were estion of vectors in T ’s mode-n space by the matrix M. Concretely,
timated from detailed 3D scans (∼ 30K vertices) acquired with
T ×2 M would replace each mode-2 vector v (Figure 2c) with a
3dMD/3Q’s structured light scanner (http://www.3dmd.com/) in
transformed vector Mv.
a process similar to regular flash photography, although our methods would apply equally to other geometric data sets such as motion
Tensor Decomposition. One particularly useful linear transforcapture. As a preprocess, the scans were smoothed using the bilatmation of mode data is the N-mode singular value decomposition
eral filter [Jones et al. 2003] to eliminate some of the capture noise.
(N-mode SVD). It rotates the mode spaces of a data tensor T proThe subject pool included men, women, Caucasians, and Asians,
ducing a core tensor C , whose variance monotonically decreases
from the mid-20s to mid-50s.
from first to last element in each mode (analogous to matrix SVD).
This enables us to truncate the insignificant components and get a
Bilinear model. 15 subjects were scanned performing the same
reduced model of our data.
source: Anguelov, Srinivasan, Koller, Thrun, Rodgers, Davis. SCAPE:
Shape expressions.
Completion and Animation
of People.
10 facial
The expressions
were picked for their familMathematically, N-mode SVD can be expressed with mode
Example
source: Anguelov, Srinivasan, Koller, Thrun, Rodgers, Davis. SCAPE: Shape Completion and Animation of People.
Overview
• State of the art.
• Body models.
• Animation
• Vote.
• Questions
source: 3dscience.com
Overview
• State of the art.
• Body models.
• Animation
• Vote.
• Questions
source: 3dscience.com
Data Driven Human
Animation
source: Kovar, Gleicher, Pighin. Motion Graphs.
To appear in Proceedings of SIGGRAPH ’02
et al. 2001], which (like motion graphs) are
enting connections in a database of motion.
e created manually — short motion clips are
cripted capture sessions and blends are creractive tools. Motion graphs are constructed
ove trees are typically geared for rudimen“I want to turn left, so I should follow this
d to more complicated objectives.
sitions is an important part of our approach.
was done by Perlin [1995], who presented a
othly interpolating between two clips to credefined orientation filters that allowed these
be performed on rotational data in a more
se et al. [1996] presented a more complex
nsitions that preserved kinematic constraints
perties.
of motion graphs is to control a character’s
blem is important enough to have received
attention. Because a character’s path isn’t
vance, synthesis is required. Procedural and
esis methods have been developed for a few
ing [Multon et al. 1999; Sun and Metaxas
odgins et al. 1995; Bruderlin and Calvert
es such as these can generate flexible motion
e of movement styles is limited. Also, these
ce the quality of motion attainable by hand
apture. While Gleicher [2001] presented a
path traversed in a clip of motion capture,
need for continuous streams of motion, nor
Figure 2: Consider a motion graph built from two initial clips. (top) We can trivially
insert a node to divide an initial clip into two smaller clips. (bottom) We can also insert
a transition joining either two different initial clips or different parts of the same initial
clip.
Motion Graph Schematic
also allow clips (or, more generally, sets of frames) to be annotated with other information, such as descriptive labels (“walking,”
“karate”) and constraint information (left heel must be planted on
these frames).
A motion graph is a directed graph where all edges correspond to
clips of motion. Nodes serve as choice points connecting these
clips, i.e., each outgoing edge is potentially the successor to any
incoming edge. A trivial motion graph can be created by placing
all the initial clips from the database as arcs in the graph. This cre-
Finding Candidate
Transitions
Examples
Overview
• State of the art.
• Body models.
• Animation
• Vote.
• Questions
source: 3dscience.com
Overview
• State of the art.
• Body models.
• Animation
• Vote.
• Questions
source: 3dscience.com
Vote
• We have only two lectures left!
• Possible topics:
Reduction / Real-time
• Model
Simulation - 10
human
• Physics-based
animation. - 10
• Animal Motion / Morphology
• Optimization Control - 5
• Anything else?
Overview
• State of the art.
• Body models.
• Animation
• Vote.
• Questions
source: 3dscience.com
Overview
• State of the art.
• Body models.
• Animation
• Vote.
• Questions
source: 3dscience.com
Questions
• How do we fix the foot skate problem?
• How can we generalize away from
existing motion capture data?
• How could we search for motion clips?
• How could we motion capture wild
animals?
• How could we go from “motion capture”
to “physics capture?”