Three-dimensional Proxies for
Hand-drawn Characters
1
1
2
1,2
Eakta Jain Yaser Sheikh Moshe Mahler Jessica Hodgins
1
Carnegie Mellon University
2
Disney Research Pittsburgh
1
Monday, September 3, 2012
Hand-drawn animation
3D CG animation
2
Monday, September 3, 2012
Hand-drawn animation
3D CG animation
2
Monday, September 3, 2012
Differences between hand animation
and computer animation
Hand animator’s workdesk
3
Monday, September 3, 2012
Differences between hand animation
and computer animation
3D animation software
Hand animator’s workdesk
3
Monday, September 3, 2012
Input
4
Monday, September 3, 2012
Input
4
Monday, September 3, 2012
5
Monday, September 3, 2012
5
Monday, September 3, 2012
Hand-drawn character
3D proxy
6
Monday, September 3, 2012
Geometry
Motion
Hand-drawn character
3D proxy
6
Monday, September 3, 2012
Challenge: Inferring third dimension
Possible 3D points
2D points
Camera
center
Image
plane
7
Monday, September 3, 2012
Challenge: Composite motion ambiguity
8
Monday, September 3, 2012
Challenge: Composite motion ambiguity
8
Monday, September 3, 2012
Challenge: Composite motion ambiguity
Character turns,
camera in place
8
Monday, September 3, 2012
Challenge: Composite motion ambiguity
Character turns,
camera in place
Camera turns,
character in place
8
Monday, September 3, 2012
Challenge: Artistic license
Frame #1
Frame #40
9
Monday, September 3, 2012
Challenge: Artistic license
Frame #1
Frame #40
9
Monday, September 3, 2012
Challenge: Artistic license
Frame #1
Frame #40
9
Monday, September 3, 2012
Challenge: Artistic license
Frame #1
Frame #40
9
Monday, September 3, 2012
Overlaid
Challenge: Artistic license
Frame #1
Frame #40
9
Monday, September 3, 2012
Overlaid
Three-dimensional proxies with
different levels of detail
Single points
(3D markers)
3D Polygonal shapes
10
Monday, September 3, 2012
Joint hierarchy
based skeleton
Three-dimensional proxies with
different levels of detail
Single points
(3D markers)
3D Polygonal shapes
10
Monday, September 3, 2012
Joint hierarchy
based skeleton
Past work
(e)(b)
Warped
model
Final in
frame
3-D model
(c)(f)Edges
model
art
(e) Warped model
et
al. Li et al.(f) Final fram
Petrovic
et al.(d) Edges inDavis
Correa et
al.
ating one texture mapped
frame.
Figure 2: The process
of creating one texture mapped
frame.
(2003)
(2000)
(1998)
(2003)
(d)Hand-drawn
Edges in artart
(a)
Figure 14: Motion of walking cartoon character retargeted to 3
model
Figure
Tracking
Baloo’s dance
and retargeting
a flower. used later as the input to the warp. Sectio
edges
can be
usedexisting
later asproduction
the input topipeline
the warp.
Section
Our
method
fitssafely
into the
for11:
cel
ani-of4.4
edges
can beto safely
c
�Disney
shows[6,
how
to specify
theand
edges
2-D drawing
that correspond
mation
18].
Steps (a)
(f) on
arethe
stages
in the current
digital
shows how to specify the edges on the 2-D drawing that corres
to the edges
foundwith
on the
model. stage between them. We
production
process,
the 3-D
ink-and-paint
to the edges found on the 3-D model.
are offering, as an alternative to the constant colors of the ink-andpaint stage, a process that applies complex textures to the drawings.
4.1
Silhouette
Detection
4.1 Silhouette Detection
The problem
of applying
textures to hand-drawn artwork poses
a challenge:
the line
must be
interpreted
somevisible
kind ofsilhouette
shape.
Figure a
15:scheme
Motion of jumping
character retargeted
to 3Dsilho
mode
In this section
we art
describe
a scheme
for as
finding
In this section we describe
for finding
visible
12: acquire
Broom Retargeting Sequence. (The source is not shown
Given
set of black
paper,
the computer
must
and aborder
edges lines
in a on
3-Dwhite
model
represented
by aFigure
polygon
mesh.
and
edges inin a 3-D model represented by a polygon m
here
– It is the famous
sequence from
the border
Sorcerer’s Apprentice
9, we show some of the key poses.
Disney’s
at These
least a features
primitiveare
model
3-D formstoconveyed
by
the handart.
likelyfortothe
correspond
features
in Fantasia)
the
These features are likely
to correspond to features in the
Figure 14 shows some final rendered frames in the retarget
This
information
necessary
if we are to
3-D effects
forthe
sequence. Again, please consult
for more information.
drawn
line art; is
such
correspondences
areprovide
the primary
input to
drawn line art; such correspondences
are the
thevideo
primary
input
In
the
next
example,
we
capture
a
cartoon
character jumping in
single
line
running
through
the
character,
that
represents
the
overthewarp
texture
such
as
self-occlusion
and
foreshortening.
(See,
for
we describe in Section 5. We also allow thealluser
to specify
warp we describe in Section
We for
also
allow
theanduser
way that is 5.
impossible
an average
human,
retargetto
ontosp
t
force and direction of each drawing. Generally, before drawing
same
3D
model.
Again,
contour
capture
is
used
to
estimate
the
m
example,
the difference
in occlusion
betweenonFigures
and
2ebut
oranimator
the2b
full
character,
an
in the line-of-action
helpdrawing them directly on the 3-D model,
model markers
by drawing
them directly
the 3-D
model,
it draws
model
markerstoby
tion parameters. Figure 15 shows some frames from the retarget
determine
the position of the character. By simply changing the
thewould
foreshortening
shown to
in have
Figure
Note all
thatmodel
withline-of-action
the
constant
11
sequence.
be cumbersome
to7.)
specify
marker
curves
would
be
cumbersome
to have to specify all model marker c
– making it more curved, sloped or arched in a difThe video shows all results for the different target media. Our e
ferenteffects
direction – the
essence of the drawing can be changed.
colors
of theThus,
traditional
ink-and-paintconstruct
stage, these
3-D
areentire
this
way.
we
automatically
model
markers
for
all
this
way.
Thus,
we
automatically
markers
f
periments
in cartoonconstruct
capture providemodel
an opportunity
to test the ca
Monday, September 3, 2012
We experiment with using the line-of-action as the source in order
Johnston
(2002)
Bregler et al.
(2002)
User Input
Virtual
markers
Limb
bounding
boxes
Color coded
body parts
12
Monday, September 3, 2012
User Input
Virtual
markers
Limb
bounding
boxes
Color coded Motion capture segment with
body parts similar depth information,
time-warped via Dynamic
Time Warping
12
Monday, September 3, 2012
Camera Estimation
3D points
Motion capture poses
Estimated camera
Image plane
Hand drawings
Monday, September 3, 2012
3D points
Back-projected rays
2D points
z-depth
Image plane
Estimated camera
14
Monday, September 3, 2012
arg min(ea (x) + em (x) + es (x))
x
3D points
Back-projected rays
2D points
z-depth
Image plane
Estimated camera
14
Monday, September 3, 2012
arg min(ea (x) + em (x) + es (x))
x
input-matching
3D points
Back-projected rays
2D points
z-depth
Image plane
Estimated camera
14
Monday, September 3, 2012
arg min(ea (x) + em (x) + es (x))
x
input-matching
depth prior
3D points
Back-projected rays
2D points
z-depth
Image plane
Estimated camera
14
Monday, September 3, 2012
arg min(ea (x) + em (x) + es (x))
x
input-matching
depth prior
smoothing term
3D points
Back-projected rays
2D points
z-depth
Image plane
Estimated camera
14
Monday, September 3, 2012
arg min(ea (x) + em (x) + es (x))
x
input-matching
depth prior
smoothing term
Linear system: least squares solution
3D points
Back-projected rays
2D points
z-depth
Image plane
Estimated camera
14
Monday, September 3, 2012
arg min(ea (x) + em (x) + es (x))
x
input-matching
depth prior
smoothing term
Linear system: least squares solution
3D points
Back-projected rays
2D points
z-depth
Image plane
Estimated camera
14
Monday, September 3, 2012
arg min(ea (x) + em (x) + es (x))
x
input-matching
depth prior
smoothing term
Linear system: least squares solution
3D points
Back-projected rays
2D points
z-depth
Image plane
Estimated camera
14
Monday, September 3, 2012
arg min(ea (x) + em (x) + es (x))
x
input-matching
depth prior
smoothing term
Linear system: least squares solution
3D points
Back-projected rays
2D points
z-depth
Image plane
Estimated camera
14
Monday, September 3, 2012
arg min(ea (x) + em (x) + es (x))
x
input-matching
depth prior
smoothing term
Linear system: least squares solution
3D points
Back-projected rays
2D points
z-depth
Image plane
Estimated camera
14
Monday, September 3, 2012
arg min(ea (x) + em (x) + es (x))
x
input-matching
depth prior
smoothing term
Linear system: least squares solution
3D points
Back-projected rays
2D points
z-depth
Image plane
Estimated camera
14
Monday, September 3, 2012
arg min(ea (x) + em (x) + es (x))
x
input-matching
depth prior
smoothing term
Linear system: least squares solution
3D points
Back-projected rays
2D points
z-depth
Image plane
Estimated camera
14
Monday, September 3, 2012
Depth compositing
Rendered image
15
Monday, September 3, 2012
Depth compositing
Rendered image
Depth map for
rendered image
15
Monday, September 3, 2012
Depth compositing
Rendered image
Depth map for
rendered image
Depth map for
hand drawing
15
Monday, September 3, 2012
Depth compositing
Rendered image
Depth map for
rendered image
Depth map for
hand drawing
Composited frame
15
Monday, September 3, 2012
16
Monday, September 3, 2012
16
Monday, September 3, 2012
17
Monday, September 3, 2012
17
Monday, September 3, 2012
18
Monday, September 3, 2012
18
Monday, September 3, 2012
19
Monday, September 3, 2012
20
Monday, September 3, 2012
21
Monday, September 3, 2012
22
Monday, September 3, 2012
22
Monday, September 3, 2012
23
Monday, September 3, 2012
23
Monday, September 3, 2012
24
Monday, September 3, 2012
24
Monday, September 3, 2012
25
Monday, September 3, 2012
25
Monday, September 3, 2012
Summary
3D Polygonal shapes
26
Monday, September 3, 2012
Summary
3D Polygonal shapes
26
Monday, September 3, 2012
Summary
3D Joint hierarchy skeleton
3D Polygonal shapes
26
Monday, September 3, 2012
Summary
3D Joint hierarchy skeleton
3D Polygonal shapes
26
Hand animator modifies
physical simulation?
Monday, September 3, 2012
Summary
3D Joint hierarchy skeleton
3D Polygonal shapes
26
Hand animator modifies
physical simulation?
Monday, September 3, 2012
Learn cartoon physics?
Extra Slides
27
Monday, September 3, 2012
Camera Estimation
Camera rotation
and translation
ρ(i) = (θx (i), θy (i), θz (i), tx (i), ty (i), tz (i))T
∗
ρ (i) = arg min(w1 eg + w2 el + w3 eo + w4 es )
ρ
28
Monday, September 3, 2012
Camera Estimation
Camera rotation
and translation
ρ(i) = (θx (i), θy (i), θz (i), tx (i), ty (i), tz (i))T
∗
ρ (i) = arg min(w1 eg + w2 el + w3 eo + w4 es )
ρ
K/2
Geometric
projection error
eg =
�
t=−K/2
Hand drawings
||x̃i+t −
proj
xi+t ||
proj
where xi+t
∼
= Mi X̃i+t
Motion capture poses
28
Monday, September 3, 2012
arg min(λa ea (x) + λm em (x) + λs es (x))
x
input-matching
ea = ||x̃ij −
proj
xij
smoothing term
proj
xij ||
∼
= M i Xw
ij
x̃ij × Mi Xw
ij = 0
 w 
Xij
 Yijw 

CMi 
w =0
 Zij
1
 T 
m1
M =  mT2 
T
m3
Monday, September 3, 2012
motion prior
29
arg min(λa ea (x) + λm em (x) + λs es (x))
x
input-matching
ea = ||x̃ij −
proj
xij
proj
xij ||
∼
= M i Xw
ij
x̃ij × Mi Xw
ij = 0
 w 
Xij
 Yijw 

CMi 
w =0
 Zij
1
 T 
m1
M =  mT2 
T
m3
Monday, September 3, 2012
motion prior
smoothing term
T
−
m
em = ||mT3 Xw
3 X̃ij ||
ij
T
=
m
mT3 Xw
3 X̃ij
ij
29
arg min(λa ea (x) + λm em (x) + λs es (x))
x
input-matching
ea = ||x̃ij −
proj
xij
proj
xij ||
∼
= M i Xw
ij
x̃ij × Mi Xw
ij = 0
 w 
Xij
 Yijw 

CMi 
w =0
 Zij
1
 T 
m1
M =  mT2 
T
m3
Monday, September 3, 2012
motion prior
smoothing term
T
−
m
em = ||mT3 Xw
3 X̃ij ||
ij
T
=
m
mT3 Xw
3 X̃ij
ij
w
−
X
es = ||Xw
ij
(i+1)j ||
�
I
−I
�
�
Xw
ij
Xw
(i+1)j
�
=
�
0
29
�
arg min(λa ea (x) + λm em (x) + λs es (x))
x
input-matching
ea = ||x̃ij −
proj
xij
proj
xij ||
∼
= M i Xw
ij
x̃ij × Mi Xw
ij = 0
 w 
Xij
 Yijw 

CMi 
w =0
 Zij
1
 T 
m1
M =  mT2 
T
m3
Monday, September 3, 2012
motion prior
smoothing term
T
−
m
em = ||mT3 Xw
3 X̃ij ||
ij
T
=
m
mT3 Xw
3 X̃ij
ij
w
−
X
es = ||Xw
ij
(i+1)j ||
�
I
−I
�
�
Xw
ij
Xw
(i+1)j
�
=
�
0
WAi Xw
i = bi
29
�
30
Monday, September 3, 2012
30
Monday, September 3, 2012