Introduction
Image models
Camera coordinate space
Registration and minimization
Who Painted this Painting?
A Project in Image Registration
S. Allen Broughton - Rose-Hulman Institute of Technology
Rose Math Seminar, April 25, 2012
What’s left
Introduction
Image models
Camera coordinate space
Registration and minimization
What’s left
Where did this project come from?
the unknown painting
I received an email from an art collector leading to the
following story and question.
An art collector possesses a "painting of unknown origin"
which he suspects was painted by a "very famous painter".
The only possible evidence is a portion of the painting in
an old photograph taken in the painter’s studio.
Is there a way to match the two, such as matching
fingerprints?
Introduction
Image models
Camera coordinate space
Registration and minimization
Where did this project come from?
answer to the question and a way forward
I hope so!
The art collector hopes so!
Some kind of image processing might be helpful, That is
where I come in.
What’s left
Introduction
Image models
Camera coordinate space
Registration and minimization
Where did this project come from?
painter in his studio
Can you guess the painter in the photo in this link?
http://artmuseum.bowdoin.edu/Obj11581
What’s left
Introduction
Image models
Camera coordinate space
Registration and minimization
What’s left
Overview of problem solution and a future talk
talks in the problem
The problem is one in image registration or image comparison.
Talk 1: develop the math (this talk)
Talk 2: implementation in Matlab (future talk)
Introduction
Image models
Camera coordinate space
Registration and minimization
Overview of problem solution and a future talk
develop the math
describe models of images and image comparison
techniques
describe the camera orientation
describe image registration functions on camera
orientation
discuss minimization techniques
What’s left
Introduction
Image models
Camera coordinate space
Registration and minimization
Overview of problem solution and a future talk
implementation in Matlab
develop fast algorithms for computation of image
registration functions and their derivatives
select/develop minimization algorithm
calibrate method on known example
apply to problem
modify as needed
What’s left
Introduction
Image models
Camera coordinate space
Registration and minimization
What’s left
image models
continuous image models
consider only monochromatic images - can do since photo
is black and white
image can be modeled by a intensity function f (x, y )
defined on a rectangle
assume domain of image is a ≤ x ≤ b, c ≤ y ≤ d
may assume that intensity function satisfies 0 ≤ f (x, y ) ≤ 1
f (x, y ) = 0 for black f (x, y ) = 1 for white.
Introduction
Image models
Camera coordinate space
Registration and minimization
image models
discrete image models
may also consider image as an m × n matrix A where
Ai,j = f (d − (i − 1)∆y , (a + (j − 1)∆x)
where ∆x = (b − a)/n, ∆y = (d − c)/m
upper left of image is the (1, 1) entry.
may also consider Ai,j as some kind of average about
(d − (i − 1)∆y , (a + (j − 1)∆x)
convenient model for Matlab
What’s left
Introduction
Image models
Camera coordinate space
Registration and minimization
image comparison
comparisons for both models
let f (x, y ) and g(x, y ) be two images
a measure of the distance between the images is
RbRd
1
2
d(f , g) = (b−a)(d−c)
a c (f (x, y ) − g(x, y )) dxdy
alternatively let A and B be two images in discrete form
a measure of P
the distance
between the images is
m Pn
1
2
d(A, B) = mn
(A
i,j − Bi,j )
i=1
j=1
other exponents could be used
the normalization constants give a distance of 1 between
pure black and pure white
What’s left
Introduction
Image models
Camera coordinate space
Registration and minimization
taking pictures
pinhole camera
here is a picture of a pinhole camera taking a picture
(stolen form Wikipedia)
we will use this picture to describe camera orientation
What’s left
Introduction
Image models
Camera coordinate space
Registration and minimization
What’s left
taking pictures
the two photos and painting
our problem may formulated as follows
Take a camera, take a picture of a painting at some
orientation.
Take a reference photo with the camera pinhole at a unit
distance from the painting, painting perpendicular to the
camera axis, and painting rotated into a standard position
with respect to the camera.
Introduction
Image models
Camera coordinate space
Registration and minimization
What’s left
set up coordinates
film, real, and space coordinates - 1
Consider a camera with pinhole at the origin and film
located at the plane z = −1 so that the camera axis is the
positive z-axis. Call this the film plane with film coordinates
(u, v ) ↔ (u, v , −1) .
Consider also a separate plane with coordinates (x, y )
located at z = 0 and an image function f (x, y ) defined on
the plane for all (x, y ). This plane will model the painting
being photographed. The coordinates (x, y ) ↔ (x, y , 0) will
be called real coordinates.
Introduction
Image models
Camera coordinate space
Registration and minimization
What’s left
set up coordinates
film, real, and space coordinates - 2
If (x, y , z) is an arbitrary point in 3-space and (u, v , −1) is
the corresponding point in the image plane then
u = −x/z, v = −y /z.
in functional notation
space2cam(x, y , z) = (−x/z, −y /z)
If (u, v ) is point on the film then the corresponding points in
three space that map to (u, v ) have the form (zu, zv , −z).
Introduction
Image models
Camera coordinate space
Registration and minimization
set up coordinates
reference photo
Move the painting in parallel fashion to the plane z = 1,
(x, y , 0) → (x, y , 1)
take a picture and call this the reference photo
the mapping from film coordinates to real coordinates is
then given by
cam2real0 (u, v ) = (−u, −v )
What’s left
Introduction
Image models
Camera coordinate space
Registration and minimization
set up coordinates
painting constraints - 1
Move the painting from standard position. Take a picture.
The painting lies in a plane Π of the form
ax + by + cz = d
or
EtX = d
where E and X are the obvious column vectors.
E and d for the standard position are given by
E = E0 = (0, 0, 1)t , d = d0 = 0
E and d for the plane of the reference photo are given by
E = (0, 0, 1)t , d = 1
What’s left
Introduction
Image models
Camera coordinate space
Registration and minimization
What’s left
set up coordinates
painting constraints - 2
the point (x, y , z) on the plane Π meeting the line
generated by film point (u, v ) satisfies
z=
d
au + bv − c
the map from a film point (u, v ) to a point (x, y , z) on the
plane of the painting is given by
cam2spaceΠ (u, v ) = (
ud
vd
−d
,
,
)
au + bv − c au + bv − c au + bv − c
Introduction
Image models
Camera coordinate space
Registration and minimization
What’s left
camera space
moving the painting - 1
place the painting in standard position in the xy plane
(z = 0)
apply a rigid motion to the painting so that it lies in the
plane Π
this transformation has the form Y = SX + T where S is a
3x3 orthogonal matrix and T is a vector
the inverse transformation has the form X = S t (Y − T )
the transform for the reference photo has
S = id, T = (0, 0, 1)t
Introduction
Image models
Camera coordinate space
Registration and minimization
What’s left
camera space
moving the painting - 2
We would like to know the transformation from film points
(u, v ) to real points (x, y )
Compute a new E and d for the plane Π of the painting.
Compute Y = cam2spaceΠ (u, v )
apply the inverse map X = S t (Y − T )
to find E and d set X = S t (Y − T ) in E0t X = d0
to get E = SE0 , d = d0 + (SE0 )t T = (SE0 )t T
given (S, T ) let cam2realS,T (u, v ) denote the composition
of the two functions functions so defined
cam2realS,T (u, v ) = S t (cam2spaceΠ (u, v ) − T )
Introduction
Image models
Camera coordinate space
Registration and minimization
What’s left
camera space
camera space
The space of painting orientations is
SO(3) × R 3 = {(S, T )} where S is an orthogonal matrix
and T is a vector.
Now fix painting and move camera then SO(3) × R 3 is the
still the set of camera orientations. We will call the set
SO(3) × R 3 the camera space because it sounds "cooler".
Introduction
Image models
Camera coordinate space
Registration and minimization
What’s left
registration functions
registration function - 1
Let f (x, y ) and g(x, y ) denote two paintings defined for all
(x, y ) in the standard plane z = 0.
Let f0 (u, v ) be the reference photo defined by
f0 (u, v ) = f (cam2real0 (u, v ))
Let gS,T (u, v ) be the moving photo defined by
gS,T (u, v ) = g(cam2realS,T (u, v ))
Introduction
Image models
Camera coordinate space
Registration and minimization
What’s left
registration functions
registration function - 2
Finally define r (S, T ) to be the function defined by
r (S, T ) = d(f0 , gS,T )
or
r (S, T ) =
1
(b − a)(d − c)
Z
a
b
Z
c
d
(f0 (u, v )−gS,T (u, v ))2 dudv
Introduction
Image models
Camera coordinate space
Registration and minimization
minimization
minimization - 1
A local minimum for minimum for r (S, T ) will be local
closest match for the photos with a given camera
orientation.
An exact match occurs when r (S, T ) = 0
What’s left
Introduction
Image models
Camera coordinate space
Registration and minimization
minimization
minimization - 2
There are various iterative procedures that come to mind
to minimize r (S, T ), for example:
steepest descent.
steepest descent with random multiple starts.
A problem is that the domain of r (S, T ) is not Euclidian
space.
How do we compute gradients?
We finish with an approach to solve this problem.
What’s left
Introduction
Image models
Camera coordinate space
Registration and minimization
What’s left
minimization
minimization - 3
Near a point (S0 , T0 ) in camera space we want to linearize
SO(3) × R 3 .
Let dT be small vector and let dS be a small skew
symmetric matrix dS t = −dS.
Every point near (S0 , T0 ) has the form (S0 edS , T0 + dT ),
where
dS 2
+ ···
edS = I + dS +
2!
Introduction
Image models
Camera coordinate space
Registration and minimization
What’s left
minimization
minimization - 4
The gradient in the direction (dS, dT ) can be computed by
∇dS,dT =
d
r (S0 etdS , T0 + tdT )
dt
this would be a good time to use Maple
or calculate gradients numerically
Introduction
Image models
Camera coordinate space
Registration and minimization
work to do
compute gradient formulas in a nice way
compute gradients efficiently
translate to the discrete case
set up minimization
account for varying light intensity and other effects
What’s left
Introduction
Image models
Camera coordinate space
your questions
Any Questions?
Registration and minimization
What’s left