Colorado School of Mines
Computer Vision
Professor William Hoff
Dept of Electrical Engineering &Computer Science
Colorado School of Mines
Computer Vision
http://inside.mines.edu/~whoff/
1
Uncertainty
Colorado School of Mines
Computer Vision
2
Uncertainty
• Let’s say that we have computed a result (such as
pose of an object), from image data
• How do we estimate the uncertainty in our
answer?
• For reference, see the paper
– Hoff, W. A. and T. Vincent, “Analysis of Head Pose Accuracy in Augmented
Reality,” IEEE Trans. Visualization and Computer Graphics, Vol 6., No. 4,
pp. 319-334, 2000.
– Link at http://inside.mines.edu/~whoff/publications/pubs.htm
3
Colorado School of Mines
Computer Vision
Review - Mean and variance
• Assume we have a random variable x (e.g., a measurement)
– We have a sample of values x1, x2, …, xN
• We estimate its mean (expected value) by
1
m x  E ( x) 
N
N
x
i
i 1
• We estimate its variance by
sx
2
1

N
N
 x  m 
i 1
i
2
x
• The standard deviation sx is a good measure of variability of x
– If probability density of x is Gaussian, then a sample xi will fall within
m±sx 68% of the time
4
Colorado School of Mines
Computer Vision
Review - Covariance
• Covariance (of two variables, x and y) is a matrix Cxy
s
C xy  
s

2
xx
2
yx
s
s
2
xy
2
yy



1 N
2
s 
x

m
 i x 
N  1 i 1
2
xx
s
2
yy
2
1 N

yi  m y 


N  1 i 1
s s
2
xy
2
yx
1 N

 xi  m x   yi  m y 

N  1 i 1
• Or, if we have a vector x = (x y)T
T

Cx  E  x  μ x  x  μ x  


 x  m x 
x  m x
C x  E 

 y  m y 


 E  x  m x 2

 E  x  m  y  m 
x
y




y  m y 




5
Colorado School of Mines
Computer Vision

E  x  m x  y  m y  

2

E y  m y 

Examples
3
3
2
2
1
1
0
0
-1
-1
-2
-2
-3
-3
-2
-1
C =
0.0497
0.0123
0
1
2
-3
-3
3
-2
-1
C =
0.8590
0.8836
0.0123
0.8590
0
1
2
3
0.8836
1.1069
• Notes
– Off diagonal values are small if variables are independent
– Off diagonal values are large if variables are correlated (they vary
together)
Colorado School of Mines
Computer Vision
Matlab “cov” function
6
Functions of random variables
• Let’s say we have random variables x and y, and a function z =
f(x,y)
• If we know the uncertainty in x and y, what is the uncertainty
in z?
• Simple cases
z = a x, where a is a constant
1
mz 
N
z
i
1
2
s    zi  m z 
N
2
z
Colorado School of Mines
Computer Vision
7
Functions of random variables
• Sum of two variables
z=x+y
1
mz 
N
z
i
1
2
s    zi  m z 
N
2
z
Colorado School of Mines
Computer Vision
8
Example - Uncertainty in stereo disparity
• Disparity is the difference in positions
– d = xR – xL
• Assume
– Measurement uncertainty is sR=1, sL=1 pixel
– That sR, sL are independent (i.e., s2RL = 0)
• Then
2
s d2  s R2  s L2  2s RL
Colorado School of Mines
Computer Vision
9
General case
• Say we have z = f(x), where z and x are vectors
Cz 
1
T
z

μ
z

μ
 i z  i z 

N
• We can also write as
T
Cz  E  z  μ z  z  μ z  


• Or
T

Cz  E  z  z  


• It can be difficult to compute the uncertainty analytically
• But you can always get a numerical estimate of Cz, near
the current solution for z=f(x)
Colorado School of Mines
Computer Vision
10
Computing Covariance
• Doing a Taylor series expansion of z=f(x) and ignoring higher
order derivatives
z 
f
x  Jx
x
 f1 x1 f1 x2  f1 xM 


 f   f x f 2 x2  f 2 xM 
J  i   2 1



 x j   

 f x f x  f x 
N
2
N
M 
 N 1
• So
T
T
C z  E  z  z    E  Jx  Jx  




 E  J x xT JT   J E  x xT  JT
 J Cx J T
Recall
(AB)T = BTAT
• where Cx is the covariance matrix of the vector x
Colorado School of Mines
Computer Vision
11
Example – estimating stereo Z error
• We want to find CZ , if we know Z = f b/d
• Let Z=f(x), where x = (f b d)T . The Jacobian of Z=f(x) is

J  b/d
f /d
 fb / d 2

• Now find Cx = E( x xT ), assuming errors in f,b,d are independent
 f 
 
 E  b f
 d 
  s 2f
 

b d    0
  0

0
s b2
0
0 

0 
s d2 
• Then CZ = J Cx JT where J is the Jacobian evaluated at the current
value of x
Colorado School of Mines
Computer Vision
12
Probability Density
• Let’s assume that
errors have a
Gaussian probability
density
px 
0.4
0.3
0.2
0.1
0
• The probability
density for an 2dimensional vector
x is
px  
1
2 Cx
12
3
2
1
2
0
-1
x2 - axis
exp  12 xT Cx1x 
-1
-2
-3
0
3
1
-2
x1 - axis
x 
x   1 
 x2 
13
Colorado School of Mines
Computer Vision
Interpreting Probability Density
• Look at where the probability is a constant. This is
where the exponent is a constant:
xT Cx1x  z 2
• This is the equation of an ellipse. For example, with
uncorrelated errors this reduces to
x 2
s xx2

y 2
s yy2
 z2
• Can choose z to get desired probability. For z=3, the
cumulative probability is about 97%.
14
Colorado School of Mines
Computer Vision
Plotting
3
3
2
2
1
1
x2 - axis
x2 - axis
• Contours of constant probability
0
0
-1
-1
-2
-2
-3
-3
-2
-1
Colorado School of Mines
0
x1 - axis
1
2
3
-3
-3
Computer Vision
-2
-1
0
x1 - axis
1
2
3
15
% Show covariance of two variables
clear all
close all
Matlab code
randn('state',0);
yp = randn(40,1);
xp = 0.25 * randn(40,1);
% xp = randn(40,1);
% yp = xp + 0.5*randn(40,1);
plot(xp,yp, '+'), axis equal;
axis([-3.0 3.0 -3.0 3.0]);
C = cov(xp,yp)
Cinv = inv(C);
detCsqrt = sqrt(det(C));
% Plot the probability density,
% p(x,y) = (1/(2pi det(C)^0.5))exp(-x Cinv x/2)
L = 3.0;
delta = 0.1;
[x1,x2] = meshgrid(-L:delta:L,-L:delta:L);
for i=1:size(x1,1)
for j=1:size(x1,2)
x = [x1(i,j); x2(i,j)];
fX(i,j) = (1/(2*pi*detCsqrt)) * exp( -0.5*x'*Cinv*x );
end
end
hold on
% meshc(x1,x2,fX);
contour(x1,x2,fX);
xlabel('x1 - axis');
ylabel('x2 - axis');
Colorado School of Mines
% this does a surface plot
% this does a contour plot
Computer Vision
16
Estimation of Pose Covariance
•
Recall how we solved for the pose of an object x = (ax,ay,az,tx,ty,tz) using
iterative least squares
•
Given n measured 2D image points {pi}, and the corresponding 3D points
on the object {Pi}
•
We know how to predict the image points, given an estimate of the pose,
p = f(P, x)
f is the function that returns the predicted image points p = (p1,p2, … )T given the object points
P = (P1,P2, …)T
•
To estimate the pose, we took the derivative of p = f(P, x) to get
•
where J is the Jacobian of f, evaluated at (P, x)
Then we solved for the correction:
 f 
p    x  Jx
 x 
T
 
x  JT J
1
J T p
and added the correction to x, and repeated the steps until convergence
Colorado School of Mines
Computer Vision
17
Estimation of Pose Covariance
• Now we want to find the 6x6 covariance matrix of the pose error x
Cx = E ( x xT )
• From before, we had
 
x  J T J
1
J T p  J  p
• Now, Cp is the covariance
of the measured image
points
where J+ is the psuedoinverse
• So Cx is


C x  E J  p J  p
• Simplifying


T
• If independent, we have
 s x2 0
0
0 


2
0
0 
 0 sy

Cp  


2
 0
0
sx 0 

2

0
0
0
s
y 

 
 J E pp J 
 J C J 
C x  E J  ppT J 

T
T
 T
 T

p
Colorado School of Mines
Computer Vision
18
Example
•
•
Model based pose
estimation, from 6
target fiducials
Pose:
2
1
3
X =
1.5481
-0.8329
0.0936
0.9431
2.9714
18.3498
•
4
5
6
Covariance:
C =
0.0158
-0.0001
-0.0165
0.0170
0.0238
0.0434
Colorado School of Mines
-0.0001
0.0037
0.0004
-0.0122
0.0012
-0.0490
-0.0165
0.0004
0.0182
-0.0223
-0.0232
-0.0483
0.0170
-0.0122
-0.0223
0.0884
0.0150
0.2011
Computer Vision
0.0238
0.0012
-0.0232
0.0150
0.0546
0.0195
0.0434
-0.0490
-0.0483
0.2011
0.0195
1.2558
19
Estimating Measurement Uncertainties
• How to estimate the uncertainty in locating image points?
– Could estimate accuracy analytically, from knowledge of the image noise and
the performance of the feature operator in noise
– Another way is to look at the image residual error
• If we use N image points to determine the pose, and get a sum-of-squared
residuals of
N
ssr   p i  f x, Pi 
2
i 1
• Then an estimate of the image error is
s p2 
ssr
2N  6
• because we are summing a total of 2N values, and there are 6 parameters
to be fit
Colorado School of Mines
Computer Vision
20
Drawing in 3D
40
30
20
z
• Transform wireframe model using H10
0
• Draw segments using
line([pStart(1,i) pEnd(1,i)], -20
...
[pStart(2,i) pEnd(2,i)], ...
[pStart(3,i) pEnd(3,i);
-15
• Transform target points using H
• Draw targets using
-10
-5
plot3(P_c(1,:), P_c(2,:), P_c(3,:), '+');
0
• Create a box camera model as a set of
5
polygons (specify the vertices)
10
• Draw box using
y
fill3(Xvertices,Yvertices,Zvertices,'c');
15
Colorado School of Mines
Computer Vision
20
-20
-15
-10
-5
0
5
21 10
Covariances as Ellipsoids
• The joint probability density for n-dimensional vector x is

px  2
N 2
Cx

1 2 1

exp  12 xT Cx 1x

• A surface of constant probability is
xT Cx1x  z 2
• This is the equation of an ellipsoid in n dimensions
• For a 6-DOF pose, a 6x6 covariance matrix is difficult to
visualize
– We extract just the 3x3 submatrix corresponding to the translation
Colorado School of Mines
Computer Vision
22
Matlab Ellipsoid function
• [x,y,z] = ellipsoid(xc,yc,zc,xrad,yrad,zrad)
– generates (x,y,z) vertices for an ellipsoid centered at
xc,yc,zc
– radii are xrad,yrad,zrad
– ellipsoid is aligned with x,y,z axes
• We need to
– determine the radii
– rotate the vertices appropriately
Colorado School of Mines
Computer Vision
23
Rotating and scaling
• We have the equation of the ellipsoid
xT Cx1x  z 2
• Let y = Rx, where R is the rotation matrix that aligns the
ellipsoid with the x,y,z axes
• Then yT D y = z2, where D is a diagonal matrix
• Or, xT RT D R x = xT (RT D R) x = z2
• Thus, C-1 = RT D R
• We can get R by taking the SVD of C-1
• The length of axis i is 1/sqrt(si /z^2)
Using SVD,
C-1 = U D VT
24
Colorado School of Mines
Computer Vision
% We will draw the ellipsoid defined by the surface x' Cinv x = z^2,
% where Cinv = Cz^-1. For z=3, this is the 97% probability surface.
% First, center the ellispoid at the location of the model.
xc = X(4);
yc = X(5);
zc = X(6);
% Let y = Rx, where R is the rotation matrix that aligns the axes.
% Then y'*D*y = z^2, where D is a diagonal matrix.
% Or, x'R'*D*Rx = z^2. x'(R'DR)x = z^2.
% So Cinv = R'DR. This is just taking the SVD of Cinv.
[U,S,V] = svd(inv(Cz));
R = V;
% The length of the ellipsoid axes are len=1/sqrt(Si/z^2)
% where Si is the ith eigenvalue
xrad = sqrt( 9/S(1,1) );
yrad = sqrt( 9/S(2,2) );
zrad = sqrt( 9/S(3,3) );
[x,y,z] = ellipsoid(0,0,0,xrad,yrad,zrad);
% Rotate the ellipsoid
for i=1:size(x,1)
for j=1:size(x,2)
Y = R' * [x(i,j); y(i,j); z(i,j)];
xr(i,j) = Y(1)+xc;
yr(i,j) = Y(2)+yc;
zr(i,j) = Y(3)+zc;
end
end
surf(xr,yr,zr);
Colorado School of Mines
Computer Vision
25
0
z
Example
-10
0
10
y
26
Computer Vision
20
Colorado School of Mines
x
More Examples (Hoff & Vincent paper)
Uncertainty
ellipsoids
The rotational uncertainty is depicted as
elongated cones about each axis.
Colorado School of Mines
Computer Vision
A sensor estimates the pose of the target on the top
of the head. Its covariance is shown as a small
ellipsoid, barely visible. Using the known pose of
the glasses with respect to the target, the pose of
the glasses is then estimated, along with its
27
covariance matrix (larger ellipsoid).
Combining pose estimates
The ellipsoids from two sensors are nearly orthogonal. The ellipsoid corresponding to the
combined estimate is much smaller and is contained in the volume of intersection.
Colorado School of Mines
Computer Vision
28