Image Processing
and Computer Vision
Chapter 11: Bundle adjustment
Structure reconstruction SFM
from N-frames
Bundle adjustment– structure
reconstruction V5a
1
Reconstruction from N-frames
• Factorization (linear, fast, not too accurate)
• Bundle adjustment (slower but more accurate),
can use factorization results as the first guess.
– Non linear iterative methods are more accurate
than linear method, require first guess (e.g. From
factorization).
– Many different implementations, but the concept
is the same.
Bundle adjustment– structure
reconstruction V5a
2
Problem definition
• There are N features in the 3D object .
• We take  pictures of the object at different
views.
• Input :
• Image sequence I1,I2,…I .
• Each image has n image feature points
• Output (structure=model, and motion=pose)
• 3-D coordinates of all 3-D model points X1,X2,..,XN.
• Camera pose for each image taken [R(t),T(t)] t=1,… 
Bundle adjustment– structure
reconstruction V5a
3
Example: Bundle adjustment 3D reconstruction (see also
http://www.cse.cuhk.edu.hk/khwong/demo/index.html)
• Grand Canyon Demo
• Flask
• Robot
http://www.youtube.com/watch?v=2KLFRILlOjc
http://www.youtube.com/watch?v=xgCnV--wf2k
http://www.youtube.com/watch?v=ONx4cyYYyrI
http://www.youtube.com/watch?v=4h1pN2DIs6g
Bundle adjustment– structure
reconstruction V5a
4
The iterative SFM alternating bundle adjustment method
• Break down the system into two phases:
--SFM1: find pose phase
--SFM2: find model phase
• Initialize first guess of model
– The first guess is a flat model perpendicular to the image
and is Zinit away (e.g. Zinit = 0.5 meters or any reasonable
guess)
• Iterative while ( Err is not small )
• {
– SFM1: find pose phase
– SFM2: find model phase
– Measurement error(Err) or(model and pose stabilized)
• }
Bundle adjustment– structure
reconstruction V5a
5
SFM1 : find pose phase
Pose estimation
discussed in the last chapter
Bundle adjustment– structure
reconstruction V5a
6
SFM2:
Model finding by the iterative method
• Similar to pose estimation.
– In pose estimation: model is known, pose is
unknown.
– Here (Model finding by the iterative method)
Assume pose is known, model is unknown.
– The algorithms are similar.
Bundle adjustment– structure
reconstruction V5a
7
Problem
Formulation
From the pose estimation slides (chapter iv10 - pose estimation ) :
Assume for pose R,T  θ   1 2 3 T1 T2 T3 
T
T1 , T2 , T3 are translati ons along X, Y, Z axes repectivel y,
1 , 2 , 3  are rotation angles around X, Y, Z axes respective ly,
T
M i  X i Yi Z i  is the i th model point (i  1,2, , N) , xi  ui
vi  is
T
the image point of Pi , rewrite (1a),1(b)
ui  f
r11 X i  r12Yi  r13Z i  T1
 g u ,i ( , M i )        (2a )
r31 X i  r32Yi  r33Z i  T3
vi  f
r21 X i  r22Yi  r23Z i  T2
 g v ,i ( , M i )        (2b)
r31 X i  r32Yi  r33Z i  T3
- - - - - - - - - - - - - - - In this chapter - - - - - - - - - - - - - - - - - - - - - - - - - Problem definition : There are t  1,2, , image frames, and
the object i  1,2,..., N features
Asume the pose θt 1,2 ,.., Γ  [φ1,φ2 ,φ3 ,T1,T2 ,T3 ]tT ,
and image measuremen ts xi,t  ui
vi t are known.
T
SFM2 task : From the given image frames and pose θt 1,2 ,.., Γ
Find M i 1, 2,.. N  X i
Yi
Zi 
T
i 1, 2 ,.. N
Bundle adjustment– structure
reconstruction V5a
•
8
Derivation for the model partial derivatives
Given ui  g u (1 , 2 , 3 , T1 , T2 , T3 , X i , Yi , Z i ) 
 f
r11 X i  r12Yi  r13Z i  T1
X
 f
r31 X i  r32Yi  r33Z i  T3
Z
R
i
R
i
 f
X i  3Yi  2 Z i  T1
 2 X i  1Yi  Z i  T3
 r11 r12 r13   1  3 2 
R  r21 r22 r23    3
1  1 
r31 r32 r33   2 1
1 
ui
Find
.
X
 r11 X i  r12Yi  r13Z i  T1 
 X iR 
  f R 
 f
r31 X i  r32Yi  r33Z i  T3 
Zi 
ui


 
X
X
X
 f 
 R 
Z i  R  fX iR 1


Xi 
X
X Z iR


 f 
 R  R
Z i  Z i
 X iR f

R

Xi 
Z iR X
X Z iR
 
 1 
r31 X iR  r11 f
f
 ZR 2 
Z iR
 i 
r XR 
r
  f  31 i2   f 11R
 ZR 
Zi
 i 
r
r X 
 f  11R  31 2i 
 Zi
Z iR 

 
 
 
Given ui  g u (1 , 2 , 3 , T1 , T2 , T3 , X i , Yi , Z i ) 
X i  3Yi  2 Z i  T1
X iR
r11 X i  r12Yi  r13Z i  T1
 f R  f
 f
 2 X i  1Yi  Z i  T3
Zi
r31 X i  r32Yi  r33Z i  T3
so r11  1, r12  3 , r13  2
r31  2 , r32  1 , r33  1
ui
.
Y
 X iR 
 r11 X i  r12Yi  r13Z i  T1 


 f
  f Z R 
T

Z
r

Y
r

X
r
ui
i 
  31 i 32 i 33 i 3   
Y
Y
Y
 f 
 R 
Z i  R   fX iR  1

Xi 

Y Z iR
Y
Find
 f 
 R  R
Z i  Z i
 X iR  f

R
Xi 

Y Z iR
Z iR Y
 1 
r32 X iR  r12 f
f
2
 Z R  
Z iR
 i 
r XR 
r
  f  32 i2   f 12R
 Z R  
Zi
 i 
r
r X 
 f  12R  32 2i 
 Z i Z R  
i


Bundle adjustment– structure
reconstruction V5a
9
CMSC5711: Exercise11.1:
Given ui  g u (1 , 2 , 3 , T1 , T2 , T3 , X i , Yi , Z i ) 
r11 X i  r12Yi  r13Z i  T1
X iR
X i  3Yi  2 Z i  T1
 f
 f R  f
r31 X i  r32Yi  r33Z i  T3
Zi
 2 X i  1Yi  Z i  T3
so r11  1, r12  3 , r13  2
r31  2 , r32  1 , r33  1, or
 r11 r12
R  r21 r22
r31 r32
Find
r13   1
r23    3
r33   2
 3
1
1
2 
 1 
1 
 r11 X i  r12Yi  r13Z i  T1 
 X iR 
  f R 
 f
r31 X i  r32Yi  r33Z i  T3 
Zi 
ui




?
Z
Z
Z
Bundle adjustment– structure
reconstruction V5a
•
10
Exercise11.2: proves all these
partial derivatives
r
r31 X i 
ui
11


 f
R
R 2 

Z
X i
Z
i

 i
r
r32 X i 
ui
12


 f
R
R 2 

Z
Yi
Z
i
i


r
r33 X i 
ui
13


 f
R
R 2 

Z
Z i
Z
i
i


r
r31Yi 
vi
21


 f
R
R 2 

Z
X i
Z
i
i



r
Y
r
vi
i
32
22

 f R 
2
R
 Zi
Yi
Z i 

 
 
 
 
 
r
r33Yi 
vi
23


 f
R
R 2 

Z
Z i
Z
i

 i
 
Bundle adjustment– structure
reconstruction V5a
•
11
Recall
•
Bundle adjustment– structure
reconstruction V5a
12
Assume  t 1,..,  are known, for the i th model point :
~
~ ~ ~
M i  (X i ,Yi ,Z i ) is a guessed 3 - D model point
u  u~ 
i ,t
Continue
i ,t
~
ui ,t  g u ( i ,t , M i ) 
~
~
~
g u ( i , M )
g u ( i , M )
~
~ g u ( i , M )
~
(Xi  Xi) 
(Yi  Yi ) 
( Z i  Z i ) - -(4a)
X
Y
Z
~
vi ,t  g v ( i ,t , M t ) 
~
~
~
g v ( i , M )
g v ( i , M )
~
~ g v ( i , M )
~
(Xi  Xi) 
(Yi  Yi ) 
( Z i  Z i ) - -(4b)
X
Y
Z
combine (4a) and (4b), put them in a matrix form
~


X

X
i
i
~
u  u 

~ 
e   i ~i   jM  Yi  Yi  
 vi  vi 
 Z i  Z~i 


~
~
~
 g u ( i , M ) g u ( i , M ) g u ( i , M )   X  X~ 

  i ~i
X ~
Y ~
Z ~ 
Yi  Yi     (5)


 g v ( i , M ) g v ( i , M ) g v ( i , M )   Z  Z~ 
i 

 23  i
31
X
Y
Z
Bundle adjustment– structure
reconstruction V5a
13
ui  u~i 
e( t ) 21  
~
 vi  vi  ( t )21
~
~
 gu ( t , M )
Xi  Xi 


~ 
X ~
 jt( m ) 23  Yi  Yi 

 g v ( t , M )
 Z i  Z~i 

 ( t )31 
X
~
gu ( t , M )
Y ~
g v ( t , M )
Y
continue
~
~
gu (t , M ) 
Xi  Xi 


~ 
Z ~ 
Y

Y
   (5)
i
i 

g v ( t , M ) 
 Z i  Z~i 
 31
Z
 ( t ) 23 
If we have the i th model point view ed t  1,2 ,,.Γ times, stack the matrix relations
 ui  u~i 

~
m
 et 1    vi  v~i  t 1   jt 1   X i  X i 

 
~ 
 
E2 1   :   
:
  :   Yi  Yi  

 
~
~
et     ui  ui    jt(m)   Z i  Z i 
2 3
31
  v  v~  
i
i


t




~
~
~
  gu (i , M

) gu (i , M ) gu (i , M ) 




X

Y

Z


~
~
~ 

g
(

,
M
)

g
(

,
M
)

g
(

,
M
)
v
i
v
i
v
i



~


X

X
i
i

 

X
Y
Z
 t 1

~



:
Y

Y
 i i             ( 6)
~
~
~
  g ( , M

) gu (i , M ) gu (i , M ) 
 Z i  Z~i 
  u i


 31
X ~
Y ~
Z ~  
 
  g v (i , M ) g v (i , M ) g v (i , M )  
 
X
Y
Z
 t  

2 3
~
Xi  Xi 
 j ( m ) t 1 

~ 


Setup the Jacobain for the model J ( m )   :  , M   Yi  Yi 
 Z i  Z~i 
 j ( m )t  

 2 3

 31
E2 1  J ( m ) 2 3 * M 31                                    (7)
Bundle adjustment– structure
reconstruction V5a
14
SFM2: Iteration for finding the model point i: In this algorithm each point i (i=1,2,..N) is
found independently , so the following algorithm will be run N times.
E  J ( m ) * M , so J -1 * ke  ΔM
M  X
Z 
Y
T
~
Base on the first guess M k 0 , we can find
SFM2: This
algorithm is
to find the
model M
Ek 0  J ( m ) * M k 0
Iterate (k  0 ,k  K _ max ,k  k  1 )
{
find E k and J k-1 // if J is not a square matrx, use pseudo inverse

M k  J

( m ) 1
Ek
Break if M k is small enough
~
~
Next guess is M k 1  M k  M k
}
~
After the end of the above loop M k is the result
Bundle adjustment– structure
reconstruction V5a
•
15
The iterative SFM alternating bundle adjustment method
• Break down the system into two phases:
--SFM1: find pose phase
--SFM2: find model phase (method (A) or (B))
• Initialize first guess of model
– The first guess is a flat model perpendicular to the image
and is Zinit away (e.g. Zinit = 0.5 meters or any reasonable
guess)
• Iterative while ( Err is not small )
• {
– SFM1: find pose phase
– SFM2: find model phase
– Measurement error(Err) small or model and pose
stabilized
• }
Bundle adjustment– structure
reconstruction V5a
16
Overall processing revisit
• Given: measurements
– Images of N frames
• Point feature tracked by KLT Kanade–Lucas–
Tomasi_feature_tracker or SURF (Speeded Up Robust
Features) methods
• Examples, demo
• http://www.youtube.com/watch?v=RXpX9TJlpd0
• To find pose (Rotation R, translation T ) of
every frame, and the model structure X
Bundle adjustment– structure
reconstruction V5a
17
Putting it altogether
Repeat the explanations
SFM1 (find pose phase) and
SFM2 (find model phase) with
implementation details.
Bundle adjustment– structure
reconstruction V5a
18
•
•
•
•
•
•
Putting it altogether
Use KLT (or SIFT, Harris then correlation) to obtain features in [u,v]T
There are t=1,2,…, image frames,
So there are t=1={R,T} t=1 , t=2={R,T} t=2 , …., t=={R,T} t=  poses.
There is only one model Mi=[X,Y,Z]I,with i=1,..,N features
Initialize first guess of model
– The first guess is a flat model perpendicular to the image and is Zinit away
(e.g. Zinit = 0.5 meters or any reasonable guess)
Iterative while ( Err is not small ){
–
–
–
–
–
–
–
–
–
–
–
/////////////// SFM1: Pose finding ////////////////////////////////////
//(for every time frame t, use all N features, run SFM1 once); so SFM1 runs  times here
For (t=1; t<; t++)
{ Inputs: You have f(focal length), Mi=[X,Y,Z]i
For each frame t, you have i=1,,,N, image feature points and measurements [u,v]Ti,t
Output: pose t
}
After the above is run
t=1={R,T} t=1 , t=2={R,T} t=2 , …., t=={R,T} t=  poses are found
////////////////////// SFM2: model finding //////////////////////
(For i=1,i<N;i++) (for every feature, use all  frames, run SFM2 once; so SFM2
runs N times here)
Bundle adjustment– structure
reconstruction V5a
• {SFM2: find model phase}
19
– Measurement error(Err) small or model and pose stabilized}
Recall: SFM1:Find Pose phase
//(for every time frame t, use all N features,
run SFM1 once); so SFM1 runs  times here
For (t=1; t<; t++)
{ Inputs: You have f(focal length), Mi=[X,Y,Z]i
For each frame t, you have i=1,,,N, image
feature points and measurements [u,v]Ti,t}
Output: pose t
}
After the above is run
t=1={R,T} t=1 , t=2={R,T} t=2 , …., t=={R,T} t= 
poses are found
Bundle adjustment– structure
reconstruction V5a
20
Recall in pose estimation SFM1: a slide in ch.iv10: pose estimation
For each frame t, we have N correspond ence points, i  1, , N, stack the matrix relations
~
 1  1 
• The formulas apply to one
~

 ui 1  ui 1  
~
frame at time t. There are
2  2 

~ 
j
 ei 1 


~
v

v
i

1
3  3 
  i 1 i 1  
 : 
i=1,2,…N features.
E2 N 1   : 

:



~


~
T T
• each time t, t is found.
ei  N  2 N 1  ui  N  ui  N  
 ji  N  2 N 6  1 ~1 





~
T T
• SFM1 will  times, each
 2 ~2 
  vi  N  vi  N   2 N 1
T3  T3  61
time is independent.


Yi X iR
X i X iR  Z i Z iR
Yi
X iR 
f

f
f

f
0

f


Z iR Z iR
Z iR 2
Z iR 2
Z iR 2      ~ 

1
1


YiYi R  Z i Z iR
Yi R X i
Xi
Yi R 
f

~

f
f
f
0

f
 


Z iR
Z iR
Z iR 2
Z iR 2
Z iR 2  i1  2  ~2 
 

E2 N 1  
:
 3 ~3   (6)
 

 T1  T1 
YXR
X X R Z ZR
Y
X iR 
f
   f i i2

f i i 2 i i  f iR
0

f
T  T~ 
R
R
R
R 2
Z
Z
 
Z i 
Z i 
Z i     2 ~2 
i
i
 

T3  T3  61
Y Y R  Z i Z iR
Yi R X i
Xi
Yi R 
f
  f i i

f
f
0

f

2
2
2
Z iR
Z iR
 



Z iR 
Z iR 
Z iR   i  N 

 2 N 6
~
 1  1 

~
2  2 

 ji 1 
3  ~3 


Put the Jacobain J   : 
,   
~
T1  T1 

 ji  N  2 N 6
T  T~ 
 2 ~2 
T3  T3  61
E2 N 1
At time t,
there are
N features
•
Bundle adjustment– structure 21
 J 2 N 6 *  61                                    (7)
reconstruction V5a
• Exercise 11.3 : If the model is a
checker board plane, each square
is 1cm2.. It is perpendicular to the
camera principal axis and at Z=0.5
meters away from the camera
center. Pixel width is 5um.
• Find the 3D positions of X1,X2,X3
and X4 in pixels
Exercise
•
X4
1cm
Y
1cm
x3=
[0,0]
X2
X
Bundle adjustment– structure
reconstruction V5a
X1
22
How to get the first guess of the
model?
• We have an image with [u,v]i=1,2,..,N
• Camera focal length is f
• In theory
Image
– ui=f*Xi/Zi
– vi=f*Yi/Zi
•
•
•
•
Zguess
Camera
f center
First guess of the model is all points on a plane
X’i=ui*Zguess/f, Y’i=vi*Zguess/f,
Z’i=Zguess=0.5 meters (for example)
So the guessed i-th 3D point is at [X’I, Y’I, Z’i]
Bundle adjustment– structure
reconstruction V5a
23
Exercise 11.4 Revision for pose estimation SFM1: : Point out which are know variables
and unknown variables in this page.
For each frame t, we have N correspond ence points, i  1, , N, stack the matrix relations
~
 1  1 
~

 ui 1  ui 1  
~



2
2


~ 
j
 ei 1 


~
v

v
i

1
i

1
i

1

3  3 

E2 N 1   : 

:
  : 


~
~
T

T
u

u



 i N
i N 
ei  N  2 N 1
 ji  N  2 N 6 1 ~1 
T  T 


v  v~ 
 2 ~2 
  i  N i  N   2 N 1
T3  T3  61
• The formulas apply to
one frame at time t.
There are i=1,2,…N
features.
• each time t, t is found.
• SFM1 will  times, each
time is independent.


Yi X iR
X i X iR  Z i Z iR
Yi
X iR 
f

f
f

f
0

f


Z iR Z iR
Z iR 2
Z iR 2
Z iR 2      ~ 

1
1


YiYi R  Z i Z iR
Yi R X i
Xi
Yi R 
f


~
f
f R
0
f



  f


2
2
2
R
Zi
Zi
Z iR 
Z iR 
Z iR   i1  2  ~2 
 

E2 N 1  
:
 3 ~3   (6)
R
R
R
R
 

 T1  T1 
YX
X X Z Z
Y
Xi 
f
   f i i2

~
f i i 2 i i  f iR
0

f
Zi
Z iR
 
Z iR 
Z iR 
Z iR 2   T2  T~2 
 

T3  T3  61
Y Y R  Z i Z iR
Yi R X i
Xi
Yi R 
f
  f i i

f
f
0

f

2
2
2
Z iR
Z iR
 



Z iR 
Z iR 
Z iR   i  N 

 2 N 6
~
 1  1 

~



 2 2
 ji 1 
3  ~3 


Put the Jacobain J   : 
,   
~
T1  T1 

 ji  N  2 N 6
T  T~ 
 2 ~2 
T3  T3  61
E2 N 1  J 2 N 6 *  61                                (7)
At time t,
there are
N features
•
Bundle adjustment– structure 24
reconstruction V5a
Exercise 11.5 Revision for pose estimation SFM1: : Explain why E is
known here in the following formula.
If we have N correspond ence points, i  1, , N, stack the matrix relations
~
 1  1 

 ui 1  u~i 1  
~
2  2 





~
 ei 1 
 ji 1 
3  ~3 
  vi 1  vi 1  




E2 N 1   : 

:
 : 


~
~
T

T
u

u



 iN
iN 
ei  N  2 N 1
 ji  N  2 N 6 1 ~1 
T  T 

~ 
v

v
i

N
i

N


 2 ~2 

 2 N 1
T3  T3  61
•
Bundle adjustment– structure
reconstruction V5a
25
Revision for pose estimation SFM1: Explain why J is known here.
Answer: because guessed M, and guessed pose are known
  g u (~, M 1 ) 


1


~
  g v ( , M 1 ) 




1



:



J 
:



:




:



~
 g u ( , M 1 ) 




2


~
 g v ( , M 1 ) 


 2


:
:
:
:
:
: :
:
: :
~
 g u ( , M i  I ) 




3

 : :
~
 g v ( , M i  I ) 


 3


:
: :
:
: :
Bundle adjustment– structure
reconstruction V5a
~
 g u ( , M i 1 )  




6


~
 g v ( , M i 1 )  


 6


:




:



:

~
 g u ( , M i  N )  




6


~
 g v ( , M i  N )  


 6

  2 N 6
•
26
Exercise 11.6 : Revision for pose estimation SFM1: Identify known variables and
unknown variables when k=0 and k=5 in this iterative pose estimation algorithm
E  J *  ,
Note :
  1 , 2 , 3 , T1 , T2 , T3 
so J -1 * E  Δθ
  1
2
3
T1
T2
T3 
T
~
Base on the first guess  k 0 , we can find
SFM1: This
~
1
 k 0  J k 0 ( ) * Ek 0
algorithm is
Iterate (k  0 ,k  K _ max ,k  k  1 )
{
to find the
pose 
~
find Ek and J k10 ( ) // if no J -1 use pseudo inverse
~
 k  J k10 ( ) Ek
Break if  k is small enough
~
~
Next guess is  k 1   k   k
 is the pose that we want to find
~
~
~ ~ ~ ~ ~
  1 , 2 , 3 , T1 , T2 , T3

~
  guessed pose
~
 1  1 

~



 2 2
3  ~3 
  
~
T

T
 1 1
T  T~ 
 2 ~2 
T3  T3  2 N6
• The formulas apply to one
frame at time t. There are
i=1,2,…N features.
• SFM1 will  times, each
time is independent,
}
Bundle adjustment– structure
reconstruction V5a
27
SFM2: Find model phase
Similar to SFM1 but pose is
known, find model here.
(for every feature, use all  frames, run SFM2
once; so SFM2 runs N times here)
For i=1,i<N;i++
{ SFM2: find model phase
}
Bundle adjustment– structure
reconstruction V5a
28
Exercise11.7: Revision of SFM2: Identify which are known which are known
here. Explain why J(m) is known at this stage.
Measured
Result from the guessed model and given pose
~
~
 X i  X i   g u ( i , M )
~
ui ,t  ui ,t 
~  
(m) 
X ~
e

j
Y

Y
~ 
 i i    g ( , M
v

v
)
~
 i ,t i ,t 
 Zi  Zi   v i

 t 
X
Jacobian
~
g u ( i , M )
Y ~
g v ( i , M )
Y
~
g u ( i , M )   X  X~ 
i
i

Z ~   Y  Y~     (5)
i
i 
g v ( i , M )  
 Z i  Z~i 

Z
 t 
New Guessed
model
• The formulas apply to one
feature (i) for all time
frames t=1,2,….
• SFM2 will N times , each
time is independent.
• See next slide for the
graphical illustration,
Current Guessed model
If we have Γ frames, i.e. t  1,2 ,Γ , stack the matrix relations for t  1,2 ,..Γ
~
~
~
  g u ( i , M

) g u ( i , M ) g u ( i , M ) 



X ~
Y ~
Z ~ 


~

 ui  ui  
g v ( i , M ) g v ( i , M ) g v ( i , M ) 



~
~
(m)

~ 
Xi  Xi 
Xi  Xi 

j
i 1 
 et 1 


v

v


X
Y
Z

 t 1
  i i  t 1 
~ 

 
  Y  Y~   (6)
E   : 

:
  :   Yi  Yi   
:


  i ~i 
~
~
~
~
 g u ( i , M ) g u ( i , M ) g u ( i , M ) 
 j ( m ) i    Z i  Z~i 
et   2 1  ui  ui  

  Z i  Z i 

 2 3 
 31



31
~ 


v

v
X
Y
Z


~
~
~
  i i  t   2 1
 g ( , M ) g ( , M ) g ( , M )

v
i
v
i

  v i

X
Y
Z
 t  
 
2 3
~
(m)
Xi  Xi 
 j t 1 

~ 


Put the Jacobian for the model J ( m ) 2 3   :  , M   Yi  Yi 
 Z i  Z~i 
 j ( m ) t  




•
2 3
E2 1  J
(m)
2 3
31
* M 31                                    (7)
Bundle adjustment– structure
reconstruction V5a
29
In SFM2, we handle a feature i, at one time.
• The formulas in the pervious slide apply to one feature (i) for
all time farmes t=1,2,…
X
v1
[u,v]i,t=1 v2
v3
Ot=1 Image
t=1
t=1={R,T} t=1
Image
Ot=2 t=2
t=2={R,T} t=2
Bundle adjustment– structure
reconstruction V5a
vm
[u,v] i,t=2
[u,v] i,t=3
…
Image
t=3
Ot=3
t=3={R,T} t=3
[u,v] i,t=
Ot=
Image
t=
t=={R,T} t=
30
Camera motion
Ot=camera center at time t
Exercise11.8 :
SFM2: Algo. to find the i-th model point (repeat this N times to get all N points)
Identify which are known and unknown when K=0, K=5
E  J ( m ) * M , so J -1 * ke  ΔM
M  X
Note :
Z 
Y
T
~
Base on the first guess M k 0 , we can find
Ek  0  J
(m)
* M k 0
SFM2: This
algorithm is
to find the
model M
~
Xi  Xi 

~ 
M   Yi  Yi 
 Z i  Z~i 


M i  [ X i , Yi , Z i ]T  unknwon
M i  is the model we want to find
~
~ ~ ~
M i  [ X i , Yi , Z i ]T  Guessed model
~
M i at _ k 0  a plane at k thiterative
~
M i at _ k  model found by the previous SFM2
Iterate (k  0 ,k  K _ max ,k  k  1 )
 
 
{
find E k and J k-1 // if J is not square, use pseudo inverse

M k  J ( m )

1
Ek
Break if M k is small enough
~
~
~
Next guess is M k 1  M k  M k
}
~
After the end of the above loop M k is the result
Bundle adjustment– structure
reconstruction V5a
• The formulas apply to one
feature (i) for all time
frames t=1,2,….
• SFM2 will N times , each
time is independent.
• See next slide for the
graphical illustration,
•
31
From [2] Result for rotation angles
Bundle adjustment– structure
reconstruction V5a
32
From [2] Result for translations
Bundle adjustment– structure
reconstruction V5a
33
From [2] Result: compare full/classical(+)
and 2-pass algorithm (o)
Bundle adjustment– structure
reconstruction V5a
34
From [2] Results for real images
Bundle adjustment– structure
reconstruction V5a
35
Conclusions
• Bundle adjustment can be used for structure
and motion SAM (model structure
reconstruction and pose estimation).
• Bundle adjustment is an accurate method for
Structure from motion SFM.
• It can made more efficient by using a two pass
(pose finding step, model fining step)
algorithm
Bundle adjustment– structure
reconstruction V5a
36
Appendices
Bundle adjustment– structure
reconstruction V5a
37
Demo Newton's method
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
function new_x=demo_newton1(x)
•
>> demo_newton1(1)
%This is to solve x^3-2x=3
•
new_x=5.000,err=4.000, err is still too big
%assume x is the guessed x
% 3= f(true_x)=f(x)+f'(x)(new_x-x)+ small_terms_ignored %
•
new_x=3.466,err=1.534, err is still too big
Taylor series
% 3-f(x)/f'(x)=new_x-x, or
•
new_x=2.534,err=0.931, err is still too big
% new_x=x+((3-f(x))/f'(x))= new_x, until new_x not changed
% so that
•
new_x=2.059,err=0.475, err is still too big
% new_x=x+((3-f(x))/f'(x))
% new_x=x+((3-(x^3-2*x))/(3*x^2-2));
while (1)
•
new_x=1.909,err=0.150, err is still too big
new_x=x+((3-(x^3-2*x))/(3*x^2-2));
err=abs(x - new_x);
•
new_x=1.893,err=0.015, err is still too big
st=sprintf('new_x=%2.3f,err=%2.3f, err is still too
big\n',new_x,err);
•
new_x=1.893,err=0.000,
disp(st);
•
ans =
if (err < 0.01)
•
err is small new_x is the solution
break;
•
new_x =
end
•
1.8933
%'err still big, hit key to continue'
pause
•
ans =
x=new_x;
•
1.8933
end
'err is small new_x is the solution'
Bundle adjustment– structure
new_x
reconstruction V5a
38
Rotation matrix
•
 r11 r12 r13 
 r21 r22 r23  
r31 r32 r33 
cos(2 )cos(3 )
- cos(2 )sin( 3 )
sin( 2 )


 sin(  )sin(  )cos( ) + cos( )sin(  ) - sin(  )sin(  )sin(  ) + cos( )cos( ) - sin(  )cos( )
1
2
3
1
3
1
2
3
1
3
1
2 

- cos(1 )sin( 2 )cos(3 ) + sin( 1 )sin( 3 ) cos(1 )sin( 2 )sin( 3 ) + sin( 1 )cos(3 ) cos(1 )cos(2 ) 
And
R 1  RT , R T R  I 3 , det( R)  1
Also, when 1 , 2 , 3 are small
 r11 r12
R  r21 r22
r31 r32
r13   1
r23    3
r33   2
 3
1
1
2 
 1 , when 1 , 2 , 3 are small, so
1 
Bundle adjustment– structure
reconstruction V5a
39
Jacobian for model :JacobM
% Jacobian for model :JacobM %%%%%%%%
•N = size(model,2); %model=3,N
•if N~=1
• error('JacobM: model size must be 4*1');
•end
•T = size(rt,3);
%rt=3,4,T
•J=zeros(2*T,3);
•for t=1:T
%index T vertical blocks
• V = rt(:,:,t)*model;
• X = V(1,:);
• Y = V(2,:);
• Z = V(3,:);
• Z2 = Z.*Z;
• XZ2 = X./Z2;
• YZ2 = Y./Z2;
• a11 = rt(1,1,t)./Z - rt(3,1,t).*XZ2;
• a12 = rt(1,2,t)./Z - rt(3,2,t).*XZ2;
• a13 = rt(1,3,t)./Z - rt(3,3,t).*XZ2;
• a21 = rt(2,1,t)./Z - rt(3,1,t).*YZ2;
• a22 = rt(2,2,t)./Z - rt(3,2,t).*YZ2;
• a23 = rt(2,3,t)./Z - rt(3,3,t).*YZ2;
• a1 = [a11' a12' a13'];
• a2 = [a21' a22' a23'];
•
• J(t,:) = a1;
• J(T+t,:) = a2;
•end
Bundle adjustment– structure
•J = flen.*J;
reconstruction V5a
40
Angles and R pose conversion
 r11 r12
R   r21 r22

 r31 r32
r13 
r23 

r33 
cos( ) *sin(  )  sin( ) *sin( )
cos( ) * cos(  ) cos( ) *sin(  ) *sin( )  sin( ) * cos( )


 sin( ) * cos(  )
sin  *sin  *sin   cos 
sin( ) *sin(  ) * cos( )  cos( ) *sin( ) 


 sin(  )
cos  sin 
cos(  ) * cos( )


When _ angles _ are _ small ,
 1
R   3
 2
3
1
1
2 
1 
1 
Bundle adjustment– structure
reconstruction V5a
41
jacobian for chang,wong ieee_mm 2 pass lowe
•
•
•
•
•
'==========test jacobian for chang,wong ieee_mm 2 pass lowe=================='
clear
% a1=yaw, a2=pitch, a3=roll,
% t1=translation in x, t2=translation in y, t3=translation in z,
syms R dR M TT XYZ ZZ x y z f u v a1 a2 a3 t1 t2 t3 aa1 aa2 aa3 tt1 tt2 tt3
•
•
•
•
•
•
R=[1 -aa3 aa2
aa3 1 -aa1
-aa2 aa1 1];
dR=[1 -a3 a2
a3 1 -a1
-a2 a1 1];
•
•
•
•
•
•
•
•
•
•
•
•
M=[x;y;z];
TT=[tt1;tt2;tt3];
dt=[t1;t2;t3]
% XX=(dR.*R)*M+TT; %not correct, becuase R is a matrix multiplication transform
XYZ=dR*R*M+TT+dt; %correct, becuase R is a matrix multiplication transform
% XX=(dR+R)*M+TT; %not correct becuase R is not an addition transform
u=f*XYZ(1)/XYZ(3);
v=f*XYZ(2)/XYZ(3);
%diff (u,a3)
%diff (v,a3)
ja=jacobian([u ;v],[a1 a2 a3])
jt=jacobian([u ;v],[t1 t2 t3]) Bundle adjustment– structure
reconstruction V5a
42
Delaunay algorithm for
generation of VRML files
• VRML specifications
– Viewers
• Cortona3d, Cosmoplayer, Vivaty
• http://cic.nist.gov/vrml/vbdetect.html
• Delaunay algorithm
Bundle adjustment– structure
reconstruction V5a
43
Alternative method fro finding the
model
• To find model by triangulation (not iterative
method )
• It is faster but may be not very accurate.
Bundle adjustment– structure
reconstruction V5a
44
Alternative method for
SFM2 : find model phase
There are two methods:
(SFM2: method A) Direct triangulation
(SFM2: method B) Iterative method (in the main body
of this power point)
Either (A) or (B) can be used
Bundle adjustment– structure
reconstruction V5a
45
SFM2(method A): direct triangulation
model finding procedure
• Assume you have m views,
• Using the first view and each of the other views we can (m-1)pairs of
images.
• Each pair gives one version of X (using the triangulation method in the
chapter on stereo (chapter iv08
http://www.cse.cuhk.edu.hk/%7Ekhwong/www2/cmsc5711/iv08_stereo.
ppt)
• So you have m-1 models X1, X2,… Xm-1 (all referring to the first camera
coordinate system as reference)
• The solution X=Xmean is the mean of all these (X1, X2,… Xm-1 )
• So a temporally model X is found at this stage.
– Also measure the error:
– Measurement error (Err)
– Err=||(current model
- adjustment–
previousstructure
model)||2
Bundle
reconstruction V5a
46
SFM2 (method B) :The iterative steps
• Initialize first guess of model and pose
• The first guess is a flat model perpendicular to the image and is
Zinit away (e.g. Zinit = 0.5 meters or any reasonable guess)
• Iterative while ( Err is small)
• {
–
–
–
–
–
SFM1 find pose
SFM2 find model
Measurement error (Err)
Err=  ||(current model - previous model)||2
//If the model is stabilized, the solution is final.
• }
Bundle adjustment– structure
reconstruction V5a
47
Pose estimation result
• Recall of SFM1 (pose estimation)
– Input :
• Image sequence I1,I2,…Im.
• There are N features in the 3D object .
• Each image has n image feature points
– Output: pose [R(t),T(t)] t=1,…m
θ found  Pose for finding R, T   T1 T2
T3 1 2 3  T
– At time t, each image feature will give you a vector vt
from the camera center O(t) to the 3D point in X passing
the image point xi,t
Xi,t=[ui,vi]tT
Camera center
O(t)
Bundle adjustment– structure
Image
reconstruction V5a
Xi in 3D
vt
48
After pose is found in SFM1
•
•
•
•
•
We can use triangulation to find the model
Example fro an 3D feature X
After pose estimation SFM1, v1,v2,vm can be found
0t=camera centers at time t
R(t),T(t)=pose at time t
X
v1
[u,v]1
v2
[u,v]2
O1 Image
R1,T1t=1
Image
O2 t=2
R2,T2
Bundle adjustment– structure
reconstruction V5a
v3
vm
[u,v]3
… [u,v]m
Image
t=3
O3
R3,T3
Image
t=m
Om
Rm,Tm
49
Camera motion
SFM2(method A): From vectors
find the closes point
• So you have v1,v2,..vm vectors in 3D
• You want to find a point closes to this point
• So hew to find the closest point between 2 vectors? Of
the first sand second views
– Recall: we learned this in stereo vision
– We know
• P1,P2 (projection matrices) of two cameras, (yes we know it
here because we have guess solution R,T)
• We know the 2D correspondences points (yes we know it here)
• We can find the model point X in 3D .
Bundle adjustment– structure
reconstruction V5a
50
SFM2(method A): Just concentrate on
the first two views
• Find X from 2 views
– From RT found (SFM1 pose
finding phase), we have
P1,P2.
– We also have 2D point
correspondences:
[u,v]1,[u,v]2
– We can find X, see the next
two slides
X
Projection matrix
P1
v1
[u,v]1
v2
[u,v]2
O1 Image
R1,T1t=1
Image
O2 t=2
Projection matrix
R2,T2
P2
Bundle adjustment– structure
reconstruction V5a
51
SFM2(method A): Recall: Triangulation to find X
X is the point at a minimum
distance between two vectors
{O1,(x1,y1)} and {O2,(x2,y2)}
•
(x1,y1)
Left epipolar line
Left
Frame
plane1 1
Ol
Focal
length=f1
(x2,y2)
O2
e2
Epipole e2
e1
Epipole e1
F
Bundle adjustment– structure
reconstruction V5a
Focal
length=f252
SFM2(method A): Recall:
3D reconstruction: from P1 and P2 find 3D points X by
triangulation (p.312[1A],p297[1B])
•
x1  P1, x2  P2  for 2 cameras (P1,P2 )
x1  x1  x1  P1  0
after_some_manipulations
x ( p  )  ( p  )  0      (i )
3T
1T
y ( p 3T  )  ( p 2T  )  0      (ii )
x ( p 2T  )  y ( p1T  )  0      (iii )
P( 3 x 4 )
 p11
  p21
 p31
p12
p22
p32
p13
p23
p33
p14   P1T 


p24    P 2T 
p44   P 3T 
where P iT  i th row of P
i.e. P1T   p11
p12
p13
p14 ,
 p31 
 p11 
 p21 
p 
p 
p 
12 
22 
1
2
3


p 
,p 
, p   32 
 p33 
 p13 
 p23 
 
 
 
 p14 
 p24 
 p34 
p iT  i th row of P, use (i) and (ii) for 2 cameras (P1,P2 )
 x1 p13T  p11T 

2T 
3T
p

p
y
1 
  0, solve by SVD [appendix 1]
A   1 13T
1T
 x2 p2  p2 

2T 
3T
p

p
y
2 
 2 2
Bundle adjustment– structure
reconstruction V5a
53
References
1. D.G. Lowe, “Fitting Parameterized ThreeDimensional Models to Images”, IEEE Pattern
Analysis and Machine Intelligence, Volume:
13 Issue: 5 , May 1991 Page(s): 441 -450
2. Michael Ming Yuen Chang and Kin Hong
Wong, "Model reconstruction and pose
acquisition using extended Lowe's method",
IEEE Transactions on Multimedia, Volume:
7, Issue: 2, April 2005.
Bundle adjustment– structure
reconstruction V5a
54
Answers
•
Bundle adjustment– structure
reconstruction V5a
55
Answer11.1: Exercise11.1:
Given ui  g u (1 , 2 , 3 , T1 , T2 , T3 , X i , Yi , Z i ) 
r11 X i  r12Yi  r13Z i  T1
X iR
X i  3Yi  2 Z i  T1
 f
 f R  f
r31 X i  r32Yi  r33Z i  T3
Zi
 2 X i  1Yi  Z i  T3
so r11  1, r12  3 , r13  2
r31  2 , r32  1 , r33  1
Find
 r11 X i  r12Yi  r13Z i  T1 
 X iR 
  f R 
 f
r

r31 X i  r32Yi  r33Z i  T3 
Zi 
ui
r
X


13
33
i



 f R 
2
R
 Zi

Z
Z
Z
Z
i


 
•
Bundle adjustment– structure
reconstruction V5a
56
Answer11.2: Exercise11.2: proves all these partial
derivatives:
Left for students’ exercises
r
r31 X i 
ui
11


 f
R
R 2 

Z
X i
Z
i

 i
r
r32 X i 
ui
12


 f
R
R 2 

Z
Yi
Z
i
i


r
r33 X i 
ui
13


 f
R
R 2 

Z
Z i
Z
i
i


r
r31 X i 
vi
21


 f
R
R 2 

Z
X i
Z
i
i



r
X
r
vi
i
32
22

 f R 
2
R
 Zi
Yi
Z i 

 
 
 
 
 
r
r33 X i 
vi
23


 f
R
R 2 

Z
Z i
Z
i

 i
•
 
Bundle adjustment– structure
reconstruction V5a
57
• Answer 11.3: Exercise 11.3 : If the
model is a checker plane, each
square is 1cm2.. It is
perpendicular to the camera
principal axis and at Z=0.5 meters
away from the camera center.
• X4
Pixel width is 5um.
• Find the 3D positions of X1,X2,X3
and X4 in pixels
• Answer:
• All Z are the same
Z=0.5meters/5um=100,000
• X1(-2cm, -2cm,0.5 meters)= [4000,-4000,100,000]
• X2(-2cm, -1cm,0.5 meters)=[4000,-2000,100,000]
• X3(0,0,0.5 m)=[0,0,100,000]
X
• X4(2cm,2cm,0.5m)=[4000,4000,1
00,000]
Bundle adjustment– structure
reconstruction V5a
Exercise
1cm
Y
1cm
x3=
[0,0]
X2
X1
58
Answer11.4: Revision for pose estimation SFM1: Exercise 11.4 : Point out which are
know variables and unknown variables in this page.
For each frame t, we have N correspond ence points, i  1, , N, stack the matrix relations
Answer :
~
 1  1 
~ ~ ~ ~ ~ ~

 ui 1  u~i 1  
~
Gussesed 1 , 2 , 3 , T1,T2 ,T3  [0,0,0,0,0,0], so they are knwon
2  2 

~ 
j
 ei 1 


Assume the initial model is a plane (see previous slide)
v

v
i

1
3  ~3 
  i 1 i 1  




E2 N 1   : 

:
 : 
so Model [ X i , Yi , Z i ] is knwon,


~
~
T

T
~ ~ ~ ~ ~ ~
ei  N  2 N 1  ui  N  ui  N  
 ji  N  2 N 6  1 ~1 
J depends on 1 , 2 , 3 , T1,T2 ,T3 and [ X i , Yi , Z i ]





~
T T
 2 ~2 
  vi  N  vi  N   2 N 1
So J is known.
T3  T3  61  ,  ,  , T ,T ,T are knowns and to be found.
1
2
3 1 2 3
R
R
R


Yi X i
X i X i  Zi Zi
Yi
X iR 
f

f
f

f
0

f


Z iR Z iR
Z iR 2
Z iR 2
Z iR 2      ~ 

1
1


YiYi R  Z i Z iR
Yi R X i
Xi
Yi R 
f

~

f
f
f
0

f
 


Z iR
Z iR
Z iR 2
Z iR 2
Z iR 2  i1  2  ~2 
 

E2 N 1  
:
 3 ~3   (6)
 

 T1  T1 
YXR
X X R Z ZR
Y
X iR 
f
   f i i2

f i i 2 i i  f iR
0

f
T  T~ 
R
R
R
R 2
Z
Z
 

Z i 
Z i 
Z i  
i
i
 2 ~2 
 

Y Y R  Z i Z iR
Yi R X i
Xi
Yi R 
f
T3  T3  61
  f i i

f
f
0

f

2
2
2
Z iR
Z iR
 



Z iR 
Z iR 
Z iR   i  N 

 2 N 6
~
 1  1 
At time t,

~
2  2 
there are
 ji 1 
~

 
N features
Put the Jacobain J   : 
,    3 ~3 
T

T
 1 1
 ji  N  2 N 6
T  T~ 
 2 ~2 
T3  T3  61
•
E2 N 1  J 2 N 6 *  61
Bundle adjustment– structure
                reconstruction
       V5a
          (7 )
59
Answer 11.5 Revision for pose estimation SFM1: Exercise 11.5 :
Explain why E is known here in the following formula.
If we have N correspond ence points, i  1, , N, stack the matrix relations
Answer :
~
 1  1 
~ ~ ~ ~ ~ ~
Gussesed 1 , 2 , 3 , T1,T2 ,T3  [0,0,0,0,0,0], so they are knwon

 ui 1  u~i 1  
~
2  2 

~ 
j
Assume the initial model is a plane (see previous slide)
 ei 1 


v

v
i

1
3  ~3 
  i 1 i 1  




so Model [ X i , Yi , Z i ] is knwon,
E2 N 1   : 

:
 : 


~
~
T

T
u

u


~ ~ ~ ~ ~ ~


 iN
iN 
ei  N  2 N 1
 ji  N  2 N 6 1 ~1
J
depends
on

1 , 2 , 3 , T1 ,T2 ,T3 and [ X i , Yi , Z i ]



~ 
T

T
v

v
2
2

  i  N i  N   2 N 1
So J is known.
~
T3  T3  61  ,  ,  , T ,T ,T are knowns and to be found.
1
2
3 1 2 3
~
 ui 1  ui 1  

~ 
  vi 1  vi 1  
ui 
Answer : In 
:
 ,    measured by the KLT feature tracker for all i
vi 
 ui  N  u~i  N  

~ 
v

v
i

N
iN 


 2 N 1
u~i 
~ ~ ~ ~ ~ ~ ~
~
~ ~ ~
v~   found by gussed Model M i  [X i ,Yi ,Z i ]and pose  [1 , 2 , 3 , T1 , T2 , T3 ]
 i
ui 
u~i 
~
~
using the following formulas for      ; M i    M i ;      ~ 
vi 
vi 
r X  r Y  r Z T
X i  3Yi  2 Z i  T1
ui  f 11 i 12 i 13 i 1  f
       ( 2a )
r31 X i  r32Yi  r33Z i  T3
 2 X i  1Yi  Z i  T3
vi  f
r21 X i  r22Yi  r23Z i  T2
 X  Y  1Z i  T2
 f 3 i
       (2b)
r31 X i  r32Yi  r33Z i  T3
 2 X i  1Yi  Z i  T3
Bundle adjustment– structure
reconstruction V5a
•
60
Answer11.6 : Revision for pose estimation SFM1: Identify known variables and unknown
variables when k=0 and k=5 in this iterative pose estimation algorithm
Ans: K=0, we initialize (GUESS)  k=0 then E,J, can be found. Then k=0 can be calculated,
then the guessed k=1 = k=0+k=0 is found, which will be used in the iteration k=1.
When K=5, guessed  k=5 is found during k=4 , then E,J,  k=5, are calculated, i.e.
 k=5 =k=4 + k=4 , when k=4. Then k=5 can be found and it will be used to find
guessed  k=6 =k=5 + k=5
E  J *  ,
so J -1 * E  Δθ
  1
2
3
T1
T2
T3 
T
~
Base on the first guess  k 0 , we can find
~
1
 k 0  J k 0 ( ) * Ek 0
SFM1:
This
algorithm
is to find
the pose

Iterate (k  0 ,k  K _ max ,k  k  1 )
{
~
find Ek and J ( ) // if no J -1 use pseudo inverse
~
 k  J k10 ( ) Ek
1
k 0
Break if  k is small enough
~
~
Next guess is  k 1   k   k
}
•
Bundle adjustment– structure
reconstruction V5a
Note :
  1 , 2 , 3 , T1 , T2 , T3 
 is the pose that we want to find
~
~
~ ~ ~ ~ ~
  1 , 2 , 3 , T1 , T2 , T3

~
  guessed pose
~
 1  1 

~



 2 2
3  ~3 
  
~
T

T
 1 1
T  T~ 
 2 ~2 
T3  T3  2 N6
61
Exercise11.7: Answer11.7: At K=0:[X,Y,Z]i are unknown. Others are known, because pose is found by
SFM1, Model Mi is guessed initially at k=0 (a plane)
AT K=5: [X,Y,Z]i is unknown.Others are known,because pose is found by SFM1, Model(Mi(xi,y,zi)) is
(a better guess) found by the previous SFM2 phase
J(m) depends on the guessed model M and current pose (current pose is found by SFM1)
Measured
Result from the guessed model and given pose
~
~
 X i  X i   g u ( i , M )
~
ui ,t  ui ,t 
~  
(m) 
X ~
e

j
Y

Y
~ 
 i i    g ( , M
v

v
)
 i ,t i ,t 
 Z i  Z~i   v i

 t 
X
Jacobian
~
g u ( i , M )
Y ~
g v ( i , M )
Y
~
g u ( i , M )   X  X~ 
i
i

Z ~   Y  Y~     (5)
i
i 
g v ( i , M )  
 Z i  Z~i 

 t 
Z
New Guessed
model
• The formulas apply to one
feature (i) for all time
frames t=1,2,….
• SFM2 will N times , each
time is independent.
• See next slide for the
graphical illustration,
Current Guessed model
If we have Γ frames, i.e. t  1,2 ,Γ , stack the matrix relations for t  1,2 ,..Γ
~
~
~
  g u ( i , M

) g u ( i , M ) g u ( i , M ) 



X ~
Y ~
Z ~ 


~

 ui  ui  

g
(

,
M
)

g
(

,
M
)

g
(

,
M
)


v
i
v
i
v
i


~
~
(m)

~ 
Xi  Xi 

j
i 1 
 et 1 

 Xi  Xi 
v

v



X

Y

Z
 i i  t 1 


t

1

~ 

 
  Y  Y~   (6)
E   : 

:
  :   Yi  Yi   
:


  i ~i 
~
~
~
~
 g u ( i , M ) g u ( i , M ) g u ( i , M ) 
 j ( m ) i    Z i  Z~i 
et   2 1  ui  ui  

  Z i  Z i 

 2 3 
 31



31
~ 


v

v
X
Y
Z


~
~
~
  i i  t   2 1
 g ( , M ) g ( , M ) g ( , M )

v
i
v
i

  v i

X
Y
Z
 t  
 
2 3
~
(m)
Xi  Xi 
 j t 1 

~ 


Bundle adjustment– structure
Put the Jacobian for the model J ( m ) 2 3   :  , M   Yi  Yi 
reconstruction V5a
 Z i  Z~i 
 j ( m ) t  




•
2 3
E2 1  J
(m)
2 3
31
* M 31                                    (7)
62
SFM2: Algorithm to find the i-th model point (repeat this N time to get all points)
Exercise11.8 : identify which are known and unknown when K=0, K=5
Answer11.8: k=0, Model [X,Y,Z]i is a point in a plane, pose (by SFM1) is known
and the algorithm can find a better model.
Answer: k=5, Model [X,Y,Z]iis the model found in the previous SFM2 and pose is
found by SFM1, the algorithm can find a better Model.
E  J ( m ) * M , so J -1 * ke  ΔM
M  X
Note :
Z 
Y
T
~
Base on the first guess M k 0 , we can find
Ek  0  J
(m)
* M k 0
Iterate (k  0 ,k  K _ max ,k  k  1 )
SFM2: This
algorithm is
to find the
model M
{
find E k and J k-1 // if J is not a square matrx, use pseudo inverse

M k  J ( m )

1
~
Xi  Xi 

~ 
M   Yi  Yi 
 Z i  Z~i 


M i  [ X i , Yi , Z i ]T  unknwon
M i  is the model we want to find
~
~ ~ ~
M i  [ X i , Yi , Z i ]T  Guessed model
~
M i at _ k 0  a plane at k thiterative
~
M i at _ k  model found by the previous SFM2
 
 
Ek
Break if M k is small enough
~
~
~
Next guess is M k 1  M k  M k
}
~
After the end of the above loop M k is the result
Bundle adjustment– structure reconstruction V5a
•
63