COMPUTER VISION PROBLEM SET 4
For camera calibration we need at least 10 test patterns.​The important input data needed for
calibration of the camera is the set of 3D real world points and the corresponding 2D
coordinates of these points in the image. 2D image points are ​locations where two black
squares touch each other in chess boards. Images were taken from a static camera and
chess boards are placed at different locations and orientations. So we need to know (X,Y,Z)
values. But for simplicity, we can say chess board was kept stationary at XY plane, (so Z=0
always) and camera was moved accordingly. This consideration helps us to find only X,Y
values.The results we get will be in the scale of size of chess board square. To find pattern
in chess board, the function ​cv.findChessboardCorners() was used. In this part, 12x12
grids were used. It returns the corner points and ret which will be true if pattern is obtained.
Once the corners were found, their accuracy were increased by using cv.cornerSubPix().
Also the pattern was drawn using ​cv.drawChessboardCorners()​.
Part a.
After completing above steps, the object points and image points were obtained for
calibration part. The function, ​cv.calibrateCamera() was used which returns the camera
matrix, distortion coefficients, rotation and translation vectors etc. For undistorting the
images, the camera matrix was refined based on a free scaling parameter using
cv.getOptimalNewCameraMatrix()​. By calling the function ​cv.undistort()​. Images with
patterns drawn on the chess boards and undistorted versions of it are shown below:
(a)
(b)
Figure 1:​(a) Image 11 with pattern, (b) Undistorted image of image 11
​ (a)
(b)
Figure 2:​(a) Image 13 with pattern, (b) Undistorted image of image 13
(a)
(b)
Figure 3:​(a) Image 14 with pattern, (b) Undistorted image of image 14
(a)
(b)
Figure 4:​(a) Image 15 with pattern, (b) Undistorted image of image 15
(a)
(b)
Figure 5:​(a) Image 16 with pattern, (b) Undistorted image of image 16
(a)
(b)
Figure 6:​(a) Image 2 with pattern, (b) Undistorted image of image 2
(a)
(b)
Figure 7:​(a) Image 3 with pattern, (b) Undistorted image of image 3
(a)
(b)
Figure 8:​(a) Image 4 with pattern, (b) Undistorted image of image 4
(a)
(b)
Figure 9:​(a) Image 6 with pattern, (b) Undistorted image of image 6
(a)
(b)
Figure 10:​(a) Image 7 with pattern, (b) Undistorted image of image 7
Distortion parameters are:
[[-2.47495930e-01 7.03320949e-02 -1.94383447e-04 -2.82181439e-04 1.09232915e-01]]
Part b.
Camera matrix maps the 3D world scene into the image plane. The calibration algorithm
calculates the camera matrix using the extrinsic and intrinsic parameters. The intrinsic
parameters represent the optical center and focal length of the camera. Hence, the intrinsic
matrix consists the related parameters which are used to mapped the camera coordinates
into the image plane. In addition to this, the focal length and optical centers can be used to
create a camera matrix, which can be used to remove distortion due to the lenses of a
specific camera. Once, the camera matrix is calculated, it can be reused on other images
taken by the same camera. In this project, the camera matrix is denoted by the word “mtx” in
the code script. Previously obtained undistorted images were used to create the intrinsic
matrix.
The obtained intrinsic matrix,K is shown in below:
Intrinsic matrix, K:
[[659.03841658 0.
[ 0. 659.79518135
[ 0.
0.
1.
302.37834068]
243.69565429]
]]
Since, the general intrinsic matrix of a camera can be written as shown in below.
[ [ f​x​ s o​x​ ],
[ 0 f​y​ o​y​],
[ 0 0 1] ]
The obtained results for focal length of the camera system are 659.03841658 and
302.37834068, for x direction and y direction, respectively. The optical center, namely
principal points, values of the camera are 302.37834068 and 243.69565429 for x and y
planes, respectively. The skewed parameter for this camera is 0 as it can be seen from the
resulted matrix.
Part c.
Camera projection values can be obtained by doing projection from homogenous world
coordinates X​i​=[X​i​, Y​i​, Z​i​, 1] to non-homogenous image coordinates x​i​, y​i​. In order to obtain
rotation matrix, the function ​cv2.Rodrigues()​ was used by taking parameters as rotation
vector (rvecs) and output rotation matrix which is rotation_mat (3x3), respectively. It converts
rotation vector to a rotation matrix. Projection matrix estimation was done by multiplying the
camera matrix (mtx) with both rotation and translation matrices. The multiplication is
achieved by using the function ​np.matmul()​. In the end, the projection matrix was applied for
each image and the obtained matrices are shown below.
Projection matrix for Image11:
[[-6.25700311e+01 6.58169054e+02 2.97762964e+02 5.16412209e+03]
[-6.89234709e+02 9.36674787e+01 -1.04399100e+02 8.20858831e+03]
[-4.83731299e-01 1.75587597e-02 8.75040411e-01 2.88813307e+01]]
Projection matrix for Image13:
[[ 6.94535079e+02 4.98439371e+01 2.02239712e+02 2.60324781e+03]
[ 7.73984726e+01 6.95288786e+02 -7.28027006e+01 1.25704977e+03]
[ 1.44872107e-01 4.27788934e-01 8.92193197e-01 1.82063387e+01]]
Projection matrix for Image14:
[[ 6.96740298e+02 5.07940015e+01 1.94260705e+02 2.25628653e+03]
[ 7.00500162e+01 6.97750169e+02 -5.43594481e+01 9.61265405e+02]
[ 1.53095002e-01 4.04145025e-01 9.01791949e-01 1.64029218e+01]]
Projection matrix for Image15:
[[7.08519339e+02 1.00270878e+01 1.53831398e+02 4.37709698e+02]
[7.83583451e+01 6.95181333e+02 7.28020457e+01 8.91570324e+02]
[2.17436566e-01 2.26615960e-01 9.49403258e-01 1.58789894e+01]]
Projection matrix for Image16:
[[ 6.42170054e+02 -1.89552627e+02 2.78301490e+02 6.69609181e+03]
[ 1.36439288e+02 5.92547685e+02 3.53537572e+02 1.89216865e+03]
[-7.35557162e-03 -1.74507293e-01 9.84628407e-01 2.32496955e+01]]
Projection matrix for Image2:
[[ 6.86368134e+02 -1.49953635e+02 1.79379451e+02 4.26705203e+03]
[ 6.31384705e+01 4.44888967e+02 5.41114215e+02 2.63485423e+03]
[ 1.06207011e-01 -5.07402773e-01 8.55138876e-01 2.53173792e+01]]
Projection matrix for Image3:
[[ 6.40265318e+02 -2.56885555e+02 2.23236263e+02 5.10231560e+03]
[ 1.02015534e+02 4.37632775e+02 5.41098548e+02 2.44456387e+03]
[-3.07997070e-02 -5.03531351e-01 8.63427794e-01 2.59124456e+01]]
Projection matrix for Image4:
[[-3.30475480e+02 -4.63631502e+02 4.48994525e+02 1.15624922e+04]
[ 3.99291414e+02 1.07463387e+02 5.68977363e+02 1.71913319e+03]
[-5.37651411e-01 4.32172367e-01 7.23987573e-01 2.12578041e+01]]
Projection matrix for Image6:
[[ 6.89604083e+02 1.09526845e+02 1.95484972e+02 1.23635325e+03]
[ 8.04012094e+01 6.45233991e+02 -2.68190222e+02 1.85456777e+03]
[ 1.24339614e-01 6.67385353e-01 7.34259117e-01 1.48719271e+01]]
Projection matrix for Image7:
[[ 5.42369816e+01 -6.94493188e+02 -2.01250732e+02 1.02772313e+04]
[ 6.82504935e+02 -3.54730383e+00 -1.69975539e+02 1.89875715e+03]
[ 5.12021524e-01 -5.79216694e-01 6.34304328e-01 2.11005085e+01]]
Part d and e.
Two images of a single scene/object are related by the epipolar geometry, which can be
described by a 3x3 singular matrix called the essential matrix if images’ internal parameters
are known. Essential Matrix contains the information about translation and rotation, which
describe the location of the second camera relative to the first in global coordinates.
Methods for computing the essential matrix:
•
•
•
•
CV_FM_7POINT for a 7-point algorithm.
CV_FM_8POINT for an 8-point algorithm.
CV_FM_RANSAC for the RANSAC algorithm.
CV_FM_LMEDS for the LMedS algorithm.
The findEssentialMat function that estimates essential matrix based on one of four methods
listed above and returns the number of fundamental matrices and the recoverPose() function
that is to recover the relative pose between cameras are used for the d and e sections of this
problem.
Estimation matrix between the first image and Image13:
[[-0.22382479 0.2175967 -0.35652612]
[ 0.2563865 -0.23081538 0.39561683]
[-0.44997158 -0.54416192 -0.03652653]]
Rotatiton matrix between the reference image and Image13:
[[ 0.01647325 -0.84883246 -0.52840523]
[ 0.87224277 0.27055219 -0.40742369]
[ 0.48879564 -0.45418605 0.74484485]]
Translation matrix between the reference image and Image13:
[[0.7422173 ]
[0.67004112]
[0.01258494]]
Estimation matrix between the first image and Image14:
[[-0.6823118 0.02665764 -0.11362726]
[-0.03568162 -0.67281911 -0.14475807]
[ 0.1377457 -0.16385187 -0.00968877]]
Rotatiton matrix between the reference image and Image14:
[[-0.0388547 -0.99924425 -0.00111308]
[ 0.99707483 -0.03869692 -0.06591147]
[ 0.06581859 -0.0036708 0.99782485]]
Translation matrix between the reference image and Image14:
[[-0.20410204]
[ 0.2239927 ]
[-0.95297934]]
Estimation matrix between the first image and Image15:
[[-0.69727166 -0.01096343 -0.09071658]
[ 0.01815938 -0.67934301 -0.1546849 ]
[ 0.07748684 -0.11578735 -0.01695628]]
Rotatiton matrix between the reference image and Image15:
[[ 0.030878 -0.99260736 -0.11737622]
[ 0.99856122 0.03578565 -0.0399359 ]
[ 0.04384105 -0.1159742 0.99228421]]
Translation matrix between the reference image and Image15:
[[-0.1045231 ]
[ 0.16873587]
[-0.98010363]]
Estimation matrix between the first image and Image16:
[[ 0.03680922 0.03961399 -0.0665873 ]
[ 0.08711609 0.39525761 -0.57407851]
[ 0.68833565 -0.1595297 -0.00152346]]
Rotatiton matrix between the reference image and Image16:
[[-0.07713807 -0.83753721 -0.54090771]
[ 0.98963567 -0.13022807 0.06051359]
[-0.12112375 -0.53063366 0.83890223]]
Translation matrix between the reference image and Image16:
[[ 0.99261467]
[-0.11503103]
[-0.03852237]]
Estimation matrix between the first image and Image2:
[[ 0.11042379 0.30016237 -0.40150511]
[ 0.25444307 0.15036382 -0.4132761 ]
[ 0.48551819 -0.48957291 -0.05427294]]
Rotatiton matrix between the reference image and Image2:
[[ 0.1314304 -0.81097276 -0.57013089]
[ 0.86545195 -0.18660481 0.46494255]
[-0.4834449 -0.55452847 0.67733241]]
Translation matrix between the reference image and Image2:
[[ 0.68775401]
[-0.69548876]
[ 0.20806202]]
Estimation matrix between the first image and Image3:
[[-0.65573398 -0.06677811 0.20953959]
[ 0.05979714 -0.64613134 -0.17379257]
[ 0.12705166 0.2326095 0.01149908]]
Rotatiton matrix between the reference image and Image3:
[[ 0.08828447 -0.99525527 -0.04089986]
[ 0.99600885 0.0876614 0.0167884 ]
[-0.0131234 -0.04221878 0.9990222 ]]
Translation matrix between the reference image and Image3:
[[-0.20807112]
[-0.31220102]
[-0.92695034]]
Estimation matrix between the first image and Image4:
[[ 0.14776158 0.3834517 0.47934594]
[-0.34094535 -0.05983848 -0.22762466]
[ 0.55951865 -0.33115636 -0.08364072]]
Rotatiton matrix between the reference image and Image4:
[[-0.96276001 0.03604382 0.26794402]
[ 0.02459065 -0.97529046 0.21955366]
[ 0.2692368 0.21796641 0.93808432]]
Translation matrix between the reference image and Image4:
[[-0.45024167]
[-0.81038588]
[-0.37490953]]
Estimation matrix between the first image and Image6:
[[ 0.65679182 -0.09398693 0.20253942]
[ 0.0968201 0.63113145 0.15546033]
[-0.10067693 0.27634298 0.01996243]]
Rotatiton matrix between the reference image and Image6:
[[-0.1326238 -0.99072236 -0.02966692]
[ 0.98755634 -0.12952924 -0.08918884]
[ 0.08451865 -0.04112632 0.99557281]]
Translation matrix between the reference image and Image6:
[[-0.19374594]
[ 0.36913694]
[-0.90895568]]
Estimation matrix between the first image and Image7:
[[ 0.2051783 -0.63578388 -0.01036002]
[ 0.64006021 0.22230658 -0.19930799]
[ 0.10410831 -0.20909824 -0.01384986]]
Rotatiton matrix between the reference image and Image7:
[[-0.95084162 -0.30608657 -0.04702374]
[ 0.3087333 -0.94879911 -0.06681311]
[-0.02416549 -0.07804648 0.9966568 ]]
Translation matrix between the reference image and Image7:
[[ 0.32733285]
[ 0.04855996]
[-0.9436605 ]]