3D computer vision
techniques
KH Wong
3D computer vision techniques v.4b2
1
Seminar Title: 3D computer
vision techniques.

Abstract
In this talk, the ideas of obtaining 3D information of objects (or
called 3D reconstruction) using different techniques are
discussed. Currently, the most popular one is the image based
method that uses 2D cameras for 3D reconstruction; in particular
reconstruction based on one-image, two-image and multipleimage are discussed. Moreover, batch and sequential treatments
of input data are studied. I will also talk about novel techniques,
such as using multiple cameras and laser based methods to
obtain 3D information. And I will discuss how 3D computer vision
is used in film and game production. Finally naked-eye 3D
display technologies will be mentioned.
3D computer vision techniques v.4b2
2
Overview (part1)


Introduction
From 2D to 3D



Camera systems/calibration
Feature extraction/correspondence
Reconstruction algorithms



Previous projects



Virtual viewer/ Projector camera systems
Keystone correction
Novel setups


2 views, 3 views , N views
Real-time algorithms/Kalman filter
Multiple cameras/ Camera array
Obtain 3D directly




Structured light
Laser approach
Kinect approach
Photometric stereo
3D computer vision techniques v.4b2
3
Overview (part 2)

Applications



Photos from tourists (photo tourism)
http://phototour.cs.washington.edu/
3D displays
Possible future research



Classification based on 3D information
Content search 3D based on 3D keys
Merging with sound information
3D computer vision techniques v.4b2
4
Motivation



We live in a 3D world
We see 2D images but perceive the world in
3D
Intelligent robot should have this 3D
reconstruction capability
3D computer vision techniques v.4b2
5
How to obtain 3D information?


Cameras-2D
Range sensors-3D
3D computer vision techniques v.4b2
6
Challenges

Obtain 3D information for tasks in a 3D world.



2D-to-3D reconstruction from a camera
3D directly— laser range sensor, kinect sensor
Novel sensors



Camera array/ multiple camera
One pixel camera
light field camera
3D computer vision techniques v.4b2
7
2D-to-3D reconstruction
(feature based method)



Camera (perspective projection)
Features-extraction and correspondences
Methods




One-image method
Two-image (Stereo) method
Three-image method
N-image method


Bundle adjustment
Kalman filter
3D computer vision techniques v.4b2
8
http://upload.wikimedia.org/wikipedia/en/8/81/Pinhole-camera.png
Camera: 3D to 2D projection
Perspective model
u=F*X/Z (nonlinear relation)
v=F*Y/Z
Virtual
Screen
or CCD
sensor
Y
Pinhole
Camera
World
center
v
Z
3D computer vision techniques v.4b2
F
F
Thin lens
or a pin hole
Real
Screen
Or
CCD
9
sensor
Perspective
Projective

Zw
Yw
World
Coordinates
Rc,Tc
Xw
Model M at t=1
v-axis
X,Y,Z
image
Xc-axis
(u,v)
Zc-axis
Principal axis
u-axis
Camera
Coordinates.
Oc=
c (Image center, (0,0,0)
ox,oy)
(Camera
center)
F=focal
length
Yc-axis
3D computer vision techniques v.4b2
(0,0) of image plane
10
In paintings

Western

Fresco by Raphael, 1510 1511, Stanza della Signatura,
Vatican Palace, Rome.

Chinese
《富春山居圖》
是元朝畫家黃公望
的作品，創作於
1347年至1350年
Dwelling in the
Fuchun
Mountains (富春
山居圖) by Huang
Gongwang
(1269–1354)


3D computer vision techniques v.4b2
11
http://www.es.flinders.edu.au/~mattom/science+society/lectures/illustrations/lecture17/schoolathens.html
http://jsl641124.blog.163.com/blog/static/17702514320115219508530/
Feature correspondences
--Camera moved, find correspondences for neighboring images
--We can use feature to identify the motions of projected 3D
features in 2D.

Area a
Image at t=t0
(or left image)
3D computer vision techniques v.4b2
Image at t=t0+dt
(or right image)
12
Demo


Youtube Movie
http://www.youtube.com/watch?v=azlDGK6e1U
3D computer vision techniques v.4b2
13
One-image
2D-to-3D reconstruction
3D computer vision techniques v.4b2
14
One image 2D-to-3D
reconstruction method

Difficult and with ambiguity
3D computer vision techniques v.4b2
http://ai.stanford.edu/~asaxena/reconstruction3d/
15
One image 2D-to-3D

Using prior knowledge (e.g. face)
http://www.wisdom.weizmann.ac.il/~ronen/papers/
Hassner Basri - Example Based 3D Reconstruction from Single 2D Images.pdf
3D computer vision techniques v.4b2
16
Two-image
2D-to-3D reconstruction
3D computer vision techniques v.4b2
17
Two-image 2D-to-3D reconstruction
method: stereo vision

Objectives:


Basic idea of stereo vision
Stereo reconstruction by epipolar geometry


Stereo camera pair calibration (find Fundamental
matrix F)
Construct the 3D (graphic) model from 2 images
Inside a computer
3D computer vision techniques v.4b2
Graphic
model
18
if camera motion is pure translation
: Triangular calculation

Left
Camera
Principle
axis
Object
Right
Camera Px(x,y,z)
Principle
axis
By similar triangle,
w.r.t left camera lens center
x xl'
= ,
z
f
z
Left
Image
plane
X’l
X’r
Right
Image
plane
Focal
Length
f
b (Baseline)
Left camera center
Horizontal
(reference point)
Disparity=xL-xR
( x - b) x'r
=
z
f
f b
elimate x  z = ' '
( xl - xr )
By similar triangle,
w.r.t right camera lens center
3D computer vision techniques v.4b2
One major problem is to locate x’l and x’r The correspondence problem
19
If camera motion is NOT pure translation
: Use Epipolar Geometry
X

Right_image_pointT*E*left_image_point=0
Left side is the reference
Left epipolar line
1
Focal
length=f1
(x2,y2) Right epipolar line
(x1,y1)
left
Frame
Plane-1 1
O
Plane-3 3
Perpendicular to
TX2 or TX1
right Frame
Plane-2 2
O2
e1
R,T
Base line=||T||
3D computer vision techniques v.4b2
e2
Focal
length=f2
20
Method: 8-point algorithm
http://www.cs.manchester.ac.uk/ugt/COMP37111/papers/Hartley.pdf

Find 8 point corresponded (


Map 8 Right_image_points to left_image_point
Solve the epeiolar formula




Right_image_pointT*E*left_image_point=0
Find E.
From E we can find camera R (rotation) ,T
(translation)
From R,T we can find model (3D positions of the
left feature points (using left as reference)
3D computer vision techniques v.4b2
21
An example of stereo
reconstruction


An example
Short-Baseline
Stereo Systems
for Mobile
Devices

http://www.lelaps.de/videos.html#SQx5vU8BA-M
http://www.lelaps.de/projects/stmobile.html
3D computer vision techniques v.4b2
22
Stereo-based Free-space
Estimation

Another example
http://www.lelaps.de/videos.html#VrKBNtAN03o
http://www.lelaps.de/projects/freespace.html
3D computer vision techniques v.4b2
23
Three-image
2D-to-3D reconstruction
3D computer vision techniques v.4b2
24
Three-image 2D-to-3D
reconstruction method


More robust using 3 views
It contains 3 epipolar
relations





Stereo1: view1,2 ,
Stereo2: view2,3,
Stereo3 :view 3,1.
Combine 3 epipolar geometry
information together.
Similar to the algorithm in
epipolar geometry (apply 3
times)
http://www.cs.unc.edu/~marc/tutorial/node45.html
3D computer vision techniques v.4b2
M=3-D model point
M, m’, m” are image points
C,C’,C” are camera centers
25
Example of 3-image
reconstruction

Example
LIBVISO: Feature Matching for Visual Odometry
http://www.youtube.com/watch?v=DPLh6MoxPAk
3D computer vision techniques v.4b2
26
N-image
2D-to-3D reconstruction
(batched method: order of
images can be random )
3D computer vision techniques v.4b2
27
N-image 2D-to-3D
reconstruction method

Bundle adjustment approach


Guess iteratively the solution for 3D to explain the
measurements of feature points in all images
Math: Q(u,v)=g(X), g is nonlinear (projection) because
 u=fX/Z
 v=fY/Z, f=focal length
 Given Q (image measurement) , we want to find X=(X,Y,Z)i
from image points (u,v)i of all N model points (i=1,,,N), g is
the projection formulas
 A typical non linear optimization problem,
 Gauss-Newton for non linear optimization method is used.
3D computer vision techniques v.4b2
28
Batched method: order of images can
be random

From measurement [u,v]I find X
X
v1
[u,v]1
v2
[u,v]2
O1 Image
R1,T1t=1
Image
O2 t=2
R2,T2
v3
vm
[u,v]3
… [u,v]m
Image
t=3
3D computer vision techniques v.4b2
O3
R3,T3
Image
t=m
Om
Rm,Tm
29
Camera motion
Example

Bundle adjustment reconstruction
http://www.cse.cuhk.edu.hk/%7Ekhwong/demo/canyon2b2.mpg
3D computer vision techniques v.4b2
30
N-image
2D-to-3D reconstruction
(Sequential method: order of
images are used like in a move )
3D computer vision techniques v.4b2
31
Sequential method: order of
images are used like in a move

From measurement [u,v]I find X
X
v1
[u,v]1
v2
[u,v]2
O1 Image
R1,T1t=1
Image
O2 t=2
R2,T2
v3
vm
[u,v]3
… [u,v]m
Image
t=3
3D computer vision techniques v.4b2
O3
R3,T3
Image
t=m
Om
Rm,Tm
32
Camera motion
Kalman Filter
Prediction
pictures by Ko Hoi Fung
Correction
3D computer vision techniques v.4b2
33
33
Kalman filter example
t’ = 1:
Prediction
Position = x1’
Velocity = v1’
x1’ = v0 * t + x0
t = 0:
Position = x0
Velocity = v0
States:
• Position
• Velocity
Measurements:
t = 1:
Position = x1
Update
• Position
3D computer vision techniques v.4b2
34
34
Example

Hernan Badino and Takeo
Kanade:
"A Head-Wearable ShortBaseline Stereo System for
the Simultaneous
Estimation of Structure and
Motion".
IAPR Conference on
Machine Vision Applications
(MVA), Nara, Japan, June
2011
http://www.youtube.com/watch?v=SQx5vU8BA-M
3D computer vision techniques v.4b2
35
Novel sensors : Camera array/
Multiple camera systems

Camera array/ multiple camera: High
Performance Imaging - Using Large Camera
Array
http://www.youtube.com/watch?v=0W_1Ce2lTBo
http://graphics.stanford.edu/papers/CameraArray/
3D computer vision techniques v.4b2
36
The Self-Reconfigurable
Camera Array
Each camera
Demo movie
http://chenlab.ece.cornell.edu/projects/MobileCamArray/videos/train.mov
http://chenlab.ece.cornell.edu/projects/MobileCamArray/videos/self_reconfiguration.mov
3D computer vision techniques v.4b2
37
http://chenlab.ece.cornell.edu/projects/MobileCamArray/
Applications
3D computer vision techniques v.4b2
38
Photo tourism

http://phototour.cs.washington.edu/
3D computer vision techniques v.4b2
39
Projector-camera
system
Application of computer vision
3D computer vision techniques v.4b2
40
A Projector-Camera system
3D computer vision techniques v.4b2
41
Projector-Camera calibration

http://www.youtube.com/watch?v=YHhQSglmuqY&feature=channel_page
3D computer vision techniques v.4b2
42
Our setup

3D computer vision techniques v.4b2
43
Calibration procedure

3D computer vision techniques v.4b2
44
Quadrangle tracking

3D computer vision techniques v.4b2
45
Experiments

3D computer vision techniques v.4b2
46
Projection result

3D computer vision techniques v.4b2
47
Results

3D computer vision techniques v.4b2
48
Hand held direct manipulation
3D Display

http://www.youtube.com/watch?v=vVW9QXuKfoQ&feature=relmfu
3D computer vision techniques v.4b2
49
Keystone correction

Configuration
3D computer vision techniques v.4b2
50
Aim of this work

Desired Results
Keystoned projection
Corrected projection
3D computer vision techniques v.4b2
51
Overview

Three major modules



Projector-camera pair calibration
Projection region detection and tracking
Automatic keystone correction
Projectorcamera pair
calibration
Projection region
tracking
G, K
Keystoned
projection
Keystone correction
3D projection
region recovery
Camera
frame
Flow chart
3D rectangle
Pre-warped image
3D computer vision techniques v.4b2
Corrected
projection
52
Pre-warp projection image
Pre-warped projection image
Display result
http://www.youtube.com/watch?v=y5XYdeh8Bno&list=UUfy2EumiHMeoUorMFR0woZA&index=1&feature=plcp
3D computer vision techniques v.4b2
53
Keystone correction

Some real correction results
3D computer vision techniques v.4b2
54
Obtain 3D directly

Laser range sensor


Time of flight
Kinect
3D computer vision techniques v.4b2
55
Photometric stereo

Lamertian light formula
http://www.wisdom.weizmann.ac.il/~vision/photostereo/
•Given 3 or more known light source we can find the normal N
•From the set of N we can approximate the surface
http://www.taurusstudio.net/research/photex/ps/equation.htm
3D computer vision techniques v.4b2
56
Photometric stereo using multiple cameras
and multiple light sources

Demo
Dynamic Shape Capture using Multi-View Photometric Stereo SIGGRAPH 2009
http://www.youtube.com/watch?v=9hgs5zN38lk
3D computer vision techniques v.4b2
57
Multiple cameras fro human body
reconstruction

3D computer vision techniques v.4b2
Homepage://media.au.tsinghua.edu.cn
58
Experimental Results
3D computer vision techniques v.4b2
59
3D Modeling Using MVML Dome
59
2016/3/19
Multiple camera doom

http://www.mpi-inf.mpg.de/~yliu/
3D computer vision techniques v.4b2
60
Structured light method

Calculate the shape by how the strip is
distorted.

http://www.laserfocusworld.com/articles/2011/01/lasers-bring-gesture-recognition-to-the-home.html
3D computer vision techniques v.4b2
61
Real time Virtual 3D Scanner Structured Light Technology

Demo
http://www.youtube.com/watch?v=a6pgzNUjh_s
3D computer vision techniques v.4b2
62
Time of flight laser method


Send the IR-laser light to
different directions and
sense how each beam is
delayed.
Use the delay to
calculate the distance of
the object point
http://www.laserfocusworld.com/articles/2011/01/lasers-bring-gesture-recognition-to-the-home.html
3D computer vision techniques v.4b2
63
LIDAR light detection and
ranging scanner
http://www.youtube.com/watch?v=MuwQTc8KK44

Leica terrestrial
lidar (light
detection and
ranging) scanner
3D computer vision techniques v.4b2
http://hodcivil.edublogs.org/2011/11/06/lidar-%E2%80%93-light-detection-and-ranging/
http://commons.wikimedia.org/wiki/File:Lidar_P1270901.jpg
64
3D Laser Scanning Underground Mine Mapping

Demo
http://www.youtube.com/watch?v=BZbvz8fePeQ
3D computer vision techniques v.4b2
65
Motion capture for film
production (MOCAP)

IR light
emitter
and
camera
http://www.youtube.com/watch?v=IxJrhnynlN8
http://upload.wikimedia.org/wikipedia/commons/7/73/MotionCapture.jpg
3D computer vision techniques v.4b2
http://www.naturalpoint.com/optitrack/products/s250e/indepth.html
66
3D body scanner

http://www.youtube.com/watch?v=86hN0x9RycM
3D computer vision techniques v.4b2
http://www.cyberware.com/products/scanners/ps.html
http://www.cyberware.com/products/scanners/wbx.html
67
3-D Face capture

http://www.youtube.com/watch?v=-TTR0JrocsI&feature=related
3D computer vision techniques v.4b2
http://www.captivemotion.com/products/
68
Dimensional Imaging 4D Video
Face Capture with Textures

http://www.youtube.com/watch?v=XtTN7tWaXTM&feature=related
Dimensional Imaging 4D Video Face Capture with Textures
3D computer vision techniques v.4b2
69
Kinect


Another structure light
method
Use dost rather than
strips
http://www.laserfocusworld.com/articles/2011/01/lasers-bring-gesture-recognition-to-the-home.html
3D computer vision techniques v.4b2
70
Kinect Hardware

3D computer vision techniques v.4b2
71
See the IR-dots emitted by
KINECT

http://www.youtube.com/watch?v=-gbzXjdHfJA
http://www.youtube.com/watch?v=dTKlNGSH9Po&feature=related
3D computer vision techniques v.4b2
72
Novel sensors : light field camera
Spin off from Stanford camera array


light field camera :
LYTRO camera
Be able to refocus
after the picture is
taken
http://www.youtube.com/watch?v=7QV152jc3Ac
3D computer vision techniques v.4b2
https://www.lytro.com/camera
73
light field camera
How does it work

3D computer vision techniques v.4b2
http://www.quora.com/Lytro/How-does-the-new-Lytro-camera-work
74
3D (Volumetric) display
Rendering for an Interactive 360º Light Field Display
SIGGRAPH 2007 Papers Proceedings

http://www.youtube.com/watch?v=h6aUIS44ezE
http://gl.ict.usc.edu/Research/3DDisplay/
3D computer vision techniques v.4b2
75
Occlusion-Capable Volumetric 3D Display by Cossairt,etal.
Actuality Systems, Inc

http://www.youtube.com/watch?v=8KaQmn2VTzs
3D computer vision techniques v.4b2
http://www.3dcgi.com/cooltech/displays/displays.htm
76
3D display
Using a lattice with thin slits, viewer's eyes see different
pixels on the screen to create 3d perception

http://www.televisions.com/tv-articles/TV-in-3D/Displaying-3D-Without-Glasses.php
3D computer vision techniques v.4b2
77
The future




Content search in 3D video data bases
Shot boundary detection
Video data mining
Video classification
3D computer vision techniques v.4b2
78
Appendix
3D computer vision techniques v.4b2
79
Essential matrix E (a 3x3 matrix) P.110[2] X1 is 3-D X in left camera
(reference) system X2 is 3-D X in right camera system
Exercise1:
Draw vectors TX2 or TX1 in the diagram
 2  R1  T , multiply both sides by T 
T    T  R      (i )


2
1
Since [TA ]x TB  TA  TB and
T   skew symmetric
matrix and T  T  T  T  0
T and  2 are on the same plane,  2 is perpendicu lar to (T   2 )
so X 2 (T   2 )  0, same as X 2 (T   2 )  0
T
T
from (i),  2 (T  R1 )  0
T
hence  2 * E * 1  0
T
where E  T  R
  t1  
 0


[T ]x   t 2     t3
 t3  
 t 2
x
[TA ]x TB  TA  TB
3D computer vision techniques v.4b2
 t3
0
t1
t2 
 t1 
0 
80
Essential Matrix E
X 2T * E * X1  0
 X 2 Y2
Z 2  * E *  X 1 Y1
Z1   0
T
f 2 f1
no _ harm _ to _ prefix _ some _ cons tan t _ terms ,
Z 2 Z1
f2
 X 2 Y2
Z2
 x2
y2
since  X

Z2 
f1
* E *  X 1 Y1
Z1
f 2  * E *  x1
Y
y1
f1   0
T
Z   X, and x  f
T
Z1   0
T
X
Y
,y  f
Z
Z
Right_image_pointT*E*left_image_point=0
3D computer vision techniques v.4b2
81