3D Stereo Reconstruction using
iPhone Devices
Final Presentation
24/12/2012
Performed By:
• Ron Slossberg
• Omer Shaked
1
Supervised By:
• Aaron Wetzler
Project’s Goal
• Building a self-contained mobile 3D
reconstruction system using two iPhone
devices
2
Background – Pinhole Camera Model
• The basic camera model
• Transformation from 3D to 2D coordinates:
x  K  R t  X
• Distortion also taken into account
3
Background – Stereo Vision
• Combine images from two cameras to
generate depth image
• Relative cameras’ positions in physical space
(R, T) and image planes space (F) retrieved by
stereo calibration process
4
Background – Stereo Correspondence
• OpenCV offers a number of algorithms for
stereo correspondence
• We chose to use two algorithms that offer a
good compromise between efficiency and
quality:
– Block Matching
– Semi Global Block Matching
5
Background – Stereo Correspondence
(Cont.)
• Block Matching:
– Looks at blocks of pixels along the epipolar lines
and finds matches according to cross correlation.
– Example of typical result:
6
Background – Stereo Correspondence
(Cont.)
• Semi Global Block Matching:
– Adds on to the normal block matching algorithm
by introducing global consistency constraints
– The constraints are introduced by aggregating
matching costs along several independent, onedimensional paths across the image
– Example of typical result:
7
Background – Reconstructed Scene
• The matching algorithm produces a disparity
map which is a gray scale image where every
color corresponds to a certain disparity and
thus a certain depth
• Using a reprojection matrix on the disparity
map we obtain a point in space corresponding
to each pixel
• We can render these points as a 3d mesh
using the original picture colors for each
vertex.
8
Programming Environment
• iPhone app programming
– Objective-C programming
– Model-View-Controller design pattern
• Main implemented features:
– Displaying and controlling the views
– Inter-device communication and time
synchronization
– Persistent storage of data
9
Programming Environment (cont.)
• OpenCV libraries
– C++ open-source code
– Implement all the required algorithms for
performing the calibration and reconstruction
processes
– Handle the interface with the iPhone’s camera
• Main implemented features:
– Integrating openCV functions into our iPhone app
– Correct data flow into and out of every openCV
function
10
Programming Environment (cont.)
• OpenGL ES libraries
– A lightweight version of the open-source OpenGL
libraries, which includes an iOS API
– Implement the framework for rendering 2D and
3D computer graphics
• Main implemented features:
– Displaying the reconstructed images as an
interactive 3D surface
11
Software High-Level Design
Main Menu
Connect Devices
• Generating a Bluetooth session between the
devices
12
Software High-Level Design
Main Menu
Settings
• Setting the right parameters for stereo
calibration and reconstruction
13
Software High-Level Design
Main Menu
Calibration
• Performing stereo calibration for the devices
using a chessboard pattern
14
Software High-Level Design
Main Menu
Reconstruction
• Performing 3D stereo reconstruction of the
images captured by the devices
15
Software High-Level Design
Main Menu
Photo Album
3D Image Display
• Interactive 3D color display of the images
• Disparity map images shown within table
16
17
Calibration Process Flow
Initial State
Calibration
Bluetooth session created
Message delay calculated
Validating message delay
Initializing camera
Initial State
Calibration
Capture
Wait Message
Delay
Send Capture
Indication
Send Capture
Order
Capture Image
Capture Image
Exchange image corners data
18
Extract Corners
Extract Corners
Calibration Process Flow
(Cont.)
Performed
separately at each
device
Calibrate
Process was
separated to
increase accuracy
Compute Intrinsic
Parameters
Compute Extrinsic
Stereo Parameters
Save Parameters
19
Reconstruction Process Flow
Initial State
Reconstruction
20
Bluetooth Session Created
Message delay calculated
Calibration Performed
Validating message delay
Initializing camera
Initial State
Reconstruction
Load parameters
Load parameters
Compute undistorted
rectified bitmap
Compute undistorted
rectified bitmap
Choose Reconstruction
Algorithm
Choose Reconstruction
Algorithm
Save
Disparity
Image
Reconstruction Process Flow
(Cont.)
Capture
Wait Message
Delay
Send Capture
Indication
Send Capture
Order
Capture Image
Capture Image
Send and Receive
Image
21
Send Images
Send and Receive
Image
Remap Images to get
Rectified Images
Remap Images to get
Rectified Images
Compute Disparity by
stereo correspondence
Compute Disparity by
stereo correspondence
Save
Disparity
Image
Implementation Issues
• Simultaneous Photo Capture
– Problem: need devices to capture images at the
same time to gain good results
– Solution: implemented RTT calculation algorithm
that ignores anomalous results and performs
update phases during operation
– Other Solutions: Web Service, GPS
– Advantage: messages traverse only short distance,
no dependency on GPS signals
– Main Disadvantage: GPS achieves better accuracy
22
Implementation Issues
(Cont.)
• Inter-Device Communication
– Problem: need to pass messages and data
between the two devices
– Solution: Bluetooth communication
– Other Solutions: Wi-Fi
– Advantages: existing easy-to-use framework,
simple protocol with low overhead
– Main Disadvantage: smaller BW (affects the
duration of the reconstruction process)
23
Summary
• Very challenging and enjoyable project
• Introduction with both computer vision and
mobile app development
• Final outcome is a stable, user-friendly app
providing live results
• Helpful documentation for future usage
– Detailed webpage and demo movie
• Finally – many thanks to Aaron for his guidance
and support throughout this project !
24
References
• Computer Vision course, Spring 2010, University of Illinois
–
http://www.cs.illinois.edu/~dhoiem/courses/vision_spring10/lectures/
• Developing Apps for iOS, Paul Hegarty, Stanford fall 2010 course
(available on iTunes)
• Multiple View Geometry in Computer Vision course, University of
North Carolina
–
http://www.cs.unc.edu/~marc/mvg/slides.html
• Modeling the Pinhole camera, course lecture, University of Central
Florida
• Computer Vision tutorial, GIP lab
• Learning OpenCV, Gary Bradski & Adrian Kaehler
• Stereo Vision using the OpenCV library by Sebastian Droppelmann,
Moos Hueting, Sander Latour and Martijn van der Veen
25
The End
26