Raskar, Camera Culture, MIT Media Lab
Computational Light Transport:
CORNAR: Looking Around Corners
using
Camera Culture
Trillion FPS Imaging
Ramesh Raskar
Ramesh Raskar
MIT Media Lab
http://raskar.info/cornar
What is
around the corner ?
Can you look
around the corner ?
Can you look
around the corner ?
Multi-path
Analysis
2nd Bounce
1st Bounce
3rd Bounce
Photon Hacking
Co-designing Optical and Digital Processing
Optics
Computational
Light Transport
Displays
Sensors
Illumination
Computational
Photography
Signal Processing
Computer Vision
Machine Learning
Bit Hacking
Multi-path
Analysis
2nd Bounce
1st Bounce
3rd Bounce
CORNAR: Femto-Photography
FemtoFlash
Trillion FPS camera
With M Bawendi,
MIT Chemistry
Serious Sync
Computational Optics
•2012: 3D around a corner (NatureComm, Velten, et. al.)
•2011: Material Sensing (Siggraph Asia, Naik, Zhao, Velten, Raskar, Bala)
•2011: DARPA Young Faculty Award
•2011: Motion Sensing (CVPR, Pandharkar, Velten, Bardagjy, Bawendi, Raskar)
•2009: Hidden barcode (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009)
•2008: Indirect depth (Hirsch, Raskar)
•2008: Transient Light Transport (Raskar, Davis, March 2008)
Inverting Light Transport
Multiple Scattering
[Seitz , Kutulakos, Matsushita 2005]
[Atcheson et al 2008]
[Kutulakos, Steger 2005]
Dual Photography
[Sen et al 2005]
Direct/Global
[Nayar, Raskar et al 2006]
LIDAR
Collision avoidance, robot navigation, …
…, bronchoscopies, …
z
x
S
L
s
Occluder
Streakcamera
C
B
Laser
beam
Echoes of Light
z
x
S
L
s
Occluder
Streakcamera
C
B
Laser
beam
Echoes of Light
z
x
S
L
R
s
Occluder
Streakcamera
Streak Photo
C
B
Laser
beam
3rd bounce
Echoes of Light
Multi-Dimensional
Light Transport
5-D Transport
Why Pico-second Resolution?
ToF Diff = 0.15 mm
s2
1cm
s1
Occluder
3rd bounce
Streakcamera
p1
C
1st bounce
p2
Curse of Pythagoras
z
x
S
L
R
s
Occluder
Streakcamera
Streak Photo
C
B
Laser
beam
3rd bounce
Echoes of Light
Trillion FPS
ToF Streak Tube = Inverse of CRT
Very accurate sync
1D camera: Single scan line stretched vertically in time
~2 ps resolution, 480 lines ~= 1 ns
But for small samples in biochemistry
Time-Image
R
Time Profile
for a single pixel
Time, ~2ns each row
Time Image of a single point
Space, 640 pixels
Third Bounce (First bounce not shown)
3D shape result from
synthetic data
Forward
Reconstruction
Invertibility
Analysis
Wavefront
Propagation
Non-linear
Inversion
Scene Priors
Resolution and
dimensions
Steady State 4D
[Kajiya, 1986] [Seitz.., 2005]
Impulse Response, 5D
[Raskar and Davis, 2007]
Time Resolved Multi-path Imaging
Scene with
hidden elements
Ultra fast illumination and camera
Raw
Time
profiles
5D
Capture
Photo, geometry,
reflectance
beyond
line of sight
Novel light transport
models and inference
algorithms
Signal
Proc.
t
3D Time images
Third Bounce (First bounce not shown)
Third Bounce (First bounce not shown)
Third Bounce (First bounce not shown)
Third Bounce (First bounce not shown)
Third Bounce (First bounce not shown)
Photos from Streak Camera
Capture Setup
Hidden Scene
Photos from Streak Camera
Capture Setup
Hidden Scene
Reconstruction
Overlay
Hidden 3D Shape
Space-time Photos
Motion beyond line of sight
Pandharkar, Velten, Bardagjy, Lawson, Bawendi, Raskar, CVPR 2011
BRDF (reflectance) from
Single Viewpoint and
Time Images
Naik, Zhao, Velten, Raskar, Bala, (SIGGRAPH Asia 2011)
…, cardioscopies, …
Participating Media
Space-time
Photos
Trillion
Frames Per Second
Imaging
http://raskar.info/trillionfps
Each frame = ~2ps = 0.6 mm of Light Travel
Camera Time
raskar.info/trillionfps
Camera Time
World Time
raskar.info/trillionfps
Lorentz transformation: distances, velocities, ordering
FemtoFlash
Trillion FPS
camera
Serious Sync
Computational Optics
Pixel 
Ray 
Wave

Photons
(ampli+phase)
Steady State
Transient
Impulse/Step
•2012: 3D around corner (Nature Comm, Velten et al)
•2011: Material Sensing (Siggraph Asia, Naik, Zhao, Velten, Raskar, Bala)
•2011: DARPA Young Faculty Award
•2011: Motion Sensing (CVPR, Pandharkar, Velten, Bardagjy, Bawendi, Raskar)
•2009: Hidden barcode (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009)
•2008: Indirect depth (Hirsch, Raskar)
•2008: Transient Light Transport (Raskar, Davis, March 2008)
• Collaborations Welcome
• Dataset Available
• Propose Configurations
raskar(at)media.mit.edu
http://www.media.mit.edu/~raskar/cornar/
Femto-Photography
http://raskar.info
Looking Around the Corner
BRDF Detection
Trillion FPS Movies
Space-time Transforms