Mitsubishi Electric Research Laboratories
Seamless Multi-Projector Display
on
Curved Screens
Jeroen van Bar, Thomas Willwacher,
Srinivas Rao, Ramesh Raskar
Mitsubishi Electric Research Labs
Cambridge, MA USA
Raskar May 03
Mitsubishi Electric Research Laboratories
Curved Screen Displays
Multiple overlapping projectors on curved screens
Goal :
• Replace single-proj
• Higher resoltn/brightness
• Sub-pixel auto-alignment
• Parametric solution
• Low cost infrastructure
Markets
Planetarium
Curved screens
Raskar May 03
Mitsubishi Electric Research Laboratories
Dome Projection Techniques
Edge-Blended
(Tiled/Mosaic)
Display
• Sub-Frames w/Spherical Mapping & Edge-Blends
Raskar May 03
Mitsubishi Electric Research Laboratories
1
Dome
Screen
3
2
4
Raskar May 03
Mitsubishi Electric Research Laboratories
Outline
Seamless Curved Display
•
•
•
•
Multi-projector low cost method
Simplified Quadric Image Transfer
Calibration with camera-pair
Parametric Rendering solution
Raskar May 03
Mitsubishi Electric Research Laboratories
Related Work
• Conventional Displays
– Manual alignment, expensive infrastructure
– [Jupiter,Trimensions, CAVE, Planetaria, Flight Simulators]
• Planar Screens
– Camera in loop, auto calibration, low cost
– Exploit homography parameters
– [Raskar98,Surati99,Chen00,Brown02 ……]
• Curved Screens
– Non-parametric solutions
– [Jarvis97,Raskar98,Yang01 …]
– Parametric
– ?, Siggraph 2003
Raskar May 03
Mitsubishi Electric Research Laboratories
Parametric Approach
• Advantages
– Lower camera resolution
– Tolerance for pixel localization errors
– Faster calibration
– Efficient well-defined warping
• Avoid look up tables
Raskar May 03
Mitsubishi Electric Research Laboratories
Parametric Image Transfer
X
X
i
i
j
j
Planar Homography
j  A3 x 3 i
Quadric Transfer
j  A3 x 3 i 

T

i Ei e
Raskar May 03
Mitsubishi Electric Research Laboratories
Planar projective transfer
What is homography ?
– Two images of 3D points on a plane are
related by a 3x3 matrix
M
i
j = A3 x 3 i
j
Proj 1
Proj 2
Raskar May 03
Mitsubishi Electric Research Laboratories
What is homography ?
Two images of 3D points on a plane
~ A
Related by a 3x3 matrix j =
3x3
A3 x
i
jx
a1
a2
a3
ix
jy
= b1
b2
b3
iy
1
c1
c2
c3
1
3
k
j
i
Proj 1
Proj 2
jx
= (a • i) / (c • i)
jy
= (b • i) / (c • i)
Raskar May 03
Mitsubishi Electric Research Laboratories
Planar Displays
Current Multi-Cube System
MERL Projector Planar Mosaic
1
2
1
3
3
2
4
4
Feature Comparison
Many hours
$20,000/projector cube
15 kg/projector cube + heavy
support
Many hours
Time to install
Cost
Weight
Alignment
Few minutes
$5,000 per projector
3 kg/projector, no fixed support
structure
10 seconds !
Needs regular care
Maintenance
Very low
Small gaps
Image quality
Seamless !, Subpixel alignment
Only high end: command and
control, advertising
Market
Middle segment: Shops,
schools, offices,
trade
Raskar
Mayshows
03
Mitsubishi Electric Research Laboratories
Curved Screens
View for a Sweet-spot
Projector
Sweet spot
(Static user)
Raskar May 03
Mitsubishi Electric Research Laboratories
Calibration for a
Sweet-spot
Projector
Camera at
Sweet spot
Raskar May 03
Mitsubishi Electric Research Laboratories
Discretized non-parametric approach
Projector Image
p1
p6
Projector
Camera at
Sweet spot
c1
c6
Desired
Image
Camera
= Image
Raskar May 03
Mitsubishi Electric Research Laboratories
• Off-Axis Spherical Distortion
Ideal Viewpoint
Offset Viewpoint
Raskar May 03
Mitsubishi Electric Research Laboratories
Fish-eye Projection
Planetaria and
Digital Dome Theaters
• Immersive Production Software
– Spitz - PolyDome™
– SkySkan - DigiDome™
Raskar May 03
Mitsubishi Electric Research Laboratories
Outline
Seamless Curved Display
•
•
•
•
Multi-projector low cost method
Simplified Quadric Image Transfer
Calibration with camera-pair
Parametric Rendering solution
Raskar May 03
Mitsubishi Electric Research Laboratories
Curved projective transfer
Quadric classification
Projectively equivalent to sphere:
sphere
Ruled quadrics:
ellipsoid
hyperboloid paraboloid
of two sheets
hyperboloids
of one sheet
Degenerate ruled quadrics:
cone
two planes
Raskar May 03
Mitsubishi Electric Research Laboratories
Quadrics
X QX  0
T
For 3D points X on Quadric
X
Q : 4x4 symmetric matrix, Q
Q
Nine d.o.f























In general 9 points in 3D define quadric
Raskar May 03
Mitsubishi Electric Research Laboratories
Quadric Image Transfer
• Quadratic image transfer function
[Shashua97]
X
Quadric written as
x
x’
T
x '  Bx  ( q x 
T
Q 33
Q   T
q
2
q

1
T
( q x )  x Q 33 x ) e
21 params, 4 more than necessary !
Raskar May 03
Mitsubishi Electric Research Laboratories
Simplified Quadric Image Transfer
Our Solution
T
T
x '  Bx  ( q x 
X
2
T
( q x )  x Q 33 x ) e
Based on observation ..
x
x’
A  B  eq
T

x '  Ax  

E  qq
T
 Q 33

T
x Ex  e

17 param warp
Raskar May 03
Mitsubishi Electric Research Laboratories
Simplified Quadric Image Transfer
X

x '  Ax  


x Ex  e

T
17 param warp
x
x’
Planar homography:
Quadric transfer:
4 corresponding pixels
9 corresponding pixels
Raskar May 03
Mitsubishi Electric Research Laboratories
Outline
Seamless Curved Display
•
•
•
•
Multi-projector low cost method
Simplified Quadric Image Transfer
Calibration with camera-pair
Parametric Rendering solution
Raskar May 03
Mitsubishi Electric Research Laboratories
Calibration of Quadric Screens
Raskar May 03
Mitsubishi Electric Research Laboratories
Approach
Calibration
• At each projector i,
– Project structured pattern
– View with stereo camera
– Finding camera to projector quadric transfer,  i
Run-time
• At each projector i,
– Pre-warp input image using  i
Raskar May 03
Mitsubishi Electric Research Laboratories
Calibration
Finding relationship between camera and projector
Low-res Camera 640x480 images
But each Projector 1024x768
Raskar May 03
Mitsubishi Electric Research Laboratories
Non-linear Refinement
Linear Estimation
Error ~10 pixels
NonLinear Refinement
Error ~ 1.0 pixels
Raskar May 03
Mitsubishi Electric Research Laboratories
Before Blending
Raskar May 03
Mitsubishi Electric Research Laboratories
After Blending
Raskar May 03
Mitsubishi Electric Research Laboratories
Intensity Correction in Overlap
Projector Framebuffers
Raskar May 03
Mitsubishi Electric Research Laboratories
Projector Framebuffers
Raskar May 03
Mitsubishi Electric Research Laboratories
Projector Framebuffer Intensity Weights
Raskar May 03
Mitsubishi Electric Research Laboratories
Outline
Seamless Curved Display
•
•
•
•
Multi-projector low cost method
Simplified Quadric Image Transfer
Calibration with camera-pair
Parametric Rendering solution
Raskar May 03
Mitsubishi Electric Research Laboratories
Rendering a 3D Scene
Steps at each projector
(Pre-distort vertex 3D location)
• For each triangle T with vertices {Mj}
– For each vertex M
• Find pixel m via VirtualViewProjection( M )
• Find warped pixel m’ via quadricTransfer of m
• Replace M with m’
Raskar May 03
Mitsubishi Electric Research Laboratories
Vertex Shader for
Quadric Transfer in Cg

x '  Ax  


T
x Ex  e

vertout
main(
appin
IN,
uniform
float4x4
modelViewProj,
uniform float4 constColor, uniform float3x3 A, uniform float3x3 E,
uniform float3 e) {
vertout OUT;
float4 m1 = float4(IN.position.x, IN.position.y, IN.position.z, 1.0f );
float4 m, mi ; float3 m2,mp;
float scale;
m = mul( modelViewProj, m1);
Parametric
m2.x = m.x/m.w;
m2.y = m.y/m.w; m2.z = 1;
scale = mul(m2, mul(E,m2));
Warp
mp = mul(A,m2) + sqrt(scale)*e;
mi.x = m.w * (mp.x)/(mp.z);
mi.y = m.w * (mp.y)/(mp.z);
mi.zw = m.zw;
OUT.position = mi;
OUT.color0 = IN.color0; // Use the original per-vertex color specified
return OUT;
}
Raskar May 03
Mitsubishi Electric Research Laboratories
Rendering 2D + 3D scene
Concave Dome
Convex Dome
Raskar May 03
Mitsubishi Electric Research Laboratories
Details I Skipped ..
• Estimating camera and projector params
– Internal and External params
– Issue with near-planar 3D points
• Finding pixels weights for blending
• Non-linear optimization
• Rendering
– Warping and Depth buffer issues
Raskar May 03
Mitsubishi Electric Research Laboratories
Seamless Curved Display
• Multi-projector low cost method
• Simplified Quadric Image Transfer
• Complete Parametric calib+rendering solution
More info :
www.raskar.com/Projector/
Raskar May 03
Mitsubishi Electric Research Laboratories
Projector Mailing List
majordomo@cs.unc.edu
subscribe projector
Projector bibliography
www.raskar.com/Projector/
Raskar May 03
Mitsubishi Electric Research Laboratories
Advantages
• Parametric warp
– Lower camera resolution
– Tolerance for pixel localization errors
– Faster calibration
– Efficient well-defined warping
Raskar May 03