Stereoscopic Analyzer
On-Set Assistance System
for 3D Capturing
Frederik Zilly
The Product
Frederik Zilly
With Touchscreen to see
at NAB Show 09
The Product
Frederik Zilly
With Touchscreen to see
at NAB Show 09
Outline
• Conventional Workflow in Stereo Production
• The Vision: Automization of Stereo Workflow
• Concept of Stereoscopic Analyzer (STAN)
• Implementation Details
• Applications Beyond Standard Stereo
Frederik Zilly
• Conclusions and Outlook
Current Workflow of Stereo Production
Viewfinder
Manual Adaptation of baseline to scene geometry:
Measure Distance of Near/Far Clipping Plane
Camera 1
Camera 2
Frederik Zilly
Mirror Rig
Bad Stereo
Vertical disparities
Unmatched Colours
Lens Distortions
Eye strain
Visual fatigue
Headache
Manual
Stereo
PostProduction
Good Stereo
Rectified Stereo-Pairs
Matched Colors
The Vision: Automization of Stereo Workflow
• For Post-Production: supply Meta-Data for offline processing
• For 3DTV: Apply corrections in Real-Time
Camera 1
Camera 2
Bad Stereo
Mirror Rig
Realtime
Optimization
Of Stereo
Parameters
Good Stereo
Frederik Zilly
Feedback
• Short Feedback Loop for manual and/or automatic
reconfiguration
Stereo Input
Analyze
Control
Meta Data
Update
Monitor
Frederik Zilly
3D Viewfinder Zconv
Offline Correction
3DTV Post-Production
Playout
Proxy
Live Stereo for 3DTV
Stereo Input
RT Correction
Analyze
Control
Meta Data
Update
Monitor
Frederik Zilly
3D Viewfinder Zconv
Analyze Stereo Pair
Control
•
•
•
•
•
Baseline
Color: rough
Focus
Iris
Zoom
Frederik Zilly
Monitor
•
•
•
•
Stereo Quality
Framing Alert
Vectorscope
Disparity Histogram
Correction
•
•
•
•
Rectification
Color: fine
Lens Distortion
Convergence Plane
– Shift / Crop / Scale
Analyze: Fast GPU Feature Detector
• Point correspondences:
• Detect Feature Points
Frederik Zilly
• Estimate Fundamental Matrix F
– Apply constraints induced by epipolar geometry to
feature points
Monitor: 3D Viewfinder
Scene Points
Disparity Histogram
Frederik Zilly
1/30 Screen Width
Monitor: Disparity Analysis
Convergence Plane
Frederik Zilly
Near clipping plane warning
Possible framing warning
Zconv
Correction of Geometrical Distortions
• Correct Lens Distortions & Keystones in Real-Time
• For RT-Preview or Live Transmission
• Handle Meta-Data in Machine & Human Readable Format
Frederik Zilly
• Supply Proxy Driver for Post-Production and Playout
3D Viewfinder in Detail
Left shifted Image
Right shifted Image
Disparity Range
Zconv
Actual Baseline
Frederik Zilly
Calc. Baseline
GUI Design by KUK Film Production
3D Viewfinder continued
Frederik Zilly
Disparity Histogram
Different View Modes
Color Calibration
Waveform
Vectorscope
Features under Development
• Preview of Depth Information and Scene Depth Structure
for Post-Production Purposes
Frederik Zilly
• Real-Time Depth Calculation for Live 3DTV Using VideoPlus-Depth Formats
•
Rectified Stereo enables efficient scanline search
•
Color Corrected Stereo helps SAD-based algorithms, improves coding efficiency
•
Known and/or controlled disparity range makes Depth Estimation faster and more robust
• Enable Zoom
•
Needed for Live 3DTV, however introducing new challenges
Camera 1
Camera 2
Mirror Rig
Sat Cam 2
Realtime
Depth-Estimation
Sat Cam 1
Live Registration &
Color Correction
Beyond Stereo: Scenario for Multi Video + Depth
Frederik Zilly
Depth estimation more efficient for registered and color corrected data
Live
Multi-Video+Depth
Broadcast
Conclusions and Outlook
• STAN: Camera Assistance System for 3D Production
• Proof of Concept and Algorithmic Prototype
• NAB‘09: First Presentation to Public
• First Hardware Prototype Planned for NAB‘10
• Extensions under Development
• RT Capture of Video-Plus-Depth
• Application to Multi-View Stereo
Frederik Zilly
• Live Broadcast (incl. Zooming)