Development of immersive realtime applications
Miroslav Andel
miroslav.andel@liu.se
2
Lectures
• Development of immersive real-time
applications
– Online/direct rendering
– Hardware
– Programming
• Video production
– Offline rendering
– Hybrid solutions
2014-09-03
Miroslav Andel - Development of immersive real-time applications
3
Outline
• Hardware
– Understanding the technology
– Clusters and GPUs
– Warping, color matching & blending
• Immersion in a virtual environment (VE)
• Software design
– Desktop application vs. immersive cluster
application
– Scalability & configuration of VEs
– Frameworks
– Scene graph
2014-09-03
Miroslav Andel - Development of immersive real-time applications
4
Hardware
2014-09-03
Miroslav Andel - Development of immersive real-time applications
5
Projectors
• 6x Digital Projection Titan
• 1920 x 1200 @ 120 Hz
• Frame doubling
2014-09-03
Miroslav Andel - Development of immersive real-time applications
6
Image Generators (IGs)
– Workstation high-end graphics cards
•
•
•
•
•
Frame sync
External house sync
Genlock (video signal sync)
Swaplock (swap buffer sync)
3D sync (for 3D glasses)
– Supports active stereoscopy using OpenGL
– Nvidia Quadro or AMD/ATI FirePro
– No gaming cards
2014-09-03
Miroslav Andel - Development of immersive real-time applications
7
Gaming vs. Workstation GPU
•
•
GPU is pretty much the same but different driver software
and firmware
Gaming cards are optimized for speed and DirectX
– Hi frame-rate in games, highly optimized
•
Workstation cards are optimized for precision, OpenGL &
OpenCL
– Designed for 3D modeling, CAD, video editing and computation
– Higher floating point and sub-pixel precision
– Better quality and larger memory (12 GB in Quadro K6000, 16 GB
AMD FirePro 9100)
• Ray tracing on the GPU
•
Our problem: We use gaming techniques but need workstation
features like synchronization and stereoscopy (less
performance)
2014-09-03
Miroslav Andel - Development of immersive real-time applications
8
Cluster of IGs
• Often consists of workstations
with the same specification
• Master – Slave
• In our case the master has an
additional surround sound card
(live encoding to Dolby
Digital or DTS)
Miroslav Andel - Development of immersive real-time applications
9
Clusters
(at the visualization center)
• Dome
– Alpha (primary real-time cluster used for
shows)
– Beta (secondary development cluster)
– Delta (movie and presentations playback
cluster)
– Gamma (single node eyefinity machine)
• VR-Arena
– Theta
– Videowall (just a single node with 6
outputs)
used for movie and presentations playback
Miroslav Andel - Development of immersive real-time applications
11
Color/gammut matching
• What is white?
2014-09-03
Miroslav Andel - Development of immersive real-time applications
15
Blending
• Digital blending – good bright levels
– Not a linear gradient (gamma)
– Done after color and brightness matching
2014-09-03
Miroslav Andel - Development of immersive real-time applications
16
Blending
• Optical blending for better dark levels
– Projected black is not black, still projected
light
2014-09-03
Miroslav Andel - Development of immersive real-time applications
17
Blending
• Optical blending for better dark levels
– Projected black is not black, still projected
light
2014-09-03
Miroslav Andel - Development of immersive real-time applications
18
Blending
• Blending in the dome
– Blue = Projection borders
– Red = Blend zones
2014-09-03
Miroslav Andel - Development of immersive real-time applications
19
Warping/geometric correction
• Correct the rendered frame to match the
geometry of the screen, lens and
projector’s position
• Correct overlaps between adjacent channels
• Spatial 2D transform
2014-09-03
Miroslav Andel - Development of immersive real-time applications
20
Warping and blending hardware
• Processing unit with input & outputs
cards (FPGA) with minimal latency
2014-09-03
Miroslav Andel - Development of immersive real-time applications
21
Warping and blending using OpenGL
• No latency but uses a bit of the GPU
2014-09-03
Miroslav Andel - Development of immersive real-time applications
22
Calibration process
• Manual warping and blending calibration
takes many hours
– Few reference points in the dome
– Moving mesh points by hand causes linearity
problems
• Semi-automated calibration
• Automated camera-based systems
– Current systems don’t handle optical blends
2014-09-03
Miroslav Andel - Development of immersive real-time applications
23
Hardware (end)
2014-09-03
Miroslav Andel - Development of immersive real-time applications
24
Immersion
• To give the audience the impression that
they have stepped into a synthetic world
2014-09-03
Miroslav Andel - Development of immersive real-time applications
25
Immersion
• To give the audience the impression that
they have stepped into a synthetic world
• Enhance immersion by:
–
–
–
–
–
–
2014-09-03
Large field of view
High resolution
Stereoscopy (can also reduce immersion)
Surround sound
Smooth navigation (based on a physical model)
Realistic rendering (shadows etc., to improve
depth cue)
Miroslav Andel - Development of immersive real-time applications
26
Immersion
• Stereoscopy in the dome?
– The paradox
– Field of view wearing 3D glasses
– Use stereoscopy wisely
2014-09-03
Miroslav Andel - Development of immersive real-time applications
28
Software design
2014-09-03
Miroslav Andel - Development of immersive real-time applications
29
The difference
(from normal desktop application development)
• Several projectors/screens
– A lot of pixels
– Alignment, blending & warping
– Color & Brightness matching
• Several computers (cluster)
• Synchronization
–
–
–
–
Frame
Buffers swap
Data
Video signal (genlock)
• Multi-threading
2014-09-03
Miroslav Andel - Development of immersive real-time applications
30
Stereoscopy
• How does it work?
2014-09-03
Miroslav Andel - Development of immersive real-time applications
31
Stereoscopy
2014-09-03
Miroslav Andel - Development of immersive real-time applications
32
Stereoscopy
2014-09-03
Miroslav Andel - Development of immersive real-time applications
33
Stereoscopy
2014-09-03
Miroslav Andel - Development of immersive real-time applications
34
Stereoscopy
• Co-location
– Camera position = eye position
– Projection plane = real physical screen
2014-09-03
Miroslav Andel - Development of immersive real-time applications
35
Stereoscopy
• Don’t change eye separation, change the
scale of your scene
• ~15 meters limit
2014-09-03
Miroslav Andel - Development of immersive real-time applications
36
2014-09-03
Miroslav Andel - Development of immersive real-time applications
37
2014-09-03
Miroslav Andel - Development of immersive real-time applications
38
2014-09-03
Miroslav Andel - Development of immersive real-time applications
39
Stereoscopy
• Object in focus in negative parallax
2014-09-03
Miroslav Andel - Development of immersive real-time applications
40
Stereoscopy
• Object in focus in positive parallax
2014-09-03
Miroslav Andel - Development of immersive real-time applications
41
Stereoscopy
• Problems with a fixed head setup (no head
tracking)
2014-09-03
Miroslav Andel - Development of immersive real-time applications
42
Scalability
• One application can run in several virtual
environments without the need of changing the
code
–
–
–
–
–
In
In
In
At
At
the dome
the VR-Arena
the VR-Lab
home using your computer
the bus using a laptop
• If the platform is the same then there is no
need for recompiling the application (just
copy & paste)
2014-09-03
Miroslav Andel - Development of immersive real-time applications
43
Scalability
The application uses a configuration file
that describes the hardware in terms
like:
•
•
•
•
•
•
•
•
Number of IGs
Number of displays
Number of viewports
Physical location of displays/projection
planes
Physical location of the user
Eye separation of the user
Type if Stereoscopy
Etc…
2014-09-03
Miroslav Andel - Development of immersive real-time applications
44
Configuration
• “3d drawing”
2014-09-03
45
Configuration
Monoscopic projection
Immersive applications
Often uses non-symmetric
frustums/projections, while
normal OpenGL applications
don’t
(left FOV = right FOV,
top FOV = bottom FOV)
2014-09-03
46
Configuration
Stereoscopic projection
2014-09-03
47
Configuration
48
Configuration
All users except
the target user
will perceive
the rendered
object
incorrectly.
49
Configuration
All users except
the target user
will perceive
the rendered
object
incorrectly.
One solution
would be using
HMDs or a
multi-view
screen.
50
Navigation
• In a desktop application a user navigates
by moving the camera
2014-09-03
Miroslav Andel - Development of immersive real-time applications
51
Navigation
• In a desktop application a user navigates
by moving the camera
• In a scalable application the number of
cameras/frustums is not known
2014-09-03
Miroslav Andel - Development of immersive real-time applications
52
Navigation
• In a desktop application a user navigates
by moving the camera
• In a scalable application the number of
cameras/frustums is not known
• Instead of moving the user and all
“cameras” in the same direction the whole
scene is moved in the opposite direction
2014-09-03
Miroslav Andel - Development of immersive real-time applications
53
Frameworks
(the good stuff)
• Will help you with
–
–
–
–
2014-09-03
Cluster setup & synchronization
Frustums & viewports
Input
Tracking
Miroslav Andel - Development of immersive real-time applications
54
Frameworks
• VR Juggler
• CAVElib (commercial)
• Equalizer
2014-09-03
Miroslav Andel - Development of immersive real-time applications
55
Frameworks
(the bad stuff)
•
•
•
•
Massive and have a lot of dependencies
Big initial effort to get started
Often in legacy OpenGL
Takes usually several days to get started
–
–
–
–
2014-09-03
Getting all dependencies
Compile all dependencies
Compile the whole framework
Read the documentation
Miroslav Andel - Development of immersive real-time applications
56
SGCT
Simple Graphics Cluster Toolkit
2014-09-03
Miroslav Andel - Development of immersive real-time applications
57
SGCT
Simple Graphics Cluster Toolkit
• Static library based on
–
–
–
–
–
–
–
–
–
–
2014-09-03
GLFW
GLEW
GLM
Freetype
TinyThread++
TinyXML
PNG
Z
VRPN
TurboJPEG
Miroslav Andel - Development of immersive real-time applications
58
SGCT
Simple Graphics Cluster Toolkit
• Developed here and is free for everybody to
use
• Designed to get started immediately
(rapid development)
• Cross-platform C++
• Modern and Legacy OpenGL support (OGL 2.2 –
4+)
• Single static library (no dll-hell)
• Using callbacks rather than
inheritance/polymorphism
Miroslav Andel - Development of immersive real-time applications
– Easy porting
application from GLUT, GLFW or
2014-09-03
59
SGCT
Simple Graphics Cluster Toolkit
• Wiki at: c-student.itn.liu.se
–
–
–
–
–
Documentation
Tutorials
Configuration files
Downloads (installers)
Support
• Forum (on it’s way)
2014-09-03
Miroslav Andel - Development of immersive real-time applications
60
SGCT – How it works
• Has an engine which handles
–
–
–
–
–
–
Initiation of OpenGL
Rendering
Synchronization
Network communication
Configuration handling
Input
• The user can bind callbacks (function
pointers) to the engine to customize the
functionality
2014-09-03
Miroslav Andel - Development of immersive real-time applications
61
2014-09-03
62
SGCT – How it works
• Set only the callbacks you need
• A minimal cluster application contains:
– Draw callback: Render your stuff
– Pre Sync callback: generate the data to be
shared
– Encode callback: Send the data to the slave(s)
– Decode callback: Receive the data from the
master
• Other callbacks to handle input devices
and external GUIs
2014-09-03
Miroslav Andel - Development of immersive real-time applications
63
SGCT - Features
• Frame-, swapbuffer- and application datasynchronization
• 3D/Stereoscopic rendering
–
–
–
–
–
–
2014-09-03
Active/Quadbuffer stereo
Side-by-side stereo
Top-bottom stereo
DLP/Checkerboard stereo
Line interleaved stereo
Anaglyph stereo
Miroslav Andel - Development of immersive real-time applications
64
SGCT - Features
•
•
•
•
•
•
•
•
•
•
•
•
Fisheye rendering
PNG texture management
GLSL shader management
Freetype font management
Error messaging across the cluster
External TCP control interface
Tracking support
Joystick, gamepad, Kinect, steering wheel etc.
input
Antialiasing
3D screenshots in maximum texture resolution
Overlays
Statistics and performance graphs
2014-09-03
Miroslav Andel - Development of immersive real-time applications
65
SGCT - Configuration
• XML configuration
• All nodes are using the same
configuration and can automatically
determine if they are a slave or a master
2014-09-03
Miroslav Andel - Development of immersive real-time applications
<?xml version="1.0" ?>
<Cluster masterAddress="127.0.0.1">
<Node ip="127.0.0.1" port="20401">
<Window fullscreen="false">
<Size x="640" y="360" />
<Viewport>
<Pos x="0.0" y="0.0" />
<Size x="1.0" y="1.0" />
<Viewplane>
<!-- Lower left -->
<Pos x="-1.778" y="-1.0" z="0.0" />
<!-- Upper left -->
<Pos x="-1.778" y="1.0" z="0.0" />
<!-- Upper right -->
<Pos x="1.778" y="1.0" z="0.0" />
</Viewplane>
</Viewport>
</Window>
</Node>
<User eyeSeparation="0.065">
<Pos x="0.0" y="0.0" z="4.0" />
</User>
</Cluster>
66
67
SGCT arguments
• Normal use:
app.exe –config
• Running cluster
app.exe –config
• Running cluster
app.exe –config
slave
2014-09-03
“config.xml”
application locally as master:
“cluster_config.xml” –local 0
application locally as slave:
“cluster_config.xml” –local 0 --
Miroslav Andel - Development of immersive real-time applications
68
Running SGCT
•
•
•
•
Command prompt/terminal
Create a Batch file (.bat)
Create a Shortcut
From IDE
2014-09-03
Miroslav Andel - Development of immersive real-time applications
69
Running cluster applications
• The application folder needs to be
accessible from all IGs/nodes
1. Copy the whole folder
•
Use Microsoft’s robocopy tool for mirroring folders
across a cluster
2. Use a network share
• DemoManager tool for launching (and
killing) processes across a cluster
• Or manually start the application on all
nodes
2014-09-03
Miroslav Andel - Development of immersive real-time applications
70
SGCT – Example
• Basic spinning triangle synchronized
across a cluster using less then 80 lines
of code
(including comments and spaces)
2014-09-03
Miroslav Andel - Development of immersive real-time applications
71
Data synchronization
• What to share?
– Time or dt
– Run simulations only on master or on
whole cluster?
– Random numbers?
– Minimize traffic
– Compression
– Bit-masking
2014-09-03
Miroslav Andel - Development of immersive real-time applications
72
Math
• OpenGL Mathematics (GLM) is included in
SGCT
• Computer graphics is linear algebra!
• GLM makes life easy
– Vectors
– Matrices
– Quaternions
• Similar syntax to glsl
2014-09-03
Miroslav Andel - Development of immersive real-time applications
73
Testing anywhere
• Local cluster on a single computer
• View
2014-09-03
Miroslav Andel - Development of immersive real-time applications
74
Testing anywhere
2014-09-03
Miroslav Andel - Development of immersive real-time applications
75
SGCT + OSG
Open Scene Graph (OSG)
• www.openscenegraph.org
• Feature rich
• SGCT manages
–
–
Cluster sharing & synchronization
Window, frustum and viewport creation
• OSG manages
– Content
• OSGMaxExporter (3Ds Max plugin)
2014-09-03
Miroslav Andel - Development of immersive real-time applications
76
Scene graph
• Tree structure of nodes
• Intelligent structure
– Visibility & occlusion culling
2014-09-03
Miroslav Andel - Development of immersive real-time applications
77
OSG example
2014-09-03
Miroslav Andel - Development of immersive real-time applications
78
Audio
• Positional 3D audio
– OpenAL
– Fmod Ex
2014-09-03
Miroslav Andel - Development of immersive real-time applications
79
Next time
• Video pipeline
• Offline rendering for the dome
2014-09-03
Miroslav Andel - Development of immersive real-time applications