Electrical and Computer Engineering Dept.
Emerging Applications of VR
Emerging applications of VR
 In manufacturing (especially virtual prototyping, assembly
verification, ergonomics, and marketing);
 In robotics (programming, teleoperation, space robotics);
 In data visualization (volume visualization, oil and gas
exploration, volumetric displays);
 Other areas.
VR penetration in non-medical fields
% - companies using,
Experimenting or
Considering VR
(UK VR Forum survey, 2000)
Classes of VR applications
(UK VR Forum survey, 2000)
Main benefits of using VR
(UK VR Forum survey, 2000)
VR in Manufacturing
GHOST Free Form Application
 Free Form sculpting is a new type of HumanComputer interaction;
 It functions as a convenient GUI that sits on top of the
GHOST library;
 The GUI is based on static and dynamic bar menus
“dynabars”;
 It allows model export to CAM machines for rapid
prototyping, as well as to animation packages for
computer character animation.
 Requires two Quad Core processors, 4 to 16 GB RAM,
high-end hardware graphics acceleration, 1024x768 screen
resolution (or better).
Free Form Sculpting
GHOST SDK
PHANToM Drivers
FreeForm Initial screen
Menu bar
Tool bar
Work area
border
Status bar
Dynabar
The center block is “digital clay”
FreeForm wire-cut mode
Wire Cut Menu
Sketch planes
Drawing on the cut plane
Clay after Inside wire cut
Cut Inside
button
Clay after Outside wire cut
Cut Outside
button
Free Form wire cut
Carving a hole
Smoothing while carving
Carving a corner
Getting carried away….
FreeForm carving tools
Example – Making of Saint Fruition
Artist’s sketch
Rough initial clay model
Example – Making of Saint Fruition
Wire cut arrow for support (clay)
Finished body (clay)
Finished support (clay)
Example – Making of Saint Fruition
Digital clay statue
5 ft Aluminum statue
Example – Making of Saint Fruition
Statue model used for
animation (Maya)
Textured statue used for
animation (Maya)
Link to VC 9.1 on book
CD
Key frame animation of statue
Assembly verification
 Another stage in product development when the prototype
is made of several parts;
 University of Washington developed the “Virtual Assembly
Design Environment” (VADE) to verify CAD design
assemblies;
 Parts geometry and attributes are imported from CAD into
VADE then the assembly is analyzed and robots are
programmed
Design modification in VR
Collision detection through swept volumes
Assembly verification – Car body tolerances
 Parts making up car exterior have varying tolerances.
Tighter tolerances are more esthetically pleasing but also cost
more.
 What is good enough? Inspection is done in inspection
rooms using stripped lights.
 Same can be done on a virtual car ahead of real production
Real inspection room
Virtual Inspection room
Assembly verification
 Researchers in UK developed the Visualization of the Impact
of Tolerance Allocation (VITAL) and tested it on a prototype
Rover R75;
 They constructed several models with various tolerances; by
shining the virtual car body with stripped light looking at
discontinuities;
Unacceptable tolerances discovered
in the virtual inspection room
Ergonomic Analysis
 Jack is a an intelligent agent homanoid used in ergonomic
analysis;
 the Task Analysis Toolkit computes lower-back effort and
energy consumption – relates to worker fatigue
Link to VC 9.2 on book
CD
Ergonomic Analysis - continued
Once a prototype is done, it has to be tested for ease of use
(ergonomic analysis);
 One such product is VirtualANTHROPOS developed in
Germany to test the ease of use of tractor cabins
Ergonomic Analysis
 An avatar controlled by the user is interacting with the virtual
cabin while the system computes joint discomfort levels using
ERGONAUT (an ergonomic analysis tool)
Ergonomic Analysis
 Another use of VirtualANTHROPOS is to visualize reach
envelopes; The user can drive the avatar in real time using a
wireless body suit;
Link to VC 9.3 on book
CD
Personnel Training
 Training in airplane maintenance – task has a cognitive
component (manuals) and a tool/part manipulation component –
and they are sequential.
Personnel Training
 Task-related information is placed directly in the scene using
augmented reality. Results in faster information retrieval and
enhanced associative memory. System uses vision-based tracking
to recognize worker’s view and places text in relation to objects;
Personnel Training
Training system detect removal of cover and labels parts
underneath; then it detects the cap was removed and changes
dynamically the text to “4. Press to test”. If the test fails then
additional areas of interest are highlighted (“Filter Bypass”)
VR Marketing Applications
Citröen uses virtual
showrooms
VR Marketing Applications
VR Marketing Applications
Robotics Applications
 VR applications in Robotics/manufacturing relate to several areas:
CAD design and robot programming, making the process more
intuitive;
 Teleoperation (control at a distance) alleviating problems
related to poor visibility and large time delays;
 Multiplexed teleoperation, acting as a filter of particular robot
kinematics;
 Robots are also used in VR in haptic interfaces (discussed earlier in

our course).
 The “multi-modal
teaching advisor helps
novice operators
program welding paths
for industrial robots;
 It runs on a PC
networked with trackers
and laser range finder;
 Calculates the
difference between the
pre-computed (optimal)
path and user’s input on
the teach pendant. This
is presented graphically
on the user’s HMD
Robot programming
Robotic programming - continued
 Research done in Germany for offline robot programming done in VR,
with the programmer immersed in the
task he is programming;
 The programmer specifies the
trajectories, and the simulation
performs optimized collision
detection;
 Validation is done at run time
when the real robot is controlled
using the same computer and real
sensor data is used to fine-tune the
VR-generated program.
VR Robotics Applications - continued
 Research at University of Tokyo for the teleoperation of robots in
smoke-filled remote environments. Over-imposes the visual scene
from the remote robot with the virtual scene of a kinematically
identical robot. Thus VR acts as a guide to allow teleoperation.
Operator VR GUI
Degraded video feedback
(Burdea, 1999)
Teleoperation with large time delays
 Research at NASA developed a VR-based teleoperation to allow
operation despite large time delays. Works by controlling a “phantom
robot” which responds instantaneously to the operator. Allows
preview of the move, before it is executed.
Teleoperation with large time delays - continued
 Research at NASA drove the Mars rover using VR-based
teleprogramming;
 This was used to send high-level macro commands based on the
simulation of a virtual rover on a virtual Mars surface. This overcame
a 20 minute time delay!
Supervisory control
 Researchers in Germany developed a way to naturally controlling
robots through avatars;
 The users is immersed in VR and sees a scene with avatars to
which he is mapped;
 He interacts through gestures (measured by a sensing glove)
Supervisory control
 A real robot then interacts with the remote real environment;
 If the task is visual inspection then real images from the remote
site can be overlaid on the virtual scene, and thus seen by the user;
Real remote robot
Virtual robot with viewfinder
VR Robotic Teleoperation
 Research at Jet Propulsion Lab (California) allows the
teleoperation of a remote robot indirectly by controlling a motionguide trajectory, which the robot is then constrained to follow.
Motion guide
Remote robot arm
Task line
Text mode task editor
VR Robotics Applications - continued
 Research at University of Paris to allow multiplexed (one-tomany) teleoperation of kinematically dissimilar robotic arms;
 VR acts as a high-level filter masking the detailed slave robot
configuration; Translator then converts user actions to robot actions.
Operator task-level GUI
Remote robot arms
Supervisory control - continued
INFORMATION VISUALIZATION
 Represents the transformation of abstract data into 3D scenes;
 The information visualization pipeline allows the user to control
the view to the scene using an input device and select an area of
interest;
 The data extraction loop is asynchronous, so as to maintain
interactivity. It reads user input from a FIFO buffer;
 Time-varying data represent a complex case, as user time may not
coincide with the time clock used in visualizing the time-dependent
data.
Oil and Gas Exploration and Well Management
Oil and Gas Exploration and Well Management
Volumetric graphics
 In this class we learned about surface-based (polygonal or
spline) rendering only. This leaves the interior of virtual objects
hollow;
 Volume graphics renders the surface as well as the interior of
objects, called “voxels.”
Surface rendered object
Same object rendered volumetricly
voxels
Ray casting to create a 2-D image from volumetric data
View Plane
Tri-linear interpolation
Rays
Calculate gradients and lighting
Volumetric graphics advantages
 Much richer dataset;
 Objects appear more real;
 Can be displayed on same displays as surface-based models
Volumetric graphics disadvantages
 Much larger memory requirement (order of GB);
 Requires special boards to be real-time;
 Require special displays if auto-stereoscopy is desired;
 Much less used today – small software support base.
Volumetric graphics
Link to VC 9.7 on book CD
Volumetric Graphics Hardware
 Consist of volumetric rendering boards and of volumetric
displays;
 VolumePro 1000 is a graphics accelerator sold by TeraRecon Inc.
renders 512 x 512 x 512 voxels at 30 frames/second using a
Mitsubishi chip.
 For surface
geometry data
it works
together with
the graphics
board
installed on
the same PC
VolumePro Rendering Pipeline
VolucePro 1000
processed the 3D
data through ray
casting from a view
plane.
 Rays pick up
color and opacity
information by trilinear interpolation
to the nearest lattice
point
 Gradients are
then computed
VOLUMETRIC DISPLAYS
 Early models used LED matrix panel that translates back-forth on
rails; User can see stereo with bare eyes.
 Due to eye inertia, the image appears to float in space;
 But they had low resolution, noisy, monochrome (red LEDs);
 Did not have a 360º viewing area
Object section
LED
Volumetric Display
According to Wikipedia, a Volumetric Display is defined as a
device that “creates 3-D images by emission, scattering or relaying
of illumination from well defined regions in the (x, y, z) space.”
Almost all Volumetric Displays are autostereoscopic.
Currently, Volumetric Displays are only available to major
companies, the military, and certain schools for academic purposes.
They are still unavailable for the general public.
There are two main types of Volumetric Displays: Swept-Surface
Displays, and Static Volume Displays.
Rely on the phenomena known as persistence of vision.
Persistence of Vision
Persistence of Vision occurs when a human being is exposed to an
image for a very short amount of time, usually nanoseconds.
The result is around a millisecond of sight. This occurs because the
chemical reaction time it takes for a human to process an image is
much longer than the speed of light.
Swept-Surface Displays
Also known as Swept-Volume Displays
Rely on a rotating source or surface.
Project a 2-D image that constantly changes based on the
orientation of the rotating source or surface.
An example: the Perspecta, created by Actuality Systems, Inc.
Perspecta by Actuality Systems, Inc.
Static Volume Displays
Unlike Swept-Surface Displays, Static Volume Displays do not
rely on motion.
Some use pulsating, concentrated, Infrared Lasers that pulse
around 100 times per second to establish floating balls of plasma in
space which cause a clicking sound.
The exact point is known as a voxel (similar to a pixel would be in
2-D, except 3-D)
Others use lasers to cause visible radiation with a gas, liquid, or
solid.
Static Volume Displays
Use slideable mirrors and lenses to redirect the location of the
voxel that are usually built in.
Burton Inc, Keio University and AIST has made a true 3D display.
Disadvantages
Suck up a lot of bandwidth
Unable to deal with occlusion and opacity without limiting the
FOV: Image appears distorted
The cost of a volumetric display is much greater than that of its
closest competitors, such as virtual retinal displays (retinal scanning
devices). They use hundreds of times more light and power.
How far are we from Star Wars
A volumetric display device is a graphical display device that
forms a visual representation of an object in three physical
dimensions, as opposed to the planar image of traditional screens
that simulate depth through a number of different visual effects
The Graphics Lab at the University of Southern
California has designed an easily reproducible,
low-cost 3D display system with a form factor that
offers a number of advantages for displaying 3D
objects in 3D. The display is:
autostereoscopic
- requires no special viewing
glasses
omnidirectional - generates simultaneous views
accomodating large numbers of viewers
interactive - can update content at 200Hz
The display consists of
•A high-speed video projector
•A spinning mirror covered by a holographic
diffuser
•A FPGA circuitry to decode specially
rendered DVI video signals
•A standard programmable graphics card to
render over 5,000 images per second of
interactive 3D graphics, projecting 360degree views with 1.25 degree separation up
to 20 updates per second
ANISOTROPIC SPINNING
MIRROR
 3D
display uses an anisotropic holographic
diffuser bonded onto a first surface mirror.
 Horizontally, the mirror is sharply specular to
maintain a 1.25 degree separation between
views.
 Vertically, the mirror scatters widely so the
projected image can be viewed from multiple
heights.
 http://gl.ict.usc.edu/Research/3DDisplay/
About Encoding
The FPGA takes each 24-bit color frame of video and displays each
bit sequentially as separate frames . Thus, if the incoming digital
video signal is 60Hz, the projector displays 60×24 = 1,440 frames
per second. To achieve even faster rates, we set the video card
refresh to rates of 180-240Hz.
Sony RayModeler


http://blog.discover.sonystyle.com/360-degree-3d-prototype-takes-gamingto-new-levels
There is not much information and no paper for the device. However, we
can see the refresh in the video, reveals that is kind of TV technique
We can only guess from what we can see. In the 2 snapshots from
the SONY website. We can see RayModeler did not create 3D
scene just 360 show pictures to 360 degree. Maybe kind of rasterbased view control
DMD-based volumetric display
 The display produces 200 disk-shaped
slices each refreshed at 20 Hz;
 Resolution 768x768, 8 colors;
 10”-diameter spherical image;
360º x 180º viewing angle.
lamp
Translucent screen
Projection
DMD engine
Condenser
lens
Cold
mirror
(Favarola et al., 2001)
motor
fold
mirror
Rotating
mirrors
Actual system assembly
(Favarola et al., 2002)
Auto-stereoscopic 3-D Display produced by Actuality Systems
(www.actuality.com)
Comparison of Volume Displays
Light Field Display
Sony RayModeler
Perspecta
Core Structure
Spinning Mirror
Special arranged
LED
Rotating Screen
Shape
Sphere
Cylinder
Sphere
resolution
768*768*288
96*128
768*768*198
Contradiction
high
high
low
Color
BW by 1 face mirror
Multicolor by multiface mirror
Colors
8 Color
Fresh Rate
20 Scenes/sec
Interactive
handle
price
Claim to be a cheap
Solution
24 Scenes/sec
Gesture, Game pad
Contact, Gesture
New Interaction Techniques with Volumetric Displays
(Balakrishnan et al, 2001)
Depth perception studies done at University of Toronto (2006)
 Compared desk-top mono, stereo and
stereo with head tracking with depth
perception in a volumetric display
 Tasks were 1) judge the position of a
sphere in a cube; 2) decide if two objects
were going to collide or pass by eachother
http://www.dgp.toronto.edu/~tovi/#Publications
Selection techniques study done at University of Toronto (2006)
 The display was this time only volumetric, but object selection technique varied
 Within a cluttered virtual world a ray technique will intersect several objects not
just the target object;
 Selection by point cursor, depth ray, lock ray, flower ray and smart ray (below)
 Movement times were largest for the smart ray and smallest for depth ray;
 However error rates for new techniques were smaller (13% depth ray, 11% lock
ray and flower ray and 10% smart ray) vs. point cursor (21%).
http://www.dgp.toronto.edu/~tovi/#Publications
Sources
http://charafantah.wordpress.com/2008/01/06/running-outof-time-%D9%85%D9%81%D9%8A%D8%B4%D9%88%D9%82%D8%AA/
http://www.youtube.com/watch?v=8KaQmn2VTzs
http://www.youtube.com/watch?v=He2QTpelAjE
http://en.wikipedia.org/wiki/Persistence_of_vision
http://en.wikipedia.org/wiki/Volumetric_display