SGN-5406 Virtual Reality 2012
Atanas Boev based on material by
Stanislav Stankovic and Ismo Rakkolainen
SGN-5406 Virtual Reality 2012
Department of Signal Processing
1

Physical I/O devices
Part 1: Input (haptic sensors) Part 2: Output (non-visual displays)
2
Passive Active Haptic Vestibular
No feedback
3D “fly” mouse
Wiimote
PS Move
Kinect
VR Gloves
3D Probes
Accelerometers
Etc…
Movement support
Keyboard
Mouse
Joystick
Touch screens
Steering wheel
3D “desk” mouse
(SpaceNavigator)
Etc…
Department of Signal Processing
Intentional feedback
Combination of input+output
Surgery simulator
Novint Falcon
CyberGrasp glove
Tactile Kinesthetic
Virtual keyboard + force feedback
SGN-5406 Virtual Reality 2012
Endeffector
Force feedback
Olfactory

Department of Signal Processing
SGN-5406 Virtual Reality 2012
3

• Present the artificial computer generated stimuli to the user.
• Provide the information about the state of VR environment to the user.
• Provide feedback about the results of user’s actions.
• Work with human sensory organs:
• Visual system
• Auditory apparatus
• Haptic, (tactile, kinesthetic)
• Rarely with other senses – olfactory (smell), taste, etc.
Department of Signal Processing

Display = Output Device
Video display:
• Computer Screen
• Projector
• HMD – Head Mounted Display
• 3D Screens
Audio display:
• Speakers
• Headphones
Haptic display:
• Tactile displays
• Force Feedback Devices
Department of Signal Processing

Department of Signal Processing
SGN-5406 Virtual Reality 2012
6

• Tactile – sense of touch
• Kinesthesis – sense of joint position and effort
• E.g. carrying heavy load affects both tactile (pressure on hands) and kinesthetic (muscle effort) senses.
• The tip of the finger
• The most sensitive
• Even 20 nm movement
• Termoception – sense of hot and cold (15-45°
C, 0.001°/s)
SGN-5406 Virtual Reality 2012
Department of Signal Processing

• Tactile - provides artificial stimuli (touch, vibration) to receptors in our skin
• Kinesthetic – provides artificial sense of force
• End-effector – limits the natural movement, thus providing feeling of solid objects / passive resistance
• Force feedback – applies force, gives sense of active movement / impact
SGN-5406 Virtual Reality 2012
Department of Signal Processing

• Haptic output needs to be correlated with video and audio stimuli.
• Vibrations of gamepad paired with events in the game (character getting hit).
• Vibration of a mobile phone while phone is ringing.
• Vibration of the device when a virtual key is pressed on the touch screen.
Department of Signal Processing

Department of Signal Processing
SGN-5406 Virtual Reality 2012

Department of Signal Processing
SGN-5406 Virtual Reality 2012
11

• Actuators, devices by which artificial stimuli is presented to human skin.
• Actuators present some force to a region skin.
• Types of actuators:
• Blader actuators – pneumatic, hydraulic
• Vibrator actuators – electromechanical
• Pin actuators – electromechanical
• Piezoelectric devices.
• Electro-active polymers.
• Etc.
Department of Signal Processing

• Pockets that can be expanded and contracted:
• By controlling the flow of air (pneumatic)
• By controlling the flow of liquid (hydraulic)
• Strategic placement of the pockets creates the sensation of pressure on different areas of the participant’s hand and body
Department of Signal Processing

• Most often found in mass market products:
• Game controllers
• Mobile phones
• Portable consoles
• Data Gloves
• Simulator seats
• Easier to control then other types of devices.
• Very robust and easy to implement.
• Can’t convey the sense of surface texture or shape of the object.
• Offer a limited range of effects.
• Low-frequency speakers (subwoofers) can also be used as a vibratory display.
Department of Signal Processing

• Belt with 12 vibration devices.
• At any given moment the device facing north vibrates.
• Augmentation of human senses. Humans do not have an explicit compass sense. This device gives acute awareness of sense of direction.
• See
• Udo Wächter, University of Osnabrück
• http://feelspace.cogsci.uni-osnabrueck.de/
Department of Signal Processing

• Small pin arrays placed on each finger.
• Height of pins controlled electronically.
• Textures are detected by pressure variations across the fingertip over time.
• Good for displays for blind people
Department of Signal Processing

Northrop Grumman’s TerrainTable.
Array of 4600 pins push up touchsensitive silicone screen.
Overhead projector projects 2D map.
Military applications.
Department of Signal Processing

• Can very rapidly present temperature fluctuations, typically to finger tips.
• Danger of tissue damage.
• Limits must be adjusted to the safe range of temperatures tolerated by human skin.
• Not the ”real” temperature of the simulated objects.
• Use scenarios rare, not many practical implementations.
Department of Signal Processing

• Tactile feedback on touch screens.
• Screen is covered by a grid of electrically activated elements.
• Tixel a tactile pixel.
• Tixels generate a controlled electric field which extends several millimeters above the surface.
• Exploits electro-sensory phenomenon.
See: http://senseg.com/technology/system-architecture
Department of Signal Processing

• Ultra-low electrical current is passed into the insulated electrode – the tixel.
• Electrical charge creates a small attractive Coulomb force to finger skin.
• By modulating this attractive force, any number of touch sensations can be generated from vibrations, clicks, textured surfaces, etc.
Department of Signal Processing

• SIGGRAPH 2008 Emerging
Technologies: Airborne Ultrasound
Tactile Display
• Mid-air tactile sensations by means of airstreams generated by ultrasonic actuators
• SIGGRAPH 2009 Emerging Tech:
Touchable Holography
• See: http://www.youtube.com/watch?v=Y-
P1zZAcPuw
SGN-5406 Virtual Reality 2012
Department of Signal Processing

Department of Signal Processing
SGN-5406 Virtual Reality 2012
22

• Force Feedback – End-effector Devices.
• End-effector devices are a special class of force feedback devices.
• Users limbs, hands, arms, legs are in contact with machinery.
• Input and output device.
• Movements on hands serve as input.
• Device provides feedback through the active force.
• Generally linked to mechanical tracking sensors.
Department of Signal Processing

• Stanford Research Institute,
SRI International.
• M7 Surgical Robot.
• Developed for NASA.
• Can perform operations in
Zero gravity.
• Compensates for unwanted movements in Zero-G.
Department of Signal Processing

• UNC Nanomanipulator.
• Microscop image on screen.
• Mechanical arm with 6DoF.
• Forcefeed back.
• See: http://cismm.cs.unc.edu/tag/nanomanipulator/
Department of Signal Processing

CyberForce system by CyberGlove
Systems.
Mechanical system that exerts force on hand and arm.
Sense of weight and inertia while picking up a "heavy" virtual object
Feel resistance of a simulated objects.
http://www.cyberglovesystems.com/ products/cyberforce/overview
Department of Signal Processing

Mechanical trackers.
Force generating device.
Technologies:
• Electronic motors – one for each DoF,
• Hydraulics,
• Pneumatics
Department of Signal Processing

• Often operates also as an input device, potentially providing resistance to input controls.
• Mechanical movement sensors are generally incorporated directly into the system. Mechanical tracking is generally very fast and accurate.
• Typically operate with respect to single point in the virtual world.
• Number of DOFs: 1-6.
• Tactile displays e.g. vibrator actuators can be mounted within the end-effector.
• Can also be constructed to enhance the user’s force.
Department of Signal Processing

End-effector devices which affect user’s whole body.
Two mayor types:
• Exoskeletons
(wearable robotics)
• Surround platforms
Department of Signal Processing

• Exoskeletons – Meaning external skeleton.
• Can work with whole body or just with some parts.
• Exoskeletons are not just I/O device.
• Applications:
• Military:
• Enhance the power of user’s body or limbs. Increase the load a person can carry or prologue the time a person can endure the load.
• Medical:
• Restore mobility in paralyzed limbs.
• Telepresence:
• Operate machinery at a distance, in environments dangerous for humans.
Department of Signal Processing

• Novint XIO game controller.
• Partial exoskeleton.
• Arm exoskeleton.
• Mass market product.
Department of Signal Processing

• Hybrid Assistive Limb.
• Cyberdine Inc. And Tsukuba Uni.
Japan.
• HAL detects bioelectrical impulses sent from brain to limbs, using a set of electrodes on the skin.
• Possible applications:
• Rehabilitation,
• Physical training,
• Help for disabled people,
• Rescue services,
• Entertainment (as input device)
Department of Signal Processing

Department of Signal Processing
SGN-5406 Virtual Reality 2012
33

• Work with human vestibular apparatus (sense of equilibrium).
• Work by physically moving the user.
• Artificial sense of equilibrium, acceleration and orientation that user is expected to feel during a real motion.
• Strong relationship between the vestibular and visual systems of humans.
• Prevents (or causes) simulator sickness.
• Makes better immersion.
• Typically used in specialized simulators (flight simulators).
Department of Signal Processing

• D-BOX Motion Code movie seat.
• Moving seat platform – 4 to 6DoF depending on model.
• D-BOX Motion controller device.
• Movie theaters and amusement parks.
• http://www.d-box.com
Department of Signal Processing

• Vestibular apparatus works on the same principle as human auditory system.
• Works on the same principle as Cochlear implant.
• Sending specific electric messages to a nerve in the ear that transmits balance information.
• Can alter one’s perception of balance.
• See also: lecture VR4.3
• Side effect are still not known.
Department of Signal Processing http://www.youtube.com/watch?v=ZlNfBrXYYTc

• User needs to remain within certain limited physical space.
• User needs to have an impression of physical movement in virtual space.
• Simulate the physical movement of the user in VR environment, while keeping user in the same physical position.
• Examples:
• Omni-directional treadmills.
• CirculaFloor.
Department of Signal Processing

• Two treadmills placed on top of each other.
• Movement along X and Y axes.
• Platform provides the sensation of moving ground.
• Suspension system to hold the user in place.
Department of Signal Processing

• Several movable tiles.
• With each step a new segment moves infront of user’s feet.
• Movement of the segment counters the movement of user while maintaining the illusion of motion.
Department of Signal Processing
See: http://youtu.be/rYsvB2y2Ero

• Rendering – process of generating of artificial stimuli, based on the current state of VR environment.
• Needs to be fast, up to 1000 Hz.
1. 3D objects & properties loaded from the database.
2. Collision detection. Only colliding objects are passed on.
3. Compute collision forces, smoothing, mapping.
4. Haptic texturing (vibrations, temperature, etc.).
5. Present output to the user through haptic display.
Department of Signal Processing

Department of Signal Processing
SGN-5406 Virtual Reality 2012
41

Digital signals in software code trigger the generator to emit precise amounts of the appropriate aroma.
SGN-5406 Virtual Reality 2012
Department of Signal Processing

• Artificial smell simulation
• Mostly at research, prototype phase, some niche products
• Film: Scent of Mystery (1960), with accompanying smells
• Sensorama 1960, Smellizer 1984
• Enhanced cosmetics, perfume, food advertising, aromatherapy
• Aromas can be used to enhance the experience and trigger fear, excitement and other emotions
• Multi-sensory magazines (Esquire cover, smell etc.)
• Companies
• http://www.structuralgraphics.com/sensory-effects/
• http://www.aerome.com/
• http://www.aromajet.com/
• http://www.digiscents.com/
• http://www.trisenx.com/intro.html
SGN-5406 Virtual Reality 2012
Department of Signal Processing

• ScentScape™ specifications:
• Provides 20 basic scents per cartridge
• Scent cartridges last 200 hours or more in heavy use, depending on personal settings
• ScentScape™ scent cartridges will be available in standard and media-specific versions
• Separate "volume control" to adjust overall smell strength for personal preference
See: http://www.scentsciences.com/products/scent_scape.html
SGN-5406 Virtual Reality 2012
Department of Signal Processing