7 July 2006
PhD Defense
Emanuele Ruffaldi
Scuola Superiore S.Anna, PERCRO

Robotic systems that provide kinesthetic and tactile stimuli

Classic Robot
Teleoperated System
Networked Teleoperation
Haptics in Virtual Environment
Haptic Collaborative
Virtual Environment
Augmented Reality with Haptics

● Realism of Interaction is not a requirement
● Integration with advanced visualization techniques and technologies
● Haptics fits in the Multimodal approach to VE
Visual
Haptic
Physical

●
●
● Generate a force feedback depending on the interaction of the user in the Virtual Environment
The user handles the device with a stylus or constraints its fingers
The interaction in the Virtual Environment is tool mediated, we call this tool the Haptic Handle, usually a sphere
3-DOF device
2 point 3-DOF device
How compute the force depending on the position of the haptic contact point?
It is the objective of Haptic Rendering

●
●
In 3-DOF Haptic Rendering we represent the contact point as a small sphere and we suppose it is able to move around in space
The force is obtained by computing a proxy point that is always outside the surface (Zilles95, Ruspini97)
●
●
Surface rendering can be enhanced by friction, force shading and textures
The performance of a 3-DOF rendering depends on the way the algorithm computes the intersection with the geometry
●
●
3-DOF rendering can be used for exploring surfaces, pushing objects, perceiving force fields
A device with Two 3-DOF points can be used for grasping objects

●
●
3-DOF is not adequate when the Haptic Handle has a complex shape and in specific tasks as Virtual Prototyping
6-DOF rendering can be used with 3-DOF devices for simulating complex interactions
Haption 2005
Otaduy 2003
Higher complexity in the Collision Detection
Compute force and torque at the contacting points
Involves a Dynamic Simulation

●
●
●


The force computed by the 6-DOF Collision Response could be applied directly to the body (Direct Rendering)
The varying number of contact points would produce a varying stiffness
Performance depends on the speed of the Collision Detection
Virtual Coupling overcomes these problems by smoothing the interaction (Colgate95, McNeeily99)
In VC the body and the haptic handle are connected by a damped
6-DOF spring
Haptic Handle
Virtual Body

Haptic Systems can be decomposed using a layered approach similar to the OSI stack for networked applications. Research topics can be described relative to this layering
Application
GUI and Effects
Dynamic Simulation
Haptic Rendering
Device Abstraction
Device Driver
Hardware
Haptic Stack
Application
Presentation
Session
Network
Transport
Data
Physical
OSI Stack

Application
GUI and Effects
Dynamic Simulation
Haptic Rendering
Device Abstraction
Device Driver
Hardware
Application
GUI and Effects
Dynamic Simulation
Haptic Rendering
Device Abstraction
Device Driver
Hardware
Application
GUI and Effects
Dynamic Simulation
Haptic Rendering
Device Abstraction
Device Driver
Hardware
●
●
●
●
●
New Devices

Working Tactile device
Haptic Rendering for 6-DOF

What is missing? High resolution meshes handling
Multi-finger Haptics

Realistic models for grasping
Deformable Haptics

Domain specific deformable models
Integration with applications

Medical simulators and games
Application
GUI and Effects
Dynamic Simulation
Haptic Rendering
Device Abstraction
Device Driver
Hardware
Application
GUI and Effects
Dynamic Simulation
Haptic Rendering
Device Abstraction
Device Driver
Hardware

● Improve the realism of Haptic interaction for Object Manipulation
● 6-DOF handling independent of the geometry
● 3+3-DOF realistic grasping of objects
● Evaluating the results using benchmarking

A complete solution for the manipulation of objects using Haptic interfaces
● A Soft-Finger Proxy for grasping objects using two finger 3-DOF
● A 6-DOF Volume Based algorithm for
Collision Detection and Response
● Technology integration for easy development of Haptic applications

DLR 2005
Application
GUI and Effects
Dynamic Simulation
Haptic Rendering
Device Abstraction
Device Driver
Hardware
Concept: create an object grasping model based on the concept of human finger compliance. Provide such representation by introducing in the VE the fingertip deformability (soft-finger)
Objective: provide and validate a realistic grasping of virtual objects using the
Haptic interface based on the soft-finger model modeling
Robotic
Grasping
Haptic
Grasping
Human
Grasping perception
The perceptual work starts from early research in cognitive sciences.
The mathematical models come from the problem of design robotic hands Friction assessment of the human fingertip grasping, Johansson 99

●
●
●
The finger interaction is simulated using two 3-DOF proxy enhanced with linear and rotational friction
The force feedback computed on the proxy is sent to the Dynamic
Simulator for balancing the linear and rotational sliding due to gravity
The device used is a 3-DOF device with two arms (GRAB) but the algorithm is ready for an evolution of the device providing torques

●
●
●
●
The classic friction cone algorithm uses the proxy position for simulating stick-slip friction
The algorithm has been enhanced with the rotational component
The linear condition on the position is a coupled condition on the relative position and rotation
It allows to take into account the real coupling between the two types of friction in human grasping

●
●
●
An experiment in which users has been asked to lift object with different weights and stiffness coefficients
The Haptic interface allows a simulation of the operation that is coherent with the physical parameters
But the security margin of the forces applied by the user are much higher than in the real case

Because the current haptic interface do not transmit information about the contact area
Smaller stiffness generates forces that are easier to be controlled by the Human, but the surface is less realistic
A possible future

rotation rot-slide
The coupled algorithm allows the combined behavior of sliding and rotation in the range L < L
0
It allows the simulation of a real behavior of an object sliding and rotating between the fingers
Without the algorithm the objects just slides
L
0 is the contact area

●
●
The algorithm has been implemented in a real system and tested also using Simulink
The figure shows how the coupled solution allows the simulation of the rotation during the sliding

Multirate architecture for the Soft-Finger
The system is able to provide stable interaction for the grasping with multi-finger computation of the forces.
This Haptic library for Virtual Environments written in C++ has been named EHAP

● Point rendering is useful for exploration of surfaces (PureForm 01)
● Point rendering is fast (AABB tree or Local model)
● But it is limited when the interaction tool is more complicated (McNeeily99)
● Also certain applications, like Virtual
Prototyping are not possible with point rendering

Simulation
F,T Rigid Body
Simulation
X,R
Contact response
Haptic device
X, R
-F c
, -T c
Virtual
Coupling
F c
, T c
Control
Algorithms
F r
, T r
Haptic Rendering
1 kHz
Collision detection
C
Contact
Management
C
Collision Detection
300 kHz

●
●
●
Provide 6-DOF simulation of teeth interaction in a planning tool for craniofacial surgical operations
Given an imaging dataset made of X-rays
Only one algorithm is in literature the Voxel Point Shell by
McNeely, with some limitations
Voxel Point Shell
●
●
●
●
●
The point shell is a set of surface point with normals for the Haptic Tool
The Collision Detection is obtained testing each point of the shell against a static voxel world
The Collision Response uses a single voxel penetration
The number of Contact points is managed just by a mean
The algorithm quality is limited by number of samples

●
●
●
●
Collision Detection based on a Voxel Volume and Implicit Sphere
Tree


Global information on the collision (in VPS is local)
No dependency on point shell sampling

Any object can be tested
Collision Response for the reduction of number of points and fast computation of the dynamic simulation of the body


No mean applied
No need for braking
Final Haptic Rendering based on Virtual Coupling

Same as in VPS for smoothing the interaction
The voxel volume has been obtained from X-rays scans of teeth and voxelized by segmentation using Amira

The voxelization of mesh models uses a simple flood filling algorithm
Application
GUI and Effects
Dynamic Simulation
Haptic Rendering
Device Abstraction
Device Driver
Hardware

●
●
●
The Voxel Volume is stored in a Octree
Information per Voxel and per Node can be selected at compile time depending on tradeoff between memory and speed:




Voxel Type (free, full, surface, proximity)
Distance to surface
Normal/Gradient
Normal Cone (additional Collision Detection optimization)
The CD Bounding Volume Hierarchy Is computed implicitly from the Octree


No memory usage
Additional optimizations provided by octree storage
Octree
Node Occupancy optimization

●
●
●
●
The Collision Response is based on an impulsive force approach that selects the best contact point, in simultaneous mode without requiring another CD
Contact points are excluded if separating
This approach removes the need for clustering
The impulse is applied using the formula (frictionless in this case):
1 2
3 4
Resolution of contacts by two impulses n p
1
δ p
0 p
1 n δ p
0
(not) separating contact points

●
●
●
●
●
For the completeness of the system the visualization as been addressed
Possible solutions:


Mesh from the isosurface using marching cubes
(~ 1M triangles for the 256 3 teeth)
Volumetric rendering based on GPU
(3D texture and advanced use of shaders)

Q-Splat
(introduced for rendering meshes of > 100M triangles)
Q-Splat is a point based technique that replaces every voxel with a single screen aligned quad or ellipse.
It provided level of detail rendering using the distance of the octree cube from the viewer and replace all the octree node with a single splat
It has been chosen because it uses the Octree and the Normal cones of the Voxel
Volume geometry
●
●
Coupled rendering using
CPU and some GPU
It can be used to visualize the current status of each voxel/node during the collision

● Algorithms should be measured and evaluated
● How this can be done with Haptics?
● Proposing a Benchmarking Framework for
Haptic Rendering algorithm based on real measurements
● In this first phase the algorithm addresses
3-DOF rendering

3D model scan
Haptic renderer traj out trajectory
(pos + force) extract in trajectory out trajectory timings comparer result

● Evaluation of relative performance of algorithms
● Quality of rendering
● Computational performance
● Identification of errors

●
●
●
●
Requires skills on device and algorithms
Integration with existing software is challenging
Sometimes the developer is just a psychologist that want to perform an experiment
Conceptually should be just little harder than 3D graphics
Objective - Improve Haptics from the point of view of the Platform and Development Tools

Application
GUI and Effects
Dynamic Simulation
Haptic Rendering
Device Abstraction
Device Driver
Hardware
● A system for developing Haptic applications on the Web or in VR installations
● Based on the eXtreme Virtual Reality
(XVR) technology
● Providing low level Haptic Rendering and
Dynamic Simulating
● Plus higher level interface for integrating
3D graphics and haptics

Pool for ENACTIVE 06
EHAP HapticWeb applications
ENACTIVE 06
CREATE 2004
Architectural contribution to
PERCRO projects
VICOM 2006

position graphics loop native loop 1kHz force graphics loop force script loop ~1kHz force position graphics loop force script loop ~1kHz plane native loop 1kHz force position

Application
GUI and Effects
Dynamic Simulation
Haptic Rendering
Device Abstraction
Device Driver
Hardware

● Based on the Novodex Dynamic Engine

●
●
●
●
Enhance the proposed algorithm to deformable objects
Enhance the technological solution using a GPU based approaches
Semantic Haptic Interaction, moving from low level Haptic Rendering to semantic manipulation of objects in VE
Visual Programming with Haptic behavior, taking into accounts the perceptual properties of
Haptics

●
●
●
●
●
This work has addressed one of the typical aspect of
Human interaction. Manipulation of objects.
A new realistic soft-finger proxy for rendering the grasping based on coupled linear and rotational friction
The introduction of a new Voxel Volume Collision
Detection and Response algorithm for 6-DOF rendering, with high performance
A framework for the Benchmarking of 3-DOF haptic rendering algorithm based on real measurements
Finally HapticWeb, a system for the development of haptic enabled application, for VR and desktop systems.
Provided to PERCRO and to external developers

● Thanks to all PERCRO people
● and nothing could be possible and meaningful without Elisabetta
● This Thesis is dedicated to my Parents