Berkshire Encyclopedia of Human-Computer Interaction, W. Bainbridge, Ed., Berkshire Publishing Group, 2004, pp. 311-316.
Haptics
Ralph Hollis
Carnegie Mellon University
Haptic interaction with the world is manipulation using our sense of touch. The term “haptics” arises from the Greek
root haptikos, meaning “able to grasp or perceive.” Haptic interaction with computers implies the ability to use our
natural sense of touch to feel and manipulate computed quantities. Haptic computer interaction is a relatively new
field that has generated considerable interest over the past decade.
A new way to interact with computers
Initially, computers could deal only with numbers. It took many years to realize the importance of operating with
text. The introduction of CRT display technologies allowed graphics to be displayed, giving us a new way to
interact with computers. As processing power increased over time, three-dimensional (3D) graphics became more
common, and we may now peer into synthetic worlds that seem solid and almost real. Likewise, until recently, the
notion of carrying on a “conversation” with our computer was far-fetched. Now, speech technology has progressed
to the point that many interesting applications are being considered. Just over the horizon, computer vision is
destined to play a role in face and gesture recognition. It seems clear that as the art of computing progresses, even
more of the human sensory palette will become engaged.
It is likely that the sense of touch (haptics) will be the next sense to play an important role in this evolution. We use
touch pervasively in our everyday lives, and are accustomed to easy manipulation of objects in three dimensions.
Even our conversation is peppered with references to touching. Blake Hannaford at the University of Washington
has compiled a list of verbal haptic analogies: we frequently make the analogy to haptics when we speak of our
relationship to ideas, people, and information. We often use phrases like “get a feel,” “poke [into/around],” “put
one's finger [on the problem],” when referring to exploration. We use phrases like “[to stay/keep] in touch,”
“tangible [concepts],” “[a] touchy [subject],” “hands-on learning” (often used literally), and “at my fingertips” when
referring to contact. And phrases like “pressing issues,” “pushy [behavior],” “hard hitting [presentation],” “get/lose
a grasp [of the situation],” and so forth are used when referring to impact or manipulation. In fact, it is quite
surprising, given our natural propensity to touch and manipulate things, that haptic computer interfaces are not
common.
We principally use our hands to explore and interact with our surroundings. The hand is unique in this respect
because it is both an input device and an output device: sensing and actuation are integrated within the same active
living mechanism. An important question is how best to transmit haptic information between a running computer
program and a user's hand. Providing position input to the computer from the hand is easy, providing force/torque
output from the computer to the hand has proven to be difficult. We don't understand how to invent very good
haptic devices which will link our hands in some way with a running computer program---and for that matter, we
don't understand very well how best to write programs that can derive and serve up haptic information for our
consumption.
Our tactile and kinesthetic senses
How does the skin detect pressure, friction, and vibration? We know the hand is a complicated system that includes
articulated structure, nerves, muscles (output), and senses (input). But the hand's sensory capabilities are, at best,
imperfectly understood. Because of this, fully informed design methods for haptic interfaces do not yet exist.
The sensory suite forms part of the somatosensory system which has modalities of discriminative touch (including
touch, pressure, vibration), proprioception (including joint angle, muscle length and length rate of change, and
tendon tension), and pain/temperature (including itch and tickle). Chiefly the first two modalities are important for
haptic perception.
The discriminative touch modality relies on four different kinds of receptors in the glabrous (hairless) skin of the
hand. They are Meissner's corpuscles, Pacinian corpuscles, Merkel's disks, and Ruffini endings. Both the Meissner's
and Pacinian corpuscles are considered to be rapidly adapting (RA), responding mostly to changing stimuli, whereas
the Merkel's disks and Ruffini endings are considered to be slowly adapting (SA) and continue to fire in the
presence of constant stimuli. Whereas the anatomical characteristics of these receptors are known, their precise role
in psychophysical perception is less well understood. The Pascinian corpuscles respond to high frequency
vibrations such as those encountered when running one's finger over a textured surface. The Meissner's corpuscles
are sensitive to sharp edges, the Ruffini corpuscles to skin stretch, and the Merkel's disks to edges and pressure.
The proprioception modality (kinesthesia) is also of major importance, although its receptors are less well
understood compared to those of discriminatory touch. The joint angle receptors incorporate Ruffini endings and
Pascinian corpuscles located at the joints, which respond to pressure applied to the receptor. Interestingly, subjects
can resolve distances between thumb and forefingers corresponding to about 2.5 degrees. Muscle spindles, located
between and among muscle fibers, report muscle length and rate of change of length. Being modified forms of
skeletal muscle fibers, they can not only send information to the brain, but can also receive commands causing the
spindle to contract, resetting its threshold setting. Golgi tendon organs, which respond to tension force within the
tendons, seem to play a role in muscle control. In a haptic interface, both the proprioception and discriminative
touch modalities play important roles in perception. Given these considerations, we must ask what to put (in,
around, in contact with, etc.) the hand in order for a running computer program to impart a realistic sensation of
touch to a user. There are many possible approaches to answering this question.
Haptic interface devices
Haptic interaction between a person and a computer requires a special device that can convert motion of the person
to meaningful quantities that can be input to the computer, and at the same time, convert computed quantities to
physical forces and torques that can be felt by the person. Many different kinds of devices have been invented that
enable haptic interaction with the whole hand, or arm, or even the whole body. We consider here only the type of
device that can be engaged by the hand. Haptic devices can have several degrees of freedom (DOF). For example,
an ordinary mouse device has two DOFs; that is, it can be moved left/right, and also moved forward/backward,
yielding two independent position measurements for input to the computer. Similarly, the familiar joystick gaming
device can be tilted left/right or forward/backward, yielding two independent angles. Neither the mouse nor the
joystick are haptic devices, since they provide only input to the computer. Adding motors or other actuators to these
devices would permit haptic interaction, since they could then serve not only as input devices, but also as output
devices.
Haptic devices with 3 DOFs allow translation in 3D, and those with 6 DOFs allow both translation and rotation in
3D. Haptic devices usually employ either serial or parallel linkages whose joint positions are measured with
encoders and whose joint torques are affected by motors. One may recognize that the preceding description also
applies to robot arms; in fact, many early haptic interface devices were based on robot arms. In the case of a robot
arm, the computer commands the arm’s joint motors to move the arm tip to a particular position and orientation in
space. As a haptic device, the user holds the arm tip, moving it to a particular position/orientation as the computer
monitors the joint angles. When the computed position of the tip reaches a place where motion is no longer wanted
(according to the virtual environment model), the motors are turned on to provide stiff resistance to further motion.
For this scheme to work, the arm must be back-drivable, or else the user’s hand force and torque must be measured
at the tip and used to control motion.
Critical considerations for haptic interface devices include degrees of freedom, bandwidth, resolution, and stiffness
range. First of all, haptic interaction with a realistic 3D virtual environment has proven difficult with devices having
fewer than 6 DOFs, but providing six-DOF capability often implies high mechanical complexity and cost. Second,
the response of the device should be as “lively” as possible; otherwise interaction with the virtual environment will
feel sluggish. The hand is capable of feeling frequencies of several hundred Hz, but achieving the highest possible
bandwidth in an electromechanical device is challenging. Third, for the hand to feel small nuances of the virtual
environment (e.g., fine texture or stick-slip friction) the device must be able to resolve extremely small changes in
position and force. Finally, when the user moves through free space, nothing should be felt; when a hard virtual
surface is contacted, it should feel perfectly rigid. In practical systems, neither of these objectives is achieved,
however the range of possible stiffness felt by the hand should be maximized. Haptic devices primarily engage the
proprioceptive senses in the hand and arm, but also involve the discriminative touch sensors, chiefly through highfrequency effects. Collectively, bandwidth, resolution, and impedance range determine the fidelity of the haptic
device. The ideal haptic interface device has yet to be invented. One popular commercial device is a small cabledriven highly back-drivable 3-DOF arm with three encoders and three motors. A new approach uses magnetic
levitation to provide 6-DOF interactions over limited ranges of motion with a single moving part.
Haptic rendering
To be able to “feel” computed quantities, there must be algorithms and computer programs capable of deriving
correct force and torque values for a given situation to be output to a haptic interface device. The term haptic
rendering is used to describe these operations, in analogy with the familiar rendering of graphics on a computer
display. Normally, haptic rendering is accompanied by simultaneous graphical rendering in what is more properly
referred to as a visual/haptic interface. Unlike graphical rendering which can satisfy the eye at update rates of 30
frames per second or even less, haptic rendering must be done at rates approaching a kilohertz to feel right to the
hand.
In many cases, one may desire to interact haptically with three-dimensional objects modeled in the computer. For
example, suppose we have modeled a cube and a cone by using mathematical formulas that define their surfaces,
and we wish to be able to touch these virtual objects with a point-like probe. The haptic system establishes a one-toone correspondence between the (virtual) probe point and the position of the haptic device handle or manipulandum.
This is very much like the relationship between a computer mouse on the desktop and the cursor on the computer
display screen. As the user moves the probe point about in three dimensions by moving the manipulandum, the
computer checks whether the point is outside an object in free space or inside an object, an operation termed
collision detection. This test must be done very rapidly, perhaps a thousand times a second. As long as the probe
point is in free space, the device motors are turned off and are able to turn freely. As soon as the point is determined
to be inside the virtual cube or cone, the motors are turned on providing torques to the device joints, which generate
a stopping force on the manipulandum. If the user attempts to push the point farther into the virtual object, the motor
currents are increased further to resist the motion. The user thus experiences a sensation of contacting the surface of
a real object. With a 3-DOF haptic device, the user may freely slide the point along the surface of the cone or
surfaces of the cube, feeling their shapes.
With a 6-DOF haptic device, it is possible to interact with a 3D virtual tool instead of just a point. The haptic system
associates the position and orientation of the virtual tool with the position and orientation of the manipulandum. In
our example, if the user contacts the virtual cube or cone with, say, a cube-shaped virtual tool, he or she will feel
torques as well as forces, as a surface of the virtual tool rotates into contact with a surface of the cube or cone.
Collision detection for a virtual tool is much more complicated than that for a point.
For either a point or virtual tool, once the virtual object surface is contacted, realistic contact forces and torques must
be derived. Researchers have developed a number of different algorithms for this. For example, if the surfaces are
supposed to feel rigid, a very stiff spring is modeled. If the object is deformable, its surface is modeled as a network
of springs that can deform much like a mattress deforms. Surfaces of virtual objects need not be smooth. Various
texture and friction rendering algorithms have been developed. In all these methods, there is a tradeoff between
rendering accuracy and rendering time that is severely taxing on computer resources. There are also issues of control
stability. When we interact haptically with real objects, energy is almost always dissipated, but if the interaction
between a haptic device and a virtual environment is not correctly modeled, energy can be generated leading to
vibration and sudden loss of control. Finding the best haptic rendering algorithms for a given situation continues to
be an active area of research.
Psychophysics of haptic interaction
One may characterize the operation of a haptic interface in terms of its engineering parameters, but in the end its
perception by the user is what really matters. Physiologists and psychologists have studied the human sense of
touch over many decades. Psychophysics is the scientific study of the relationship between stimuli (specified in
physical terms) and the sensations and perceptions evoked by these stimuli.
Researchers have striven to characterize and distinguish psychophysical responses from the discriminative touch
sensors by performing experiments. For example, Klatzky and Lederman conducted an extensive psychophysical
analysis of haptic performance under conditions where the fingertip is covered by a rigid sheath, held in place by a
thin rubber glove. This eliminates the spatial pressure gradient normally provided by the mechanoreceptors
(particularly the SA receptors such as the Merkel's disks) and provides only a uniform net force (with, possibly, a
gradient at the edges of the sheath that is unrelated to the felt surface), simulating conditions encountered when
using a haptic device.
Knowledge about the discriminatory touch modality can inform haptic interface design. It would seem that haptic
sensation of subtle effects relating to texture and friction, that tend to be high-frequency phenomenon communicated
mostly through the user's skin, is of major importance. It is precisely these high-frequency effects that permit a
person to rapidly assess conditions in the real world. For example, when a machinist removes a part from a lathe,
the first instinct is to feel the quality of the part’s finish; only later is it visually inspected and measured.
Recently, psychologists have begun performing experiments to evaluate the efficacy of haptic interaction systems,
i.e., experiments where subjects perform tasks using only vision, or only haptics, or a combination of vision and
haptics, with task performance objectively measured. For example, subjects may be asked to fit a virtual peg into a
close-fitting virtual hole, with position/orientation and forces/torques on the virtual peg or virtual hole measured in
real time as the task progresses. In control experiments, subjects are asked to fit a corresponding real peg into a real
hole while making the same measurements. By contriving to have the experimental setup for the virtual and real
cases to be nearly identical, it is possible to assess the degree of haptic transparency afforded by the haptic
computer interaction system. It is generally found that task performance is enhanced when haptic feedback is
included, but subjects experience more difficulty dealing with virtual environments than they do with real
environments. Results point to the need to improve both haptic devices and haptic rendering algorithms.
How haptic computer interaction can be used
Many people think that haptic computer interaction will have widespread utility in many fields of activity. A
number of application areas are under exploration, but none have as yet entered the mainstream. One such field is
Computer Augmented Design (CAD) where countless products are developed for our daily use. With a
visual/haptic interface, the designer of an engine can pick up and place parts into a complicated assembly while
feeling the fit and friction characteristics. Haptics can be used for education and medical training. Students can
learn physics involving mechanical or electromagnetic forces while actually feeling the forces. Virtual surgery can
be performed, with the student feeling modeled viscoelastic properties of tissues. Multi-dimensional scientific data
can be more easily understood through a visual/haptic interface where the user cannot only see the data, but feel it at
any point. Haptic devices can be used as hand controllers for virtual vehicles. For example, in a flight simulator,
aerodynamic forces and vibration can be transmitted to the user’s hand to provide a more immersive experience.
Many other potential applications are under consideration, including use by persons who are blind or visually
impaired. Finally, haptics can be used to control remote machinery, such as a robot, with forces and torques
reflected back to the operator.
There is growing research activity in haptics through the efforts of device designers, algorithm developers, and
psychologists. As the field evolves, workers in these three seemingly disparate areas are beginning to work together
and to share insights, generating new knowledge in this multidisciplinary endeavor. Meanwhile, many future
applications of haptics are under consideration.
Further Reading
Books on haptics:
Force and touch feedback for virtual reality, Grigore C. Burdea, John Wiley and Sons, Inc., New York, 1996.
Touch in virtual environments, M. L. McLaughlin, J. P. Hespanha, and G. S. Sukhatme, Eds., Pentice Hall IMSC
Press Multimedia Series, New Jersey, 2002.
Haptic senses:
“Physiology of touch, grip, and gait,” K. Seow, chapter in Tactile sensing for robotics and medicine, John Wiley
and Sons, New York, J. G. Webster, Ed., pp. 13-40.
“Surface microgeometry: Tactile perception and neural encoding,” R. H. LaMotte and M. A. Srinivasan, chapter in
Information processing in the somatosensory system, MacMillan Press, D. Franzen and J. Westman, Eds., 1990.
“Four channels mediate the mechanical aspects of touch,” S. J. Bolanowski, G. A. Gescheider, R. T. Verillo, and C.
M. Checkosky, J. Acoustical Society of America, vol. 84, no. 5, 1988, pp. 1680-1694.
Haptic rendering:
“Fast contact force computation for nonpenetrating rigid bodies,” D. Baraff, Computer Graphics, Proc. SIGGRAPH,
July 1994, pp. 23-34.
“ArtDefo, accurate real time deformable objects,” D. L. James and D. K. Pai, Computer Graphics, Proc.
SIGGRAPH, 1999.
Haptic devices:
“The PHANToM haptic interface: a device for probing virtual objects,” Proc. ASME Winter Annual Meeting,
Dynamic Systems and Control, Vol. 55-, DSC, 1994, pp. 295-301.
“Lorentz magnetic levitation for haptic interaction: Device design, performance, and integration with physical
simulations,” P. J. Berkelman and R. L. Hollis, Int’l. J. of Robotics Research, vol. 19, no. 7, July 2000, pp. 644-667.
Psychophysics:
“Sensing and displaying spatially distributed fingertip forces in haptic interfaces for teleoperator and virtual
environment systems,” S. J. Lederman and R. L. Klatzky, Presence, vol. 8, no. 1, 1999, pp. 86-103.
“A psychophysical study of sensory saltation with an open response paradigm,” H. Z. Tan, A. Lim, and R. M.
Traylor, Proc. 9th Int’l. Symp. On Haptic Interfaces for Virtual Environments and Teleoperator Systems, ASME
Dynamic Systems and Control Division, vol. 69, no. 2, 2000, pp. 1109-1115.
Applications of haptics:
“Project GROPE: Haptic displays for scientific visualization,” F. Brooks, Jr., M. Ouh-Young, J. J. Batter, and P.
Kilpatrick, Computer Graphics, vol. 24, no. 4, August 1990, pp. 177-185.
“Real-time surgery simulation with haptic feedback using finite elements,” S. Cotin and H. Delingette, IEEE Int’l.
Conf. On Robotics and Automation. Vol. 4, Lueven, Belgium, May 16-20, 1998, pp. 3739-44.
“Haptic virtual environments for blind people: Exploratory experiments with two devices,” G. Jansson, K.
Billberger, H. Petrie, C. Colwell, D. Kornbrot, J. F. Fänger, H. K. König, A. Hardwick, and S. Furner, The Int’l. J. of
Virtual Reality, vol. 4, no. 1, 1999, pp. 10-20.
Figure Captions
Figure 1:
Section of glabrous skin (schematic) showing discriminative touch sensors. (Adapted from …. , used with
permission.)
Figure 2:
User manipulating a mechanical assembly using a haptic interface device
from Sensable Technologies, Inc. (Photo courtesy of John Hollerbach,
University of Utah.)