1
Desney Tan: Let's go ahead and get started, and I'm sure there will be a
couple more folks rolling in as time ticks down. It's my pleasure to introduce
Ed Colgate and Michael Peshkin from Northwestern University today.
Ed and Michael are in the mechanical engineering department, but Ed also
co-directs the Segal Design Institute with Don Norman out there. They've got
an impressive slate of domains they work across, everything from human machine
interaction to robotics, to telemanipulation and obviously haptics, which he'll
talk about to us today.
They've started numerous spin-off companies, most of which are amazingly
successful. And they're going to talk to us about some of their work on
surface haptics today and then have a live demo, in fact, which is always
exciting. So Ed?
>> Ed Colgate: So thanks so much, Desney. We really appreciate the chance to
be here and to speak. The work we've been doing. So I'll do the speaking for
the talk, but this is very, very much work that Michael and I have collaborated
on here over the past few years as we've collaborated on many things through
the years.
So yeah, the topic today is surface haptics. Yeah, a lot of the work that
we've done through the years is just haptics haptics, and in the early days of
haptics, a lot of it was sort of fundamentally occurring in three dimensions so
devices, probably many of you are familiar with devices like the phantom, a
small robot that you can grab on to, sensors on it that tell a virtual
environment running on some computer, where you are in that virtual
environment. If you run it into things, physics-based models compute the
interaction forces. Those forces are fed back through actuators and that
closed-loop interaction creates a very, very compelling sense of physical
interaction with that virtual world.
But the world has changed a lot, and, you know, touch screen devices like this
one from Apple, but many others, have really changed the landscape of haptics
in recent years, because the touch surface has become such a prevalent
interface that all of us use in our daily lives in multiple instantiations.
And so the challenge has been, you know, down from that three-dimensional
world, into a two-dimensional world, how do you do haptics in a meaningful way
2
on a surface. So I define surface haptics as providing programmable haptic
effects on physical surfaces.
And the real question that's driven a lot of our work has been how do you do
this? I'll give just a very, very brief review of some of the techniques that
people have explored and then really kind of focus on what we've been doing.
Probably one of the most sort of time-honored approaches to this has been
what's called by many E-stim, which the idea of actually driving a certain
amount of current through the tissue in order to cause nerves to fire.
And it's really a pretty interesting idea. After all, the information that's
traveling to your brain about touch is ultimately conducted along nerves, which
are electrical signals. The difficulty of it is that, in fact, in your
fingertip, you've got a whole constellation of different sorts of
mechanoreceptors, four different mechanoreceptors responding to things like
deformation or vibration or stretch. As well as raw nerve endings for things
like, you know, temperature and pain.
And the problem with this approach is that it's not very selective. It's very
difficult to make it select out particular types of mechanoreceptors and
generate particular sorts of responses. So this is a challenging area in terms
of generating more natural interactions.
An approach this is really the leading approach out there in the world today is
vibro-tactile. There's a lot of good reasons why one would use vibro-tactile
for providing haptic feedback. It's a very easy thing to generate from an
engineering standpoint. Certainly, there's a lot of room for optimization.
But fundamentally generating vibrations is a pretty straightforward thing to
do.
But more to the point, we are very well adapted to feel vibration. Sort of a
famous curve, I should have included in the talk that shows that around 250,
300 hertz, you can feel vibration amplitudes of a tenth of a micron. It
doesn't make much to generate a perceptible response.
On the other hand, I've been told by some in sort of the neuroscience community
that part of the reason we're so sensitive to vibration is that if a fly lands
on your arm you, want to be able to swat it really fast, right?
3
So this underlies the point that haptics is really a whole set of sensory
perceptual systems that have different functions, and, you know, a lot of those
functions have to do with knowing when and where on your body you're in contact
with the world, including flies and mosquitos.
And so there are other things that haptics is important for. Like grabbing on
to things, manipulating things, identifying objects by touch. And vibration
isn't necessarily as important a part of those.
What many people would consider kind of the holy grail of surface haptics is
modulating the shape of the surface. You know, so there's been a lot of work
on things like rays, deformable surfaces. Very cool stuff, but obviously very,
very high level of technical challenge. And often, doing this sort of thing
has some sort of very deep incompatibility with other sorts of things that you
might want to do on a surface, like get light through it.
So I'm not going to be talking much about this, although I think the long-term
future, probably, these approaches have a lot of promise.
So what I'll be talking about is the use of shear force. This has really been
our focus. This is probably the earliest example I know in the literature of
using shear force for haptics. This was a device where there was literally a
transparent cover on top of a display, and actuators that would tug on that
cover. And it's sort of very much a conventional type of force feedback haptic
device.
Not necessarily very practical as something that could be
different settings, but sort of a very clever idea. This
Following that, not a lot of work was done on doing shear
because of the difficulty of doing this, generating these
finger. I'll come back to that in a moment.
implemented in many
was from 2004.
force. I suspect
forces on the bare
But there are very deep reasons why shear force is an important thing to do.
One of the very early and best known pieces of work in the field of haptics was
Margaret Minsky's -- Margaret was Marvin's daughter -- development of the
sandpaper system work she did at the M.I.T. media lab in the late '80s and into
the '90s.
She took a force reflecting, a force feedback joystick that can move only in
the plane, but programmed various force effects in the plane and was able to
4
demonstrate that people could perceive textures, if you will, that sort of had
the perception of bumps or ridges that would penetrate out of the plane.
So even though the motion was only in the plane, people would perceive a
three-dimensional interaction.
Then Vincent Hayward and his student, Gabriel Robles De La Torres took this
further in a very carefully controlled study that they published in a nature
paper around 2001. And what they demonstrated was really pretty cool. They
had people literally moving over bumps, not actually feeling the local shape.
So their fingers were on a flat object. But the object was moving up and down
and following that bump contour.
Then they had people moving in a perfectly flat plane. But plane, instead of
forces that were essentially the reaction forces one would feel from traversing
a bump.
Both cases led to the percept of a bump. But then they combined them.
Actually, they took the physical bump and turned it into a hole, but kept
playing the force percept of the bump and the really amazing result is that the
force percept could be strong enough to overwhelm the percept coming from
actual up and down motion. So you'd feel a bump, even though you were
physically moving down through a hole. Sure?
>>:
[inaudible].
>> Ed Colgate:
>>:
This was done also with a kinesthetic force feedback --
[inaudible] flat surfaces up and down?
>> Ed Colgate: No. In this version, what they were doing was very much what's
shown here, a mechanical device that would literally move up and down, but no
force feedback. This was done with force feedback using motors, a closed-loop
system that.
>>:
Only in plane.
>> Ed Colgate:
Only in plane, right, and then they combined the two.
So pretty important in that it shows the existence of a haptic illusion that's
5
really quite powerful, right? That you can perceive shape, three-dimensional
shape based on a pattern of forces in space.
So these are pretty cool and to me very inspiring. But again, they both dealt
with systems where you were grabbing on to or being bound to a device. And the
challenge really is how do you do this on a bare finger? That's the challenge
that Michael and I have been working on over the past few years.
How do we control the shear force vector acting in the plane on a bare finger?
So here's how we approached that problem. We used friction. It would be nice
to think in terms of, you know, what are all the different physics you could
appeal to for pushing around a finger? It would be great if your finger was,
you know, electromagnetic and you could essentially build a linear motor. But
really, what you've got to play with as far as I know is friction.
So I show this classic equation of kinetic friction, the [indiscernible] model
of kinetic friction, which is almost certainly wrong. For instance, your
finger being deformable has a non-linear dependence upon normal force.
But I don't really care about the details right now. What I care about is the
constituent pieces, okay? They are coefficient of friction, normal force and
relative velocity. How slippery is that surface you're contacting? How much
normal force is there with that surface and then what's the relative velocity?
These are the things that we can potentially manipulate in order to control
that force vector. And we've been working on all the pieces so I'm going to
kind of break this down into the pieces and then, you know, we'll show you
demos at the end of essentially all those pieces.
So coefficient of friction was really where we got our start. We kind of
stumbled into this grad student discovering that putting a finger on a
vibrating surface, that surface felt more slippery. It turned out other people
had been seeing that as well.
Back in '95, watching Watanabe and Fukui publish the first paper in this area
in which they used a big old ultrasonic horn vibrator to cause a plate to
vibrate and they put sandpaper on it. And when vibrating at 75 kilohertz, way
into the ultra sonic range, people still felt that sandpaper getting less
rough.
6
And others have worked on this as well. What we've done is take that and build
something we called a TPaD, where we've integrated these ultrasonic out of
plane vibrations with a glass surface and we can take this glass surface and
make it more or less slippery. This is one of our prototype devices.
Just to explain a little built more about how that works, this is supposed to
be a movie, but I think I don't have it set up quite right so I'll just talk
over it.
What we're looking at here is actually a measured profile of the wave form or
the mode shape of a piece of glass that's vibrating at a resonance measured
with a laser Doppler vibrometer. So there's a set of [indiscernible] actuators
glued to glass over here, and we're exciting those things, causing the glass to
bend and then doing that at a resonant frequency that has this particular mode
shape.
So basically, every point on the surface is kind of moving up and down. And
one brings the finger down on it and the surface is basically bouncing up and
down against the finger.
An interesting question is why does that cause friction to go down? And
there's a bunch of papers in the literature, including ours, that claim that
the mechanism is this. I'll show you in a moment that it may not really be.
But it's like this. Imagine that this upper plate represents the finger and
this is the plate moving up and down. There's some air in between. And air
has both compliance and air has viscosity.
And both are non-linear effects. They are non-linearly modulated by the gap
between the upper and lower plates. So that as the lower plate gets close and
squeezes the air, the pressure peaks at above atmospheric so this dashed black
line would represent atmospheric pressure. You get a pressure peak.
But, of course, you're trying to squeeze that air out. On the other hand, the
gap can get quite small so viscous effects can become pronounced, and we're
talking about gaps that are sub-micron here. So really quite small.
As it moves away, you get a larger gap, pressure drops below atmospheric. On
the other hand, the gap gets large so it's fairly easy for air to flow in. And
the net of this is that it acts like a pump and pumps a bit of air in between
the plates and the average pressure goes above atmospheric.
7
And so you're riding on an air bearing. At least, that was sort of the theory.
We've recently been able to take data using the laser Doppler vibrometer that
suggests it's not quite right. The actual data we took is represented down
here in the black. The solid black line is the velocity of the surface. So
this is the surface vibrating at something like approximately 25 kilohertz.
And then the dashed line is the velocity of the finger. And the thing to note
is that it has this very precipitous drop, followed by a little phase where the
velocities are almost matched. And so if you integrate this data, you get data
that looks like the blue up above, displacement data.
The thing is, there's always an offset of integration, and so these two blue
lines, we don't really know where they are vertically, relative to one another.
So what we've done is just slid them up to where we think they ought to be.
Based on what we're seeing here it sure looks like contact occurs, because
they've got this little area where there's this precipitous drop in velocity,
followed by kind of a tracking. So it actually looks like contact is
occurring. And the finger is then essentially bouncing off the surface.
So we've done a bit of modeling and analysis with that and, indeed, there's a
very good reason to think that that would have the effect of reducing the
apparent friction coefficient. In fact, the behavior becomes a bit more like a
viscous relationship as opposed to a [indiscernible] friction relationship, but
with a very low, low viscous coefficient.
>>:
The fingers, how do you measure the position of the finger?
>> Ed Colgate: What we're doing is we've got this piece of glass that's in
here vibrating. We bring the finger up underneath it and then we use like a
silver sharpie to make the finger a bit reflective and look through the glass
right at it.
>>:
It's really just the finger, the pad.
>> Ed Colgate: Yeah, which has been sort of nicely flattened out because of
the fact that it's touching the glass. Yeah.
So I'll say, I may say a bit more about that in just a moment, but those are
the sorts of profiles that we see. So again, apparent contact.
8
We also make friction measurements so we have a little rig where we measure
normal force and shear force, and we can relate the ratio of those two. The
friction coefficient to the amplitude of motion measured, again with that laser
Doppler vibrometer. You see a very marked relationship. It's really a pretty
impressive effect. If you start when things are off with the coefficient of
friction, a mew of about 0.5, it drops by about an order of magnitude over a
reasonable range of motion amplitudes.
Feels very nice as well. It's very nice, slippery, air bearing feeling. You
can also turn it on and off really fast. So here's some data where we've got
the normal force and frictional force both measured with the rig and we
suddenly turned the device on and the apparent coefficient of friction drops
down significantly.
You notice here an even higher coefficient of friction while the device is off.
That varies a lot with both individual and with the preparation of the glass
surface. So here it's close to one. But again, drops down to something quite
small.
Not only that, you can modulate the effect. You don't have to have friction
just on or op. So here we're trying to generate sort of a bumpy feeling on a
surface. You see that, indeed, the friction coefficient is not smoothly
modulated, but certainly not just on/off either. So we can generate some kind
of nice effects that way.
There are problems with this approach. One problem is audible noise. So you
get this surface that's vibrating up and down. We try to operate in the
ultrasonic regime so you can't hear it and so this is frequency response
looking at the motion of the surface itself. It's quite beautiful, all the
energy is here. Had, in this case, 26 kilohertz. Can't hear that, but when we
look through the glass at the finger, we often see things like this where
here's the frequency of the glass and now here's the frequency of the finger.
So what's going on there? Well, if you go back and look in the time domain,
you see things happening like this. So same thing I showed you before. A
little greater amplitude of the glass and suddenly, you see the finger skipping
a beat. So we're getting a sub-harmonic. So now most of the energy of the
vibration of the finger is at half the frequency of the glass. And so it's a
period doubling phenomenon.
9
You see your finger is acting like a little loud speaker generating noise at an
audible frequency. So this is a bit of a problem.
The thing that's kind of interesting here is that you'll see -- I don't know if
I've highlighted that. Let me see. You see period doubling, but you also see
this effect here, where without apparent contact, the motion of the finger kind
of turns around and departs from the glass. Why would it do that? Probably
because there is, in fact, squeeze film as well as contact.
So this is a bit of an issue and it's a difficult problem to solve. It's an
area of ongoing research. Frankly, the most pragmatic solution is simply to
increase the frequency of excitation of glass surface even more. That does
come up in cost and energy. So interesting problem.
So just to summarize, generate some very nice, high quality haptic effects by
making surfaces more slippery. It is difficult to eliminate the sub-harmonics.
We can just operate at higher frequencies, but I suspect that there's more
elegant solutions out there waiting for us.
Okay. So now let me talk about normal force. So that was kind of how we make
things more slippery. How can we modulate the normal force? Well, the main
technique that we and others are using here is electrostatic. And two of the
demos that we brought here today are electrostatic devices.
So those who have been following haptics in recent years have certainly seen
things from the Finnish company Senseg and more recently from Disney research
lab, the Tesla touch device. And these are operating electrostatically. The
idea is pretty straightforward. It's to take a conductive surface, apply a
potential to it, have it covered by some dialectic insulator and they bring the
finger down on the surface. The finger essentially acts as the second plate of
a parallel plate capacitor. Plus and minus charges develop. They attract one
another. You get a normal force.
And then that normal force can be turned on and off as the finger moves in
order to generate the sensation of vibrations.
You know, a lot of people ask, gee, can you really generate very much normal
force this way? And it turns out that in principle, you can. So this is sort
of the classic equation of the force that develops in parallel plate setting,
where A is the area of the finger in contact, B is the applied voltage, D is
10
essentially the gap or the thickness of the dialectic here. The Sepson is the
relative permittivity. And using reasonable values, you can come up with an
estimate of normal forces on the order of up to Newton. And maybe even a bit
greater.
So that's not bad. If you have that much normal force to work with and you
have coefficients of friction that are on the order of half to one, that means
you should be able to vary the lateral forces by up to half a Newton or a
Newton, which is a very significant effect.
But if you actually look at the devices, it ones I showed before, I don't think
you're seeing nearly as much variation in normal force. And a big reason for
that is that your finger leaks. Leaks charge in particular.
So if we look at the finger, it has the properties of a dialectic, but it also
has the properties of a resistive conductive material. So it has finite
resistivity.
So although there's a lot I don't know about the nature of, say, the charge
carriers and so forth, charge does literally leak through the finger. And once
it's through, it's going to end up on the surface. So that of the electric
field acting on those charges is no longer pulling on the finger. It's pulling
on charges that reside on the surface. That means the forces on the finger
decay.
So what's important is the time constant of that decay and so we've made some
measurements that suggest that that time constant can be as short as a hundred
microseconds or less. And that helps explain why even though in principle, you
might get fairly large forces, in practice, if you're varying that force at,
say, 2, 3 hundred hertz, a lot of the time, not much is going on. And so on
average, the forces are lower.
So we've done a fair amount of work which reflects itself in these prototypes
to counteract that effect. Primarily by using an AC technique. Another thing
that I know I didn't put it into the presentation but I know is of interest to
some people here is that these previous devices have required grounding the
finger, and that's sort of a very practical issue. And so you'll find that
we've also done things a little bit differently in order to eliminate that
problem so that no longer is it necessary to do what I'm showing here, which is
directly ground the finger.
11
So we've done these things because our interest is not so much just in
generating the sensation of a texture, vibration. That's a worthwhile goal.
But as I talked about earlier, we're very focused on the shear force, just
lateral forces. So that we can do force feedback. And we'll say more about
that later.
But if you really want to do force feedback, then you really want to be able to
generate significant forces, and for that, you really want to achieve as close
to the sort of theoretical maxima as you can.
This is some data that we've taken that shows that, in fact, we do pretty well.
Here's yet another very different friction coefficient. This is an individual,
with slippery fingers on slippery glass. So when everything's off, he's got a
coefficient of friction in this particular case of about 0.15. We rarely see
things that low, but he must have gotten -- I don't know what was going on that
day.
But when we turn the effect on, you see a significant, in this case about a 50%
increase in the effect of friction coefficient. So that means that the normal
force is being increased by about 50%. So a pretty significant effect, and
we've got lots of data like this. One of the issues that we run into is that
it varies a lot across individuals. But we can pretty much always generate a
significant effect.
So we will show you some demos of this, but the use of electrostatics to
increase normal force has a lot of virtues. It's very fast, turn this effect
on and off nor or less instantly, consumes very little power. I didn't
actually mention the nodal lines, although I showed them in the wave form that
I showed the modal shape of our vibrating surface. And those are areas where
we get less of an effect with the friction reduction and then I mentioned the
noise issues. Those are gone here. So we have a very nice, simple system.
The size of the effect needs to be saved further. I talked a lot about that.
We want to maximize it all that we can so that we can actually use this to
generate big lateral forces on the finger.
And then the other thing is that we're increasing friction rather than
decreasing it so the aesthetic of it is very different, and I would say on the
whole, not quite as pleasing. Maybe not a very big deal the way we intend to
be using it in the future, which is to actually push the finger, not just
12
resist it, and so let me now talk about that.
So this, to my mind, is the really exciting part of all this. So I've talked
about the U and N. But the thing is, friction forces are dissipative forces.
So friction is always acting to resist the relative motion between two
surfaces. And so everything we've talked about so far, the direction of the
force is always opposite your motion.
And that's not satisfactory if we're really trying to do force feedback. We
really need to be able to actively control the force on the finger, control,
have it actually push the finger. Have it push in different directions if I
want to simulate sliding along an edge. While my velocity is parallel to the
edge, but the normal force is primarily perpendicular to the edge. And I
really would like to be able to do something like that.
So I'm left with no choice but to move not just the finger, but the surface as
well. That's the really big idea here is how can we actually move the surface
so that we can turn this resistive effect, this dissipative effect into an
actuator. And so we've developed a few devices that do this. The first was
something we called the ShiverPaD. Which had sort of a complement call the
SwirlPaD and then more recently, something we called the LateralPaD, which we
have here.
The idea is pretty simple. It's so vibrate the surface and then to synchronize
those vibrations with levels of friction. At least that's what we do with the
ShiverPaD, which we're showing here.
So here we're showing how we would generate a force to the right and basically
we make it high friction as it moves to the right and then low friction as it
sort of slips back under your finger. And then the idea is to do that fast
enough that you don't perceive the underlying vibration. You only perceive the
average forcing.
And so with ShiverPaD, we were able to do this up to about 850 hertz, which is
fast enough that you don't really feel it. You have some amount of sensitivity
at 850 hertz, but not much. We were not able to go at a higher frequency
because it takes time to turn on and off friction. The way we were turning on
and off friction was by using these out of plane vibrations. It takes time to
build those up and have them decay.
13
But it does work. I'll show you a little bit of data. These are some of the
early prototypes. This was a TPaD, the device that vibrates out of plane.
Lets us control friction, and then it was hooked up initially just to a speaker
and then eventually to a voice coil in order to cause it so vibrate side to
side.
And this is data with it vibrating only at say 40 hertz but just showing kind
of that in fact, it really does work. We have periods where the TPaD is on and
it's very low friction so your forces here are very close to zero and then
periods where it's off and so the forces are much higher. And so the blue line
is an average force which you see in this case is almost 100 millinewtons,
which is really pretty significant in terms of generating haptic effects.
So the idea, then, is to synchronize this with the knowledge of where your
finger is in order to generate force feedback effects. So I'm going back to
things like the bumps, those virtual bumps from the Hayward and Robles De La
Torre that I showed you earlier, where they were showing that these in-plane
force fields generate the perception of bumps or holes out of the plane and so
that's what we're trying to do here.
This is now actual data. Up here is actual force data. And here it's
integrated. When you integrate it, you get essentially a potential function,
and the shape of that potential function gives you a sense of the percept. So
this is the percept of a bump, the percept of a hole. The way we do it is we
measure the finger's location and then we modulate the direction and amplitude
of the force to generate the corresponding force profile.
So this is kind of a restoring spring in black and the green is an unstable
spring, if you will.
And then you can begin to put those together and do things like this. The
toggle switch is sort of my favorite little effect where we have two-line
sinks, kind of holes, if you will, separated by a line-source, a bump. And
again, actual force data taken here and integrated to give you the potential
function so you see two potential wells separated by a potential hill. And so
the idea is that your finger is going to kind of go pop, pop, pop as if you
were walking into the room and flipping a light switch, which, of course, here
you don't have such old technology. But you know what I mean.
So this is sort of my favorite video that we've taken.
It's hard to do these
14
things to see haptics in a video, but I think you can see it here. You can see
the finger pop, pop, pop, pop back and forth. And really, a pretty compelling
effect.
So to me, this is kind of what it's all about is generating these closed-loop
force feedback experiences. Dramatically different sorts of interactions than
you're capable of doing with just, say, vibro-tactile.
But still, pretty clumsy device vibrating at 850 hertz makes it pretty annoying
noise. We were very interested in doing everything in the ultrasonic frequency
range. And so that led to the LateralPaD, where what we were doing was
synchronizing in-plane and out-of-plane vibrations, both at ultrasonic
frequencies.
So we have the demo here today. Basically, what we've got is a
here, looks kind of the shape of microscope slide. We've got a
pieces of disk glued to it. When we activate that, we cause it
mode shape of bending is shown here. So basically, it's moving
lot in the middle where you put your finger.
piece of glass
couple of
to bend the
up and down a
And then we pull these piezo disks out of these old [indiscernible] spot
removers we can get really cheap and we throw them on the ends here. And so we
have these piezo stacks here and here, and they are reacting against these end
masses and causing the whole glass to move side to side. And again, the mode
shape is shown here. So a lot of side-to-side motion in the middle where you
put your finger.
And then we tune these things to resonate at, essentially, the same frequency
and then we drive them with the same frequency. But we control the phase
relationship between the out-of-plane and in-plane motions. And when you do
that, so you can kind of imagine that the plane, the surface now is not only
going up and down but also side to side, and so it's kind of making elliptical
motions, this way or that way.
And by controlling the phase relationship, we can control the lateral force
quite well and so here's the relative phase between the two. And this is the
lateral force coefficient, which is lateral force divided by normal force.
You can kind of get a sense of the scale of the effect by thinking about this
vis a vis friction. It's not friction, it's active. But with friction
15
coefficients of about 0.15, how sticky is such a surface? Well, that's about
how strong this surface drives. Which is not as strong as we'd like. We'd
like it to be about double this, but I think you'll find that you can feel some
pretty cool effects.
So we do much the same sort of thing. One thing we were very interested in is
what motions of the surface are actually creating the largest forces. And so
we made some measurements using, again, a laser Doppler vibrometer, looking
both straight down at the glass, which gives us the up and down motion, and
then tilting the glass so that we're able to measure some component of the
sideways motion as well.
And then once you run through the math, what you find is that the motion of the
glass generates the largest forces on the finger are when the up and down and
side to side are in phase. Which basically means that the glass is coming up
and hitting the finger in these little pulses like this. Which is exactly the
same thing that that TPaD, the friction reduction device would do. It's moving
straight up and down. But as you move this way, you sort of tilt that angle in
the frame of the finger and so this is sort of tilting it without the finger
moving and so it generates the steady forces on the finger.
We can do the same sort of things, generate virtual bumps. Again, actual data
taken. The forces in this case are up to about 50 millinewtons, so only about
half at strong as that ShiverPaD. So again, we -- very active research is how
do you make this effect stronger.
So to summarize, you know, this is really the key part of it. We've more or
less proven that these coordinated effects work, but there's a fair amount
remaining to be done in terms especially of increasing the force levels.
So those are the big effects. I will shut up now. These are some of the
wonderful students who have really done all this work. We've had some great
collaborations and support from the NSF as well as Department of Transportation
and Ford. And so I think now we can take questions and run demos.
[Applause].
>> Ed Colgate:
So I'm going to move this over.
Well, questions?
>>: [inaudible] are you doing touch screens for automobiles or hand-held
devices or just generic?
16
>> Ed Colgate: Well, I think as you can kind of see from the nature of the
talk, we're more interested in kind of the enabling technology. That's kind of
our strength and so part of the reason that we collaborate with people hoping
to start collaborating with Huang and Karen McLean is someone who looks a lot
at human/computer interface. So we're starting to those things so that we can
begin to sort of get feedback in terms of understanding what the needs of the
applications are.
So I guess the answer is we're somewhat agnostic, but we're trying to learn
more and I guess one of the reasons that we're here today is to try to
understand what vision you may have in terms of haptics and how it can have an
effect.
But I do think that having spent a lot of years of my life studying haptics
that I feel very, very strongly that this paradigm we're doing in force
feedback, whether we've optimized it or not is another question. But this
paradigm we're doing force feedback is immensely more powerful than the
paradigm of vibro-tactile. And therefore, I think very worthy of being
developed for applications and all these arenas.
>> Michael Peshkin: Let me just add to that that for mobile devices, which is
a very attractive market power is, of course, enormous concern. And our piezo
devices are pretty high power devices and they'd have to be reduced by orders
of magnitude to be sort of in the ballpark for mobile devices. But the
electrostatics are closer, better.
>>: Can you speak a little bit about efforts to localize the force feedback to
specific area, because all of these devices seem to function on the whole
plane, right?
>> Ed Colgate: That's a great question. So we have plans for -- we have a
pathway toward localizing it. I'd say the one technology that doesn't
necessarily act over the entire plane is the electrostatic, all right? Things
that involve the mechanical vibration, it's pretty tough to build something
where you can really localize the vibrations without consuming even more
energy.
So but the electrostatic's not a problem there. So that's really the key
enabling technology. So I didn't say it here, but really our path forward is
17
combining the electrostatics with the in-plane vibrations.
Go ahead, Michael.
>> Michael Peshkin: I just wanted to make absolutely clear that when you're
working with one finger on a surface, because you can read the position of the
finger, you can make it as if you had localized the effect. But as soon as
you've got more than one finger or, as you were saying, if you want to conserve
power, it would be really nice to actually localize the -- make it different on
different parts of the plane.
>>: Imagine trying to pull your fingers to the home row. Got eight fingers
down, thumbs on the space bar but in the wrong spot, you could [indiscernible]
each finger to that spot, that would be very, very interesting. But that
sounds very difficult.
>> Michael Peshkin: I wouldn't write that off. I mean, it's a deeper
discussion of how you do that, but I think the key is that the electrostatics
can be localized, and that's probably all that you need if you can get the
vibrations.
>>: So we coming from product development as well. We look for haptics to
mice and key boards, direct and indirect touch devices, all those things, we
looked at a wide variety of tactic devices. But we're generally trying to do
is imitate dome switch clicks. Other sensations of dome require your finger or
don't expect your finger to move laterally. You come down on something,
pushing down. So we've looked, we've had Senseg come in. We've had Tesla
Touch, and the effects are kind of neat when you stroke across, especially
they'll put it into an [indiscernible], stroke across a tile or something. You
can feel it. It's interesting but not really appropriate for our application,
because we're not stroking across anything. Our fingers aren't moving anywhere
in our application.
What's quite interesting about this is it looks like your finger can be in one
spot and feel a lateral force. That gets really quite interesting in ways of
giving queues to your finger that it's in the wrong spot, say. [inaudible].
>>: Which makes it more likely to be applicable for an eyes-free interaction.
That's the thing that's always been funny about ->> Ed Colgate:
For what?
18
>>: What we call eyes-free interaction.
road or your whatever.
Don't want to take your eyes off the
>>: As we get away from more of our -- we get flatter and flatter user
interfaces, with keys that don't move anymore, we want to preserve the sense of
precision when you get your fingers in the right spot. There's nothing more
frustrating than a smart phone with a soft keyboard, period. I've never really
experienced a good one.
>> Ed Colgate: So we do have a demo of a soft keyboard or soft keys, I think
four keys. And again, I'll emphasize that the level of force isn't really
where it needs to be, but despite that, I think you'll find that especially
sort of after just a little learning period, it's quite remarkable, because the
way we do it, we create these potential wells. So not only is it a steady
force, but it's really trying to stabilize your finger at the center of the
Keefe.
I find it personally to be a pretty compelling effect.
>>:
That will be interesting to see.
>> Ed Colgate:
There's a hand up behind you.
>>: Have you thought about, since you're exciting with a coherent source, have
you thought about setting up X and Y standing waves for changing the vibration
amplitude in different spots? That way, you could target it. Like an acoustic
[indiscernible].
>> Ed Colgate: Yeah, I think I understand what you're saying. And to some
extent, and Michael, you may want to chime in here, but to some extent that's
what we do. These vibrations are standing waves. They are resonances, if you
will.
We really try to operate at resonance because it's very efficient, all right.
What some people want us to, encourage us to do is look at ways of exciting
from the corners and focusing energy at certain spots. We have not put the
energy into that. It's interesting, but I suspect -- my visceral sense is it's
knots going to be the thing I want to put my time into, so I think it's going
to be very, very difficult to avoid a significant amount of energy and lower
frequencies and then in general, it's not going to be a very efficient
19
approach.
So rather, what we've done is what I think Michael mentioned earlier, which is
we measure where the finger is, and then we control the effects. The effects
might be happening over the entire surface, but you know where the finger is.
>>:
[indiscernible] use drivers on the corner.
>> Ed Colgate:
For manipulating parts, right?
>>: Well, they have the ability to move two or three parts from the table
simultaneously, but when you play back the video right, I don't think they
quite have the audio.
>> Ed Colgate:
>>:
Oh, no, they're working way down in the audible realm.
The [indiscernible] community do that for self-assembly.
>> Michael Peshkin: Yeah, we have a colleague in Northwestern, Kevin Lynch who
does that sort of work too. You can go online and see videos of what they call
the pea pod, which is super cool, where they vibrate not only in the plane, but
they do some little tilting action as well, and you can create some very, very
nice force fields or velocity fields for parts.
But it's hard for me to imagine those things working in our realm. I think
it's very critical that we get into the ultrasonic regime. Certainly, you must
be above where you get that tactile perception. And you can get some
perception up to kilohertz or so. You got to be above that.
But then once you're there, it's very difficult to not be, you know, coupling
energy acoustically and so getting into the ultrasonic regime seems to be very,
very critical.
The other thing I want to say is our path really is to integrate this motion
and to get rid of these motions. These motions, whether you hear it or not,
are coupling into the air and throwing energy out there and it's just not an
efficient way to do things.
So really, the pathway forward is to combine this lateral motion of this device
with the electrostatic control of normal force from these devices. Okay. And
20
that's the thing that will enable us to, I think in a very power efficient way,
be able to control, also in a very authoritative way control forces on multiple
fingers. That's the focus.
>>: Have you had trouble building structures that resonate at those higher
frequencies? And the reason I ask is we tend to use not in the case of
[indiscernible] which are not as stiff, I think, as the stack.
>>:
So define trouble.
>>:
Can you get there?
>> Michael Peshkin: Yeah, we can get there. It takes modeling and there's a
fair amount of practical issues. But actually, when you look at the demo in
the middle there, you'll see both stack piezos moving this thing laterally and
also disk piezos that are placed at the right spot, estimated computationally
in order to flex the -- the vertical vibrations are done by flexing the surface
and then it goes into wave motion.
>>:
With disks?
>> Michael Peshkin:
>>:
With disks, yes.
So they're glued right up against it so they do the bat ray thing?
>> Michael Peshkin: Not exactly the bat ray thing, but they curve.
to contract. They bend the surface.
They try
>> Ed Colgate: You're contracting it when it's glued to the surface so it's
[indiscernible] bending.
>>: That's what I meant by the bat ray thing.
thing in the middle of the disk.
Because we put a little piston
>>: That would do it too. We're not bending. But we have bent by gluing the
whole thing against the surface. And now, as it contracts.
>> Michael Peshkin: We've done well with it, not only with like fancy piezos,
but just like things that are used as like buzzers, just total junk piezos.
Even those, if we glue them down, we can put the glass into a resonant mode.
21
>> Ed Colgate:
Resonance is essential.
Your static displacements are --
>>: What we've been actually trying to do is to vibrate surfaces in arbitrary
ways is try to avoid resonance because resonance is getting in our way. We
want to make a nice little -- we end up with a ring.
>> Michael Peshkin: But you're vibrating in a frequency range where you want
the person to feel that wave form.
>>: But I guess what we found is that as we're trying to move the resonant
frequency up using piezo disks and fairly heavy glass stocks, we get to three,
four, five hundred hertz and have a really hard time getting it higher. So I'm
just thinking your application is different, but the structures are similar. I
don't know how you manage to get up into the tens of kilohertz resonance when
we can't get past four or five hundred. I'm thinking it's because your piezo
stacks are so much stiffer than our.
>> Michael Peshkin: I think I know the answer. I think you can do it just as
well as we can. The issue is that we're looking for only a micron of motion,
because we're not trying to generate anything that you actually feel.
>>:
Okay.
>> Michael Peshkin: We're trying to modulate this slipperiness factor, and
you're trying to generate things that you feel.
>>:
Right.
>> Michael Peshkin:
>>:
So it's a different goal.
Because the stuff's so heavy.
>> Michael Peshkin: High frequency, of course, is moving in a significant
distance would be an enormous amount of energy so just physically in the glass.
And there's nothing you can do in efficiency that's going to change that.
I just wanted to go back to your earlier comment about sort of clicking and
finding your place. I find that idea of, you know, home keys pretty exciting.
22
But I'd hope that as glass becomes more interesting, you know, as soon as -when you have the ability to maybe individually drive your fingers around, that
this kind of interaction with the glass becomes just one of many, many user
interface pictures. I mean, we see that a little bit even for text entry now
with things like swipe. But I have no idea, I mean, this is what you guys are
really good at is what could be the new ways, what could be the ways of haptic
user interface that you could have once you have the ability to push it around.
>>: Have you thought about experimenting with different textures of glass.
Albeit static. But that close to the display, you could get away with fairly
large RMS modulation on the surface of the glass.
>> Ed Colgate:
>>:
You mean the actual texture of the glass?
Right, the static texture of the glass.
>> Ed Colgate: That's something we would like to do a lot more with. One of
the struggles for us is we're basically a robotics lab that's been slowly
coming up the learning curve on things like, you know, thin films and material
processing. And so we definitely end up at different textures of glass, not
necessarily by intention. So I think you'll -- everything here today is pretty
smooth, which isn't necessarily what we really want. When it's smooth, it
tends to promote stick/slip phenomena, which affects the mechanics of the
interaction. There's a lot of work to be done there.
>> Michael Peshkin: Not only can you vary the surface texture and rough
surfaces tend to be lower friction than very smooth ones. But there are also
oleophobic glasses and there's all sorts of amazing stuff that these people can
do on the surface of glass that are mysterious to us.
>>: Let's do one last one.
this thing.
I think we're all itching to get our fingers on
>>: What's your theory on, say, electro active polymers, which some people
think is the clue to future haptics, once the dynamic range is increased.
>> Michael Peshkin: I think they'd certainly have a role. But the way I parse
up the world isn't so much whether you're using EAPs or piezos or
electrostatics in order to do your actuation. The way I parse the world is
what are you doing with your actuation. And I guess my message here today is
23
don't put your stock in vibro-tactile, okay? Put your stock in force. Force
feedback. Because I think that from the standpoint of haptics and what we do
with it in the world, the sorts of effects we want to generate are ones that
are based upon closing the loop with force feedback. So if you can use EAPs as
part of a force feedback system, which I think you can, then absolutely.
But so that's just sort of my parsing.
[Applause]
All right, we want to show some demos.