>>Meredith Ringel Morris: Thank you all for coming. This is a great turnout. I'm glad you're all
as excited as I am about Shaun's talk. So we are pleased to welcome Shaun Kane back to
Microsoft research. Shaun was a local for many years. He got his PhD in the iSchool at the
University of Washington, and he also did several internships at Microsoft during his graduate
career, and now he is a faculty member at the University of Maryland, in Baltimore County in
their human centered computing program; and his research focuses on accessible technologies,
in particular on accessible mobile technologies and accessible touch oriented technologies.
And so we are really excited to have Shaun present on some of his work while he is here in
town to work on a joint research project. So I'll let the Shaun began.
>> Shaun Kane: Thanks Mary. So as you were introducing me, I flipped for a second to my
second slide which had my, actually my tours of duty at Microsoft. I was an undergrad intern at
Microsoft too, way back in the 70s. But, yeah. So hi, I know many of you. Thanks for coming
one and all. I'm going to talk a little bit today about the work I've been doing starting as a PhD
student and leading into [inaudible] as a faculty and working with students, primarily in the
area of accessible interaction and primarily beginning with gestures, but looking at ways that
we can also just think about how to design what I would consider to be next generation
accessible and assistive technologies. So I'm going to tell you about a few projects in that space
and show you some stuff we’ve been working on.
So Mary gave a great introduction, so I won't spend any time on this one. But I wanted to start
by just beginning with some thoughts about the kind of work that we do in accessibility. So I
know many of you here are students in an accessibility seminar and thinking about how do we
do accessibility research? How do we build accessible technologies? And two things that have
come to mind and I've been thinking about in my work for the last few years, so first is that a lot
of the work that we do in accessibility tends to be reactive. And so somewhere off in
mainstream world we developed some technology and that becomes extremely popular. So we
look at smart phones, or tablets or other kinds of contemporary technologies. And then we
tend to bolt on accessibility technology as a kind of afterthought or as a reaction. So this image
here is of a key guard. So it takes a QWERTY keyboard and provides a kind of tactile overlay to
the keyboard so that the user can actually more accurately type. So if someone had, for
example, a motor disability or some kind of tremor, then this add-on might actually help them
input, type on the keyboard. But this is really very much an after the fact add-on to existing
technology. The other piece we see that even when we look at new accessible technology and
new technologies, a lot of times this technology isn't designed with the user and their abilities
in mind or their natural means of interaction. So in the past few years we've seen quite a lot of
projects like this. This is a guy with a helmet with a connect on his head and, you know, this is
it. It's an interesting idea, right? When I think about how do we leverage sensing technologies
when we think about new, again maintain technologies, and leverage them for accessibility.
But we want to do that in a way that is in accordance with what people are naturally doing and
what people are naturally good at doing.
And so I want to talk about a few projects today that I’ve been working on for the past several
years and looking at just a few key themes in this work. So the first is really leveraging this idea
of natural interaction to make assistive technology easier to use. And as much as we can,
leverage gesture, leverage speech, leverage, natural ways of interacting. Adapting, building and
adapting interfaces based on a rigorous analysis of users’ abilities so, and looking at, for
example, touch. Well, what are an individual’s capabilities to use a touch interface or to
interact with an audio interface, and how can we design technologies to work better for them?
I'm going to be talking about some work looking at context aware accessibilities, so as our
devices get more, gain more awareness of the world and of ourselves and what we’re doing,
how can we leverage that? And also, a lot of the work I've been doing lately has really been
looking at this intersection of hardware and software. So the first phase of my career, and for
those of you who know my early work, really looking at how do we take existing devices and
change the underlying software to make them more accessible? A lot of the work that we've
been doing recently has been looking at this intersection of hardware and software.
So briefly, as Mary said, I'm a faculty at UMBC, and a lot of the work, especially the more recent
work, has been coming out of our research lab, which is the prototyping and design lab, the
PadLab, and mostly I bring this up to say that we are recruiting PhD students. The deadline is
February 1, if you're at the last minute and you haven't submitted your, all your grad-aps, you
should do so and go check out our website. But in general, in this area, we've been working
and looking at hardware and software to help solve accessibility challenges, looking with a few
populations, so blind and low vision individuals, which I'll be talking about today, work with
older adults, work for people of cognitive and related impairments. I'll be talking a little bit
about that, too. So I want to show a few projects looking at how we can take this notion of
assistive technology, so thinking about technology to help individuals be more independent,
and look at the way we interact with it and hopefully make it more interactive, make your
interactions with it more natural and hopefully more fun and enjoyable. And I'll be talking in
two areas, so some background on accessible touch interfaces, and also some new work that I
started about a year ago looking at communications tools and how do we make
communications tools more engaging, more usable.
Okay. So I'll start with just a brief introduction of some of my past work on accessible touch
screens. Many of you have seen some of these slides probably three or four or five times. I
know Richard’s probably seen them more than that. Yes. Well, we’ve got to keep it fresh. I'm
going to talk about some new work in this area and building off of this and looking at how we
can use accessible gestures in a new context, which I'll talk about this project, Access Lens, and
then I'll talk about two tools that we’re working on looking at communications tools for people
who have communications disabilities. So these slides you probably have seen, probably a
dozen times. But to start with, so I've been working in the area of accessible gestures and
accessible touch interaction since about 2008. So for quite a while now. And as time goes on,
this slide becomes less and less necessary as people understand that the technologies that we
use are really, ideally can be used by anyone. But we still live in a world where a lot of the
everyday technology that we use, including touchscreens, but also many other kinds of
technology, are still inaccessible to large groups of people. But we know now the touchscreens
and touch interactions are everywhere, so not just in our phones and our tablets, but now in
PCs, when we're out shopping, when we are voting, and also in home technologies. So, for
example, you might see this display here on the microwave, it actually isn’t a touchscreen, but
it uses touch sensitive technology, and so a lot of the challenges that an individual might have
with a touchscreen also might apply here.
So we have, it’s safe to say now that touch interaction is ubiquitous, and it’s going to stick
around for a while, and so this is an important problem, and as our touch interfaces evolve, we
also need to consider accessibility. Just pointing out here again that it’s not just a touch screens
but also touch panels and that we can see them in a variety of technologies, so office
equipment, home equipment, so really as touch interfaces become more ubiquitous, they have
a greater impact on those who cannot use them or have difficulty using them. It's also the case
that when we designed for accessibility we help lots of people. So not just someone who walks
with a white cane or has a guide dog, but people with a range of visual abilities, so about 25
million people in the US identifies having some kind of visual impairments, but an even greater
group than that may benefit from visual accessibility tools. So a study from Microsoft in 2004
said about 25 percent of computer users would benefit from technologies that helped with
reading and with visual information, making visual information more accessible. And that's,
whether or not they identify as blind or visually impaired, and of course, in looking at touch, we
know that touch is actually a very effective interaction technique for blind people. So we think
about braille, so although braille literacy is low, those who are fluent in it are, can read quite
quickly and quite ably, but also in thinking about natural interactions, we do use our hands,
everyone uses their hands to explore their environment, and for someone who is not relying on
the visual channel as much, this kind of feedback can be especially useful.
So, I'm going to talk about just briefly about some of the past work and then I want to talk
about a new project that I've been working on this space. So these projects that I'm talking
about here are part of my dissertation work, so looking at trying to understand how touch and
how mobile technologies could fit into the lives of people with visual impairments, started
looking at just getting a qualitative understanding of how these technologies are used, so I did
some work with Richard Ladner, among others, looking at how individuals who are blind and
visually impaired use mobile technologies and use touch technologies in their everyday lives.
So, did they have touch devices? Where did they encounter them? What did they do? This
work is also more design oriented work that I'm going to talk about today is also a more
performance study, so looking at how a blind or visually impaired person would perform
gestures. So we have a paper from CHI 2011 looking at gesture performance across sighted
individuals versus blind or visually impaired individuals.
Okay. So I want to talk a little bit about some projects. I'm going to zoom through some early
work in this space and talk about some new work which is actually going to be presented at CHI
this year, so you get a little bit of a preview on this project. So I want to talk about a few
projects looking in general at accessibility and touch and how we can design accessible touch
interactions across a range of devices. As our devices change, so do our interactions with them.
So this work began, actually now it’s a paper that's getting older, this work with Jacob O
Wobbrock and Jeff Bigham, called Slide Rule, and this was really one of the first one, of our first
forays into making gestures and touch accessible. So we looked at touch screen smart phones,
as they were quickly becoming popular, and looked at again how we could take an existing
touch device and by changing the software and by changing the interaction modes, could we
make that accessible to someone who could not see the device? And so we built a prototype
and we evaluated it. And I’ll just tell you little bit about how that works to give you a sense of
the big idea here and how we are designing accessible touch and some of the insights that have
come from it.
So this began with interviews, with some observations, looking at blind mobile device users,
who at the time were not using touchscreen devices, and a series of iterative prototyping, really
looking at how we could take, in this touch screen phone, and specifically without changing the
hardware, and even not even using the existing buttons on the device. So we said, our input
workspace, the input that we are working with is a flat surface, how can we make that
accessible? We used the iPhone at the time. So if we think about interacting with a touch
screen device without visual feedback, we start to think about, we have a flat screen and we’re
getting very little feedback as it is. And it's relatively featureless; we're not getting any really
haptic feedback. And so when we think about how to make gestures accessible, how to make
this kind of input device accessible, first we have to, when I think about an appropriate output,
we say, okay, well we could provide haptic feedback, we could provide vibrational feedback,
but for many devices, including a lot of mini smart phones, the kind of haptic feedback we can
get from existing devices is quite limited. We might rely on speech and audio, and so, okay,
well we have a visual interface and there's been a lot of prior work on the desktop and other
computer interfaces of transforming a visual interface into an audio interface; so we’re going to
design a screen meter for our phone.
So we take our touch interface and we just make things start talking and identifying
themselves. But this raises the question of well, how do we provide input and how do we
navigate through this space? A lot of the traditional interfaces we use for our screen reader are
keyboard based and have an idea of a cursor that we’re moving through a structured user
interface. How does that change what we look at touch and gesture? So answering questions
about well, how do users touch the device? How do we lay out the screen? And how do we
have a set of usable and reliable gestures that someone who's blind is going to be able to use
and use comfortably and confidently? And so just to give you a sense of how we first took this
challenge on, we can think about our traditional smart phone interface, we have a grid of icons
and we interact with them by looking at them, identifying where we want to interact, and
touching them. We can say, okay, well let's make this accessible to someone who's blind by
taking everything that's visual and representing it with audio. But we ran into some problems,
right? So that user interface is really designed spatially, so for easy visual use and visual
scanning, but if we turn this into audio, well how do we find stuff and how do we lay things out
on the screen? We also might have this conflict here where whitespace is often good for visual
interfaces and for audio interfaces; it can be bad because it’s essentially a dead zone. We're
not getting the feedback that we want. And in talking with blind individuals about how they’d
use a touchscreen, they were really afraid about doing the wrong thing. This was the theme
that came up most in our early discussion was they didn't want to accidentally call someone
that they are not on speaking terms with or accidentally order something or accidentally delete
something, or all the other are the kinds of things that happen when you're not getting good
feedback and you're not in control of your device.
And so we wanted to take these challenges on and made a few adjustments to the interaction
with this device to make the device more accessible. And so first was relaying out the visual
layout of the interface. So rather than having this nice visual grid, which is easy for visual
scanning, we took the entire touch surface and blew up all the interactive elements on it so that
they were, first of all, easier to hit, but also switching from a 2-D interface to a 1D interface. So
now if we’re trying to find something on the screen, we are not searching every pixel on the
screen with our finger trying to find what we're looking for, but we can actually search in a
confident, controlled way. So here we have, rather than this 2-D search, we can just scan from
the top of the device to the bottom and hear everything that's there. The other thing we have
here now is we have a type of preview interaction, so rather than things happening when you
touch the screen, in this work, we get feedback when we touch the screen, and when we want
to select something or take some action, we perform an alternative gesture. And so the way
we implemented this for this slide rule project was with a multi-touch tap. So had a multitouch touchscreen, we can feel on the screen for the options we want to select and when we
find that one that we want, we can then indicate a selection, and the way we do that is actually
keep our finger where we want to select, so we don't have to worry about losing that when we
lift our hand up, losing that context, and just touch on the screen with a second finger. A multitouch.
And so this is the basis for a lot of the work that came after that. Both our work and also it's
had influence outside of that. And really, the fundamental design considerations here were
taking this touchscreen device and enabling risk-free exploration. So I can interact with the
device in a way that I know is going to be safe, nothing is going to happen and providing safe,
reliable, usable techniques when I do want to interact with the device.
So I'm going to show some more videos later on, so I’m actually going to skip the video of this,
but it's on my webpage. Just to give you a sense of where we were, this paper is now, getting
older now, so it's, if it were a child, it would be up and walking and talking and saying things and
almost ready to go to school. So if we look at mobile devices in 2007, 2008, it was quite
different, landscape was quite different. Touchscreen devices were more rare, and so when we
tested out this technology we actually compared it to devices that are, blind participants were
using at the time, which were mostly, used physical buttons to control. So you have a four-way
control or additional buttons for interaction with the screen. And so we tested this more
gesture focused interaction and compared it to the button focused interaction with some
applications that we developed on both platforms and just looked at how fast could tasks to be
completed and how accurately.
And just to sum up very quickly, so seven of our participants preferred this gesture interface to
the interface that they were more familiar with, the more traditional button interface. So they
liked using gestures, and they were actually faster at performing the gesture interface than the
button-based interface, and there's a few reasons why that is. But one is, just in general, that it
was a richer input space, whereas with a button interface, if you want to get to the 20th item
on the list, you have to hit button, button, button, button, button, button, 20 times, with
gestures you might be able to navigate more quickly. But we did see more errors. Actually,
participants didn't make any errors with their button interfaces because they knew them well,
and so this is an issue that we are still wrestling with and still trying to understand. But, in
general, we found that taking this gesture-based device, which was really designed around
visual interaction, we could adapt it to something that was more usable for someone who
couldn't see.
So we followed up in this work in a few different areas. But also, the idea of using these kinds
of accessible gestures and this nonvisual output, but gestural input way of interaction is actually
quite popular. So, estimates have been as high as 100,000 users using accessibility features on
touch screen phones. That number might be a little high; it's hard to get a good sense. If
anyone knows, they're not actually telling in great detail. But now we see these types of
accessible gesture interfaces on a wide variety of mobile devices, so Apple devices, Google
devices, hopefully Microsoft devices.
And so this work really started off a lot of my dissertation work and looking in general at, well,
if we have accessible gestures, if we can make gestures accessible, what we actually do with
them? And so in this work I looked at what were the limitations of this initial approach, and
how could we extend it to work in other places? And so I'm going to very briefly describe some
early work I actually conducted here at MSR with Mary Morris and Daniel Wigdor, and Annuska
Perkins from the accessibility group, and Richard Ladner and Jacob O Wobbrock, my PhD
advisors. And so I'll just tell you a little bit about how, starting from this core and some base
principles about accessible gestures, how we could extend this to a different device.
So in this work, we were looking primarily at large touchscreen, so very large tablets, but also
tabletop computers and other large interactive displays, like Microsoft Surface, or PixelSense,
and the thinking about the kinds of interfaces and the kinds of interactions that we see on this
device versus a smart phone. So when we look at our smart phones, we see a lot of interfaces
are quite simplified, they're very basic, but as we scale to larger interfaces, we start to see more
complex spatial layouts, we start to see more complex information. We also see, in looking at
these applications, that location and spatial layout can also often be quite important. So if you
think about interacting with maps, or sorting documents on a touchscreen, or playing games,
it's not just important to know what's on the screen, but we also really want to know where it
is. And a lot of the techniques at the time, focusing on smaller devices, including our own Slide
Rule project, but also VoiceOver, which was the Apple Screen Reader, which was, as of now, the
most popular screen reader for touch screen mobile devices, really had made a lot of benefit
from taking spatial interface and turning it into something that could be scanned linearly, just
like I showed you with Slide Rule. But we lose then, the spatial information that helped, that
might be quite necessary for some tasks that we are performing.
And so in looking at how we can take, how we can make an accessible touchscreen where we
maintain some of this accessible information, we first started by going to people's offices and
hanging out and looking at how they organized their space. So I spent time, this is an internship
project, started as an internship project, going around to the offices of blind office workers, and
understanding, talking with them about how they organized their space, how they strategized
where they put new things, what they did if they lost something, and so forth, to try to get a
sense of what are some of the, some of their concerns were, and also strategies about
organizing their space. And our paper, which was in [inaudible] 2011, tells us a bit more about
some of the strategies that they're used to. But I'll talk about some of the strategies that we
developed and built into a project which we called Access Overlays.
So the idea here is that we have a large touchscreen application, so here we've got our world
map app on the Microsoft Surface, and we want to maintain the original app, we don't want to
write a new app and we don't want to drastically change the interface. But what we can do
here is we can take this interface and add, provide additional support that's going to help a
blind or visually impaired person to negotiate this space. So we tried a few different interactive
strategies to make it easier to search this space, to find things and also to understand the
relationships between different objects on the screen. And I'll mention these just briefly; these
will actually come up again in the next part of the talk. So first was to take all the items on the
screen and project them to the borders of the screen. So here we have our map and then we
have corresponding icons for each of the map items along the edges. So now we have, in some
sense, the best of both worlds, so we can explore in 2-D by putting our hands on the map and
feeling around, but we can also scan by just touching and sliding along the edge of the table.
We also looked at ways of dealing with dead space. So instead of having a set of discrete icons
on screen and then a lot of dead space between them, how can we fill that space? We could
use voronoi tessellation, in this case, similar to the Bubble Cursor, although while the Bubble
Cursor helps you select targets, here, with Access Overlays, we’re trying to make it easier to
find targets and understand the relationships. So we can provide this overlay onto an existing
app and provide a mode where the user can explore and understand spatial relationships
between objects on the screen.
Finally, we looked at the combination of touch gestures and voice commands to explore. So,
can we use voice commands to query at interface on a touchscreen and combine that with
touch, so we can ask questions like, what’s near here? Or where is a specific location onscreen
relative to where I'm touching? So to evaluate these techniques, we had a few that we liked.
We tested them out with blind computer users, and we were particularly interested in, not just
traditional metrics like task completion time and errors, but also spatial understanding. So we
had participants perform tasks where they explored a map and then we asked them questions
about the map. So was this city to the East or the West of the other? Or how close were they?
To help see if these techniques actually helped increase the understanding of the original
layout. And again, to go through here somewhat quickly, the techniques that we developed, in
particular, the voice space technique and the edge space production technique, were faster
than both the browsing technique, the tessellation technique, and also VoiceOver, which is our
comparison as the gold standard. And also, all three of our techniques improved spatial
understanding. So participants answered more questions correctly when they used these
techniques versus when they used Voiceover as a non-spatially oriented technique. So it's
surprising that this is the case, but what’s interesting here is that our technique did seem to
help this problem.
And in general, of the techniques that we tried, participants enjoyed both the speech-based
interaction and the edge browsing about equally, so they were both quite popular, although
the voice interaction was slightly more popular with a lot of caveats. They said oh, I really like
this one, but I wouldn't use it here and I wouldn’t use it here and I wouldn't use it if someone
else is around. But in general, these techniques seem to be enjoyed, and one thing it was
interesting was there was no clear winner here, and a lot of participants in the study actually
said that they wanted to be able to use all of these techniques and kind of apply them. I'm
thinking about how we make future screen readers for our touchscreen devices, whether it's
our tablets or our large touchscreens, this idea that we might have different kind of modes that
we can apply seems promising. We explore this to some degree here and I think there's more
interesting work to be done here.
So now I'm going to talk about the last piece related to accessible gestures that I would talk
about today. And so looking at the limitations of this project to date, well, we looked at small
devices, and we said, okay, well we seem to understand small devices. We looked at larger
devices and made some progress here, but with all the work that we've done with accessible
gestures, we are still limited by the problem that a lot of these, a lot of the devices out in the
world are still not accessible. And so these accessible gestures seem to be quite useful, people
like them, but they are limited to a small set of devices. And so I was really interested in
answering the question of well, if we have these accessible gestures, and we have a set that
people seem to like, what else can we do with them? Where else can we use them? And so I’ll
talk about that, the last project here in this space, which is a project called Access Lens. And so
this will be presented at CHI this year, I hope to see all there. It's in Paris.
And so this is work that started at the tail hand of my dissertation and has been developing
with the help of some a student, Brian Fry, and my colleagues. And what we’re looking at here
is how can we take this notion of accessible gestures and use it in a different place? And
starting with the real goal of if a person who is blind or visually impaired walked up to a touch
screen or some other device that was not accessible, could they interact with it in a way that
they're familiar with using the techniques that they know? So could we take this ATM and
could we apply the kinds of gestural overlays that I've shown you to it so that the person can
explore it using gestures. And we’ll come back later to the technical how's and why's and what
we can and we can’t do, but just thinking in terms of interaction, of how could we take
advantage of what we’ve shown so far and apply it to the real world? And so this is a project
that, again is in the space of hardware and software, so sort of really thinking about cameras,
about mobile cameras, and in particular, a wearable camera or a camera in your workspace
that could look at what you're doing and could identify items in the world, identify text on them
or other information, and then actually enable the user to use his or her hands to explore that
space and to get audio feedback about, so speech feedback, other kinds of feedback about
their environment.
So there are a few [inaudible] cases that were very excited about here. One is to access
touchscreens that provide by themselves little or no accessibility features. So these still exist.
And even on devices that have good screen reader support, like some of the more popular
tablets, a lot of the applications still aren't designed in an accessible way because the designers
don't follow their correct guidelines or they make some mistake and that information is
inaccessible. But I was also interesting looking at how we could use gestures to explore analog
data. So thinking about reading your mail, thinking about reading paper maps, thinking about
reading other kinds of documents, thinking about reading appliances in the environment. And
so this is a prototype that I built, that we built and we've been evaluating versions of this for
the past year with blind individuals. And so you can see here, this is one of our prototypes, and
it's an IKEA lamp base with the lamp cut off any camera mounted on it so it clips onto the
workspace. So the idea is that you can have it in your workspace and push it out of the way
when you're not using it, it's a lot like a kind of magnifier that a low vision user might use in
terms of, it’s something on your desk that you could use to scan something or read something,
but it provides some additional features in that we can actually use gestures to interact with
data.
So I'll show you a video in just a minute, but I want to say just a little bit first of how it worked.
First of all, this built on some prior work that I had done, a system called Bonfire, which was
looking at gestural interaction on every day surfaces. And so mostly I include this just to tell
you kind of how it came about and why the design is what it was and so we began really looking
at, if we had a laptop that was aware of the space around it, what kinds of interactions can we
do? In the case of Bonfire, we combined this with output, via pico-projector, but in general, we
were interested in looking at how a computer especially a mobile computer, could be made
aware of the environment and enable interactions with the real world.
And so just briefly, basically, we are using techniques adapted from Bonfire to scan the
environment, to identify objects in the environment, and to enable gesture interactions with
those objects. There's more detail about this in the paper, and I'm also happy to say more later
on, but in general, the idea is that we can take a document or some object, put it in our
workspace, and the software can identify the new object, can correct for perspective or skew if
we have an object that is flat and we know at least something about it, so we know things that
look approximately the size and shape of an eight and a half by eleven piece of paper. We can
then use OCR and text identification techniques to identify text on an object, and so this is the
point where, if we had a scanner, we can say, okay, we can do this with our scanner and scan
the text. But what we're doing here is not only scanning text, but identifying where it is in the
document so we can explore it. We do some OCR, we correct some identifications, or at least
attempted to, and then we can use the same camera that we’re using to capture documents to
actually track gestures. So we can track hands in the workspace, look at where they're pointing,
and give feedback based on what we know previously about the document, and now where the
user’s hand is to give them feedback.
So, in thinking about how this might actually be used in real life, it's sometimes useful, as we've
discussed before, to explore spatial documents and to get spatial understanding. So whether
it's exploring a map, whether it's finding just the one part on your electricity bill that you want
to find when you know where it is, so you can find that exactly, or other kinds of spatial
documents, being able to explore the document using gestures seems to be useful. But it can
be quite time-consuming to the space and so we are actually using the Access Overlays feature
that we developed previously, to help explore paper documents more efficiently. So I will show
you a quick demo of this.
[video] Document detected. Starting OCR. Recognized Lake City.
>> Shaun Kane: So we’re scanning our map of Seattle neighborhoods and capturing text.
[video] Recognized University District. Recognized Queen Anne. Recognized Capitol Hill.
Recognized downtown. Recognized Reindeer Valley. Recognized left Seattle.
>> Shaun Kane: And now we can track, so tracking a hand using computer vision techniques,
we can start to learn feedback about where objects are on the page as I touch it. And then one
thing we can do here is to make the document easier to scan, we actually had a virtual index.
So it’s on the side of the physical page, right? There's nothing actually there, but the user can
scan along the edge of that page and get an ordered index of items and then get, as I'll show
you in just a second, feedback toward that goal. Oh, I broke it.
[video] Hold here for directions.
>> Shaun Kane: So we can actually track the finger in real-time and give them guidance
towards the [inaudible]. So we can also uses other devices, this is a mockup of an iPad app that
looks like an ATM screen, and again, we can use Access Lens to explore the screen, although we
have to do this a little bit differently, as you might imagine, in that here we’re not actually
interfacing with the device at all, but we are just getting feedback about what's on the screen
and what we are pointing towards. So, as I said, our preliminary work in this area will be
published at Chi this year, but we are interested in exploring other applications as well. So
probably the next step is to really look at mobile and how to make this a more mobile interface,
so interacting with more documents out in the real world.
And this works fairly robustly in a workspace with, where we know the lighting and there are
not a lot of environmental changes, but turning this into something that you can use
everywhere is the next big challenge and looking at, also looking at the form factor for
hardware that we want. So do we actually want something like the Google project [inaudible],
do we want a wearable camera like I have in this demo? Do we want to hold a phone and wave
that around? It's not quite clear. And so we're also interested, and looking at other way to
bring in the environment to make this work better, so bringing context to where OCR, so if I’m
at the ATM, maybe prioritizing terms that are related to my transaction there. And also how
can we use this interface to recognize other spatial data? So we actually developed a module
for exploring colors so that you can touch an image and it will tell you what colors are under
your fingertips as you’re feeling around. This is something for one of our participants who was
really interested in exploring art. But thinking about how we could use this kind of platform for
recognizing objects and then enabling gestural exploration of those objects to explore different
kinds of information. Yeah, Sherry.
>>: Can you talk a little bit about the [inaudible] between this and [inaudible] which at least
tries to give me feedback in real-time also [inaudible]?
>> Shaun Kane: So the KNFB reader is addressing similar problems, but through different form
factors. So one of the real challenges with a lot of the smart phone based solutions is, where
am I aiming the camera? What is a camera looking at? So this is a challenge for anything where
we're trying to explore something and the question, and the question is, what is the best way
to explore? Is it moving a camera around or is it using gesture? We were particularly
interested in exploring this opportunity because of all the work we've built up in gestures and
the kinds of shortcuts we've learned. So looking at using Access Overlays, for example, as a way
of exploring spatial data. I think there are, at least in its current form, there are definitely
trade-offs in terms of, it’s not going to work anywhere, you have to be able to put your hands
on something, and I think there's still a question about when we are moving, where the
camera’s going to be and what is going to see? So I think that's something we're going to keep
looking at.
Okay. So, actually I will skip our ongoing work and touch interaction. There's quite a lot of it.
Again, if you're interested in collaborating and doing research, so looking at things like 3-D
printed techno-graphics, we just started a project with NASA to print 3-D versions of star
clusters for blind kids, working with NFD in NASA, really excited about that, looking at
accessibility in collaborations, so this is a problem I'm really interested in, looking at how we
take these interfaces that are primarily audio-based and bring them back together with visual
interfaces so blind and sighted people can collaborate together, looking at how we can make
accessible hardware and so forth. So this is still a very active area, but I wanted to end this talk
was just a little bit of some of our newer work and looking at a different population, just to get
a sense of kind of how some of these techniques might translate to different group. So I'm
going to talk just for a few minutes about this project.
So this is a project that we presented at ASSETS this year. Note the slide template change. And
so basically we're looking at communication tools for adults with aphasia or other kind of
linguistic disabilities and developing tools that could use context awareness and adaptive
interfaces to make these tools more accessible. So to give you the very high level view, aphasia
is a neurological disorder, it affects an individual's ability to understand or produce language,
written or spoken. So it can be any of those, any of that set. And it's most often caused by
stroke, so some damage to the brain, and it's noteworthy that typically aphasia is something
that affects communications ability but does not affect cognitive ability otherwise. So these are
necessarily people who are cognitively impaired, but they have communications difficulties.
The most well-known person with aphasia that you guys probably know is Senator Gabrielle
Giffords, Representative Gabby Giffords, who had a head injury and now has language
difficulties. But aphasia is a very broad term; it can describe a wide variety of abilities.
And so for those individuals who do use, do have this condition, it can be quite challenging to
communicate with anyone in any format. So speaking tends to be quite difficult, so summoning
words, think about having a word on the tip of your tongue, but that's half the words in your
vocabulary or more. So it can be quite serious, can be quite a stressful situation. And people
who have aphasia typically still have some motor ability to speak. So it's not like Stephen
Hawking or someone who has full language capability but doesn't have the motor ability to
speak, this is someone who typically has the ability to speak physically, but lacks the language
skills. And so they use a wide variety of tools, which we refer to as, or more broadly as
augmentative and alternative communication technology. This term also describes a huge
variety of technologies for a very large population. Here we're particularly looking at adults
with aphasia, and so when we look at the tools that they used, typically we see some
combination of typing based tools, so they can type out words, often with some assistance,
often quite slowly, and then often will play a sound, so this app, for example, they can type a
word, if they remember how to spell it, or approximately, they can say, speak it, and the
computer will speak it, and once they hear it, they can actually say the word themselves. But
without that audio feedback they may not necessarily do that.
They also often use these kind of menu-based tools. So you see these also for children with
autism, adults with autism, people with more cognitive impairments, where you have some
kind of ontological model or hierarchy of types of words in terms that the user can browse
through and tap and get feedback, either an image or a spoken word as a prompt that can help
them recall the word. So these are common tools, but not necessarily popular. The adults that
we are working with, with aphasia, often really hated these tools. But they relied on them
because otherwise their communication ability was quite limited. And so in looking at how
they're using AAC tools, we saw a few challenges. So first is, we have this hierarchal navigation.
So if I want to say coffee, I might say food and drinks, go into the drinks category, and then go
to coffee. Well, if you have difficulty recognizing words, going through a hierarchal menu is just
repeated torture. So it can be a reading challenge, but also a potential motor challenge, many
participants have stroke as well. And we found that many of the people with aphasia that we
worked with really had difficulty keeping items organized. So even when they used these tools,
if they added their own terms, they tended to keep them in one big list rather than separating
them out. You can probably, it's probably not surprising to know that. Making categories is
actually quite difficult. And so, while they had these tools, there were some challenges to using
them and often they ignored the tools.
And so the work that we began with this project was really looking at how can we use more
contextual data to improve the usability of these kinds of tools. And leveraging our knowledge
of our participants, of kind of where they go and who they know. So if you think about your
own conversation in everyday life, maybe you talk about a certain topic with your coworkers,
you talk about sports with your coworkers, but when you talk with your parents, you typically
talk about opera and all these patterns might change over time, or there might be current
events that are very exciting. These tend to be somewhat stable over time. You probably are
going to talk about floss at a dentist’s office more than you do elsewhere, or something. And so
we're looking at how to use context awareness to associate terms with locations and people to
make these interfaces easier to use but also looking at how do we engage with individuals who
have communication disabilities to design this technology, so that we they can understand how
it works, so they can be invested and how it develops.
So this work was conducted in part of our ongoing project with an aphasia center in Baltimore.
There are about, actually now, closer to 50 or 60 active members, all adults who had aphasia,
most who had strokes, ranging from about late 20s to, I think 100 is the oldest one, so really
quite a right range of abilities. And we've been working with this group for almost a year now,
and have been working with smaller design teams. So the work today was with a group mostly
of five people and a few occasional helpers. And just to give you a little sense of how diverse
this group is, I'm not going to ask you to read all these details, but we had five people who were
most active in developing their prototype, they really range from P1, who has difficulty
retrieving words, especially nouns, he might forget what something was called, to P5 here, who
is unable to read or write, is severely impaired in comprehension and can speak a few single
words only. So he had about five words that he could speak reliably. And this is actually,
working with such a diverse group is actually largely from feedback from our collaborators at
the aphasia center who said, yeah, we know you guys want to come in and work with the
highest functioning people and build prototypes that work well for them, we really want you to
get to know the whole group and understand.
So we developed some early prototypes through our visits at the center and our students and
combined a number of methods, which are described in the ASSETS paper, looking at
observations, interviews, tech demonstrations, focus groups, when we got to the really tricky
stuff, like how do you describe a context aware computing system to someone with a
communication disability, we really relied a lot on storyboards and drawing and playacting and
building a low-fi prototypes. So looking at this kind of idea of context, we can imagine many
different uses of context and we generated these in collaboration with our research partners.
For this current work, we focused primarily on location and conversation partner as a way to
scope interaction. So we developed a prototype on the iPad, and it was a simple kind of menubased AAC system, like the ones we saw before, but with the additional information of context.
So at any time you could press a kind of sync button and it would find your location and you
could also identify your conversation partner either using the camera, although it was more
popular to actually just pick from a menu if you were talking to, and the system actually
develops over time a set of preferences, knowing such that whenever I talk to Kyle, Kyle and I
talk about piano because Kyle plays piano. And we typically talk about music and school. So
the idea is primarily that we have another way of organizing and focusing this interface in a way
that works better.
So we tested this early prototype, again this is ongoing work, so we’re still trying to get a sense
of how this will work in the real world, and there's some obstacles to that, but we revised our
prototype through several rounds based on feedback and some general ratings from using
aphasia friendly materials. So, in general, our participants so far are really enthusiastic about
this. They've used it quite a bit in the center, and now we’re looking to get it so they can use it
out in the real world. They were especially excited about location awareness that was the
feature that they were most thrilled about. And came up with a variety of places they’d want
to be able to use it. At the aphasia center, certainly, since they spend a lot of time there, but
also were really interested in taking it out and trying it out at Para transit, at the doctor’s office,
supermarkets, and going to restaurants and trying new things. So we are working on doing a
field evaluation so we can actually do that.
Our paper presents some guidelines on conducting participatory design for people with
aphasia, especially when we are working on context aware technology. But you can check out
the paper if you're interested. So we are working on an extended field deployment, we are also
looking at ways to bring in additional contextual factors, news and current events is one that
our participants are particularly excited about, and also how to beef up the context awareness
so that we can suggest possible relevant discussion tactics and present them in a way that's not
going to be distracting and frustrating.
So I’m going to talk just for 1 minute about the other places this work is going. So this is a
project that were working on with my PhD student, Lula Albar, looking at how we can use
pictures as a communication tool in a more meaningful way. So one thing that we were
interested to learn and didn't know before, but a lot of people, at least in this aphasia center,
are really avid photographers and they're very skilled photographers. This is a photo from a set
of a photo gallery sale that they had, and so they're taking a lot of photos and they also use
photos to communicate. But typically the way they do that is very driven by the conversational
partner who doesn't have aphasia. So they might hold of a photo of someone and the person
might say, oh, who’s that? Is that your cousin? And they'll go, oh. Is that your daughter?
Yeah, yeah, yeah. Oh it's your daughter, okay. Well where is that? Is that at school? Yeah,
yeah. Oh, okay. And so it's very driven by the conversational partner, which can be really
difficult.
And so one thing that Lula is looking at is how can we take the data that we have in photos and
actually use that to improve the interface or to organize the data and to provide another way of
communication. So taking these kind of unsorted photos here and starting to provide
contextual data about the photo based on what we know. So if we have a photo, we can say, I
was at the café, maybe I have a location tag, and I was with Sarah, so maybe I recognize Sarah’s
face, or maybe a long time ago I tagged Sarah's face and when I mention now, all the images
are tagged with Sarah last week, so looking at this both as a way to organize photos and also as
a way to, as a communication tool. So I can show this photo and get almost an automatically
generated story about the photo just from data that we have or data that we've added
previously.
We are also looking at other ways to improve the motor accessibility. So this is actually work
from Karl Wiegand, who is a student at Northeastern, but we are planning to collaborate in
looking at how we can use gestures and communication tools and so bringing it all together in
that way.
Okay. So just to wrap up, so I showed a number of projects in the two areas, really, starting and
looking at accessible touch interfaces for blind and visually impaired people; a few projects
across a range of devices, and based on analysis of how blind people interact with technology,
so understanding their abilities, both qualitatively and quantitatively to build technology that
works for them; and looking at how we can use context awareness to improve these
communication tools for people with communication disorders. I just want to kind of tie it back
to my original point, which was in thinking about work for the future, to say that a lot of the
work that we've done in accessibility traditionally has been reactive. So we take things, cool
things from main stream technologies, and make them accessible. I think sometimes we take
cool things from mainstream technologies and use that to make other things more accessible,
like some of the projects here, but I think it's useful to think about how can we get ahead of the
curve and especially how we think about accessible interaction and how to make that
something that's going to work well for individuals of diverse abilities. And I showed a few
ways that we can do that, and can continue to explore, so drawing on context when possible,
utilizing existing data that we have about people and what they do, and to really focus on
interaction methods that work for these individuals. So with that, I would like to thank you very
much. And I ended before you guys have to leave.
>>: I actually have one question.
>> Shaun Kane: Sure.
>>: Could you possibly explain the Access Overlays. I understand if you took a picture, so I have
a piece of paper, moving my hand around it wouldn’t be a big deal, but if it’s an ATM screen,
does it essentially attempt to explain where to touch the screen?
>> Shaun Kane: Right. So that's a really good question. So we really struggled with thinking
about in this project, assuming that the world is hostile. Right? So when I first started talking
about this, they said, oh yeah, but the ATM’s are going to have Bluetooth, and they're going to
talk to your phone, and they're going to turn it on. And those things might happen, but they
haven't yet and so thinking about what the individual due to empower themselves and the
technology that they have. And thinking about how you interact with a touchscreen, one of the
real challenges there is if you can't connect to the touchscreen, you still have this problem of
the interface can't really lay itself out and you can't change the gestures, right? So in
interacting with touch screens, which we've really just started to explore with the Access Lens,
you typically have to either hover your finger over, which doesn't work very well, or because we
can add accessible menus, we can add, for example, an index to the side of the screen, so you
touch along the outside of the screen on the edge and get an index and then get guidance
toward where something is. So like where, in the video it was saying up, up, up, up, left. The
other place you can do that is with, so one feature that I didn't show here, is with voice
command, so voice search. So if you knew, for example, that you wanted the withdrawal
button, you could just say withdrawal, and it could tell you where to move your finger above
the withdrawal button. It's not an ideal situation, there are still potential problems that you
might accidentally press another key, but it's a way to give us information that otherwise is
hard to get.
>>: Thank you.
>> Shaun Kane: Of course. Yeah, Jessica.
>>: With photo stories, have you considered perhaps using video to do the same thing, to do
the same task?
>> Shaun Kane: Say more about that.
>>: Okay. So it sounds like you have a photo to categorize the situation. How about like
recording video [inaudible]. Is that something, it would be interesting>> Shaun Kane: Yeah, yeah. So one thing I glossed over little bit is the idea that we are taking
these huge collections of data that people already have, right? So whether that's for the
participants we've been working with, that’s mostly photos, they have these huge photo
collections, so what we'd like to be able to do is say, we have some app and you just run all the
photos that you ever taken through it or you run your Picasso collection through it or whatever
that is and then get out a bunch of structured data of, oh yeah, you tend to take pictures of
these six people, if you tag them, now we have a nice way of searching this interface. I think
you could do the same thing for video, the challenge there is, it's easier to do the more
automatic analysis with photos, but I think that's, and especially as people take more videos of
how can we do that to communicate? Definitely playing the video itself is also really valuable.
One of the things about the population that we've been working with is they really don't want
the technology to speak for them. Like it's not a case where they press a button and the device
says, you know, hi, Jessica. What they want to do, what they'd rather do, and if they have to do
this, they would have to devise say that to themselves and they could say it to you. And that's
actually quite common. So they try to use the technology to help them speak rather than have
the technology speak for them. And that's something that's very particular to the level of
ability, this is a cultural thing, too, where they all come from the same center and that's kind of
the norm. Yeah, of course. Yeah, Allen.
>>: So this is a sort of philosophical question about your use of participatory design. So if you
were doing user centered design, you would be focusing on the users and try to figure out what
they want and so forth, at least in participatory design, in its classic form, you are actually
sharing power over the design. Did you do that, and what are the issues around it when you're
working with people with communication difficulties?
>> Shaun Kane: Right. That's a very good point. I'm using the term somewhat colloquially in
the sense of, we had a team of design partners who were giving feedback and setting the
directions, so there helping set priorities for the work that we did, but I think it was not as
equitable I’d like to be, and maybe as, you know, stretching the term participatory design a
little bit. I think one of the challenges is just it is really difficult for, and in terms of how these
individuals acting interact in the everyday world, even their conversations are more about
selecting from a set than about generation, right? So when you talk with someone with
aphasia, you learn if you have training, you tend to say things like, oh, are you hungry right
now? And they’ll say yes or no. Oh, are you tired? Oh, are you feeling really good? So you kind
of go through the list and then they kind of more vote, rather than, so I think getting evocative
feedback, at least in this population, is really difficult. So I would say that we haven't really
done that yet and I think I'd be interested
>>: Do you have any ideas how you could?
>> Shaun Kane: So one place, and this is some of the early work that happened with HDI
aphasia from UMBC, so Joanna McGrenary[phonetic] and et all, we looked at working with
other members of the support team. So working with the speech language therapist, with
family members, and so we've done some of that as well, working with staff, working with the
individuals directly, I think there's a possibility for more creative tools. Like making their
creative tools more accessible to them and so in our population right now, I don't think they're
doing that, we don't really have, we’re not really set up to do that, but I think if we had a way
that they could more directly manipulate the kinds of things that we are working with, that may
be something that would bring us in that direction. But we should talk about that sometime
because I'm interested. Sherry, you had a question.
>>: Oh, yeah. I had a question, slash suggestion. So in exploring ATM’s, I wonder if there's a
way to, it’s probably not a capacitive touchscreen but wearing a glove or something with which
you can explore with one finger, because the problem is, like somebody mentioned, that you
can't actually explore the screen without accident. You kind of have to hover, which is really>> Shaun Kane: So you're basically saying with a capacitive screen you could put on a stopper.
>>: Right. [inaudible] one finger and then, you know, use your second one to [inaudible]. I
wonder if you could do something like that since you are thinking directional>> Shaun Kane: Yeah. That's a really interesting idea. I hadn’t considered that. So I think,
yeah, definitely. A lot of these are resistive touchscreens, and then we have, and so actually,
that's the least of our problems because in some cases if the sensitivity is low enough, then you
can actually just kind of graze the touchscreen since you're not, so you can explore on a
resistive touchscreen through touching very lightly.
>>: Right. Or maybe put something over the touchscreen. I don't know.
>> Shaun Kane: No. I think it's a really interesting question, and we haven't figured it out yet.
But I like that idea having something like a, some kind of protector. Yeah, Mary.
>>: Could you talk a little bit about pixels, particularly it seems like a lot of real work uses kind
of the edge of things as a landmark, [inaudible] sort of seems like industrial design [inaudible].
How does that affect usability?
>> Shaun Kane: Yeah. So this is a real problem, and this has come up in work that we've done
too, of looking at, where you have this design where there’s no, the touch screen doesn't cover
the whole surface but there's no tactile indicator, right? You’ve got a sheet of glass over it, for
example, where you don't know when the touch sensitive part ends. In the past, we've kind of,
in several cases, we kind of work around this by using tape and other kinds of tactile things and
making a minor modification to the hardware and that works, but it's not going to necessarily
work in every situation. I think one piece that's interesting is it seems to be that also the
screens are getting closer to the edge now, so I think that's maybe less of a problem as we go
on, if the bezels go away, then you have at least the edge of the device maybe as a way to
interact. I think the other, thinking about the other problem of, when we do have these
devices, how does the user know when they've gone off the edge? I think that's something that
we could look at the interaction of that and giving feedback, so when we see the finger,
because we have a lot of, a lot of the mainstream devices now have gestures where you swipe
in from the edge, for example, so we should be able to see the other way and say, oh, you
slipped off the screen, I'm going to give you feedback even though I'm not getting any input
from you. So that would be one thing that we could look at.
>>: In practice do you>>: Before you go on, [inaudible] apologize [inaudible] question.
>>: I was going to say [inaudible] modern versions of the iPhone where you can’t [inaudible]?
>> Shaun Kane: I haven't seen that. And I think partially, you just learn [inaudible], especially
for something handheld, of this is the effective range. That's my sense, but also I think a lot of
the, Voiceover, for example, doesn't use the edge in a significant way. Voiceover you can do
from the center for almost anything. So that might just be, yeah. And did you have question?
>>: I was just curious if you knew [inaudible]?
>> Shaun Kane: So Chris Harris, I mean that group is done some work with tessellated touch,
and I know they had CHI work in progress. Yeah, it was a really small set. I think it was either a
CHI, it might have just been a work in progress, a year or two ago. I can send it to you. But I
know they looked at that in particular. Beyond that, I don't, the other thing you see is, coming
from the more traditional accessibility community, the tactile pen arrays[phonetic], so you have
these big, that big haptic displays that costs tens of thousands of dollars, and so if there have
been work with those two, but otherwise, in terms of their combination of, kind of what other
popular HDI tactile feedback techniques and blind people, I haven't seen very many.
>>: Is there a particular kind of feature that you can [inaudible]?
>> Shaun Kane: Yeah. That's a good question. I think>>: [inaudible] you have to still query the screen by moving around>> Shaun Kane: Right. Yeah, so I think the big thing is making an effective region partitions.
That's the big thing that I haven't seen. Like Tesla Touch can do that in the sense of, if you cross
something, you feel it, but if there were a way that you could kind of more easily follow a path,
I think, just at the top of my head, I think that would be really useful. In terms of, and what we
really want to do is, since we have this big space, you know, if we have a big interaction space,
how can we partition it in ways that are going to be useful and really quick to get to. Right? So
if you can turn the bigger touch screen into three or four distinct input areas that are easy to
find, that seems like a big win for someone who’s nonvisual.
>>: Any other questions for Shaun?
>>: I'm curious about something. So those ATM machines they kept showing, they do have like
an interface that I could, I've never tried to use one, you know, you put your earphones, you
plug it in, I saw the little icon for it on all those pictures, is there any comparison yet between
like, what's up with that? Is that not so useful? Is it something that people have trouble with?
>> Shaun Kane: That's a question that has many answers. Yeah, so I mean, the ATM example,
so in the US, most, if not all ATM’s, are required to have a separate interface, and typically what
that is is a headphone that you plug in and then buttons. My understanding from talking to
people is that they're very, very slow. So basically what it is, is a scanning audio, so they are not
ways to interact very quickly. And so one of the things we've seen with gestures throughout is
if you get them to work well, they can actually be really fast. And that's, I think, a big deal. The
other piece is that the social perceptions are a big deal. So people want to have access to the
things that everybody else has in the same way. I mean, this is something we hear again and
again and again. It's like oh, I want it to be, I don't want it to be designed for a blind person, I
want it to be what they have. And part of that is pride, part of that is just human nature, we
want to, we don’t want to stick out. So Kristen Shenohara[phonetic] has some nice work
looking at that in terms of how people respond to, or how someone with a disability perceives
the people around them, perceiving them, right? And then what their sense of how other
people view them and how that affects their technology.
>>: Well, I can see that at the ATM machine, right? Like when I plug into that earphone, I'm
telling the people around me that I'm blind. And so if here I'm getting money out of an ATM
machine>> Shaun Kane: Yeah. Kristen’s got a great example on her CHI paper from 2011 about some
paranoid person whose thinks they are going to get, it was in Seattle too, so yeah, it's not really.
It might've been a little bit overblown, but it was like, oh, I could get mugged. But also I think
just having that be visible>>: So I’m wondering about, like your example of different kinds of phone, but if people are, if
that's something that I'm used to, then again we’re making a judgment like, oh, well you
shouldn't use that, you should use [inaudible] overlays. So [inaudible], it’s difficult as you move
forward with this work, how do you also accommodate the folks who are like, I love prologue,
should go, you know, two point o, right? I love what you're talking about, but I really feel
comfortable with this. So I know where I am when I'm using this thing.
>> Shaun Kane: So in terms of, so as a researcher, I’ll answer that first, is finding the geeks is
really important. So a lot of work that we did that it was more, less conventional, with the
people who were identified as being tech savvy and really wanted to try new things, and they
got the iPhone first when the iPhone first had the accessibility features, and they wanted to try
things, and so, you know, there's a really big blind tech community with lots of blogs and
podcasts and things where they review technology; and so in terms of how to explore these
ideas and see whether there are any good, that's one way to do that. In terms of the longerterm picture of where that technology window is moving, that's tricky. We get feedback about
that too, I mean the early touch screen stuff, when we first started doing the Slide Rule project,
we got feedback from people saying, we don't want you to do this because a touch screen, and
accessible touchscreen is always going to be a very real way less accessible and less usable than
a physical interface, and we’re worried that if the people who make the touchscreens, if the
ATM makers could get away with having some lousy touch screen interface and say it was
accessible, that they would lose that. So there is some play there about how the whole legal
and policy side of how these technologies are implemented and also where, how do products
get developed right? So I'd be interested to see, and I don't know offhand, how the explosion
of the iPhone affects the companies, the small companies that are making the blind devices, are
making the customized things,, right? Those kind of things might get squeezed out. And then,
if you're relying on custom hardware, that's a real bind to be in. I think about this work as
demonstrating possibilities and say, oh, here's a way that we can take this device and use in a
way that we have before and kind of put that out there. But I think these are real issues and
people do worry about them.
>>: I'm kind of curious about the gesture stuff you kind of touched on. How much have you
looked at its specific hand gestures. The gesture recognition you at the moment seems very
2-D. Have you looked at the ways of recognizing certain hand gestures, for example, folding
sign language into navigation and things like that?
>> Shaun Kane: So we haven't specifically looked at that, but I think that's, there is definitely
lots of research on 3-D hand gesture recognition. This work, partly as kind of an evolutionary
result, was mostly using 2-D image processing, and it worked well with the tasks we were trying
to accomplish. I think if we started it today it would be much more, 3-D gestures would be
more of a part of it, and one thing about, one advantage is having gestures on a surface, is it
does provide some kind of, at least passive tactile feedback. So we have that, that's a useful
advantage, but certainly it would be useful to have more open air gestures could be useful and
so looking at things like>>: Less contact with the screen and more just mirroring the screen [inaudible].
>> Shaun Kane: Right. And I think that's, so that's also kind of thinking about kind of how to
give more augmented feedback, you know, taking something that has really limited sensing
capability and giving it additional sensing capabilities by proxy, that's really one thing to look at.
>>: I might email you after the studies.
>> Shaun Kane: Yeah. Please do. My e-mail is right there. Cool. Thanks everybody.
>>: Well, thank you Shaun, and thank you everybody for coming.