>> Meredith Ringel Morris: Hello, in-person audience and virtual online audience on resnet. I'm excited
to have Ron Baecker here to speak to us today. So Ron is a professor in computer science. Now, an
emeritus professor in computer science at the University of Toronto where he helped found the -founded the DGP group. Now, he works primarily in the Technologies for Aging Gracefully lab there.
And Ron's been recognized by many organizations in the U.S. and Canada for his pioneering work, both
first, in computer graphics, and then later, moving into HCI. And Jonathan Grudin, who is our sort of
resident HCI historian, has provided me with this exciting trivia fact about Ron Baecker, which you may
not know, which is that he is the progenitor of the one CHI Academy family that strings across four
generations, which consists of Ron Baecker and then his protégé, Bill Buxton, and his protégé, Brad
Myers, and his protégé, James Landay, so that is a good piece of CHI trivia for you to keep in mind. And
we're lucky to have Ron this week. He's in town for the ACM ASSETS conference, which is taking place
next week, where he's going to be giving the keynote address. And so we're lucky that we're going to
get a glimpse of that for those of us who will not be able to attend ASSETS, so thanks, Ron.
>> Ron Baecker: Sorry. I'm just resetting, resetting my clock to zero so I can see. Thanks, Mary. It's
always -- I seem to get here to give a talk about every four of five years, which gives a good rhythm to
my life. The anecdote about descendants in the CHI Academy -- the full story is I'm probably the only
person thus far, maybe for a long time, who was preceded into the CHI Academy by his academic child
and academic grandchild; although, indeed, I did precede James Landay into the academy, so anyway,
there are some advantages to age. I'm not sure how many there are. Okay, so I hope this will be
moderately interactive. If it gets too interactive, in recognition of the fact that this is late Friday
afternoon, I'll start to decrease questions. And apparently, I am being webcast to various MSR research
campuses all over the world, although how many people might be watching me now in Beijing or
Bangalore or wherever the other one is remains to be seen, given the time difference. So there's
apparently no way in which those of you who are remote -- for me to know how many such people
there are or what you're thinking or what you want to ask. But in the room is Mary Morris, is Jonathan
Grudin, is Mary Czerwinski, is Gina Venolia, and a bunch of other people that I don't recognize, so
certainly feel free to text questions to them and they -- I've seen a lot of proxies today,
videoconferencing proxies, so I hope these four individuals will agree to be question proxies. This is, as
Mary implied, a second rehearsal of the keynote address I am going to give Monday, and I've made, I
believe, some improvements from the rehearsal I did on Tuesday. And hopefully, you'll enjoy it and find
it provocative. I was a little hesitant in using the words "mind" and "soul" in a title for a computer
science audience, but at my age, I figured I could get away with it. And the work that I am going to
describe from my own lab -- and I am going to try to cast my net a lot wider than just our work -- is, of
course, the work of many people, and the names you see there are just the names of the current people
in the lab. And there's probably an equal -- probably two times as many who've been part of the lab in
the last ten years. It was only named four years ago, but I started doing this line of work about 11 or 12
years ago as I could see the 60 milestone looming close on my horizon, and I said well, it's about time to
do this kind of thing before I need it. So I am going to be speaking at ASSETS on Monday, and so I
thought I'd take a look at what the current ASSETS conferences think about what accessibility is and
what areas are being worked on. And I found it very interesting. I only went back three years looking at
the session titles -- I didn't look at the posters, and I didn't look at individual paper titles -- from 2011
through '13, and found that issues of vision impairment was, by far, the largest focus of attention at
ASSETS. A surprisingly small number dealing with hearing challenges. Three dealing with mobility and
wayfinding. Three dealing with aging and a number of others. And so, what I want to do first of all is
just talk about, very, very briefly, the huge advances that have been made in those kinds of accessibility.
Certainly dealing with visual challenges, there's been huge advances made in terms of things like screen
readers, audio description, descriptive video, turning a moving image into some kind of textural
description of what's going on in the movie image, as well as sonification, turning visual images into
nonspeech audio. All of these are an example of what I would call sensory substitution, or the recreation of something in a visual modality into another modality. And there's also a lot of -- there's
been a lot of work on haptic displays. And what's, of course, interesting to recognize in this work is that
there are no uniform universal solutions. So people who have acuity problems, those issues are very
different from those who have glaucoma, very different from these who have macular degeneration, so
the technical solutions need to have a fair amount of subtlety. There's also been a lot of work in hearing
challenges, in other words, taking auditory phenomena, sounds, and turning them into, for example,
captions or sign language. And this, the use of captions, is of course, interesting because it illustrates
what some people call the curb-cut effect. But it illustrates the fact that advances that are made for
issues of accessibility often benefit those whose faculties are normal. So, for example, I normally can
hear things, but if I go into a sports bar, I can barely hear the person next to me, much less something
that's coming off the screen. So if there are captions on the screen, I can read something about who's
winning the baseball game or whatever. And thank you for coming to hear me instead of watching the
Red Sox and the Tigers, who, I think, are going on now. I hope this doesn't mean people all over
Microsoft immediately leave your desk and go turn on your TVs. There's also interesting new work
being done in haptic displays. There's a small Toronto company called Tactile Audiodisplays that has
built a chair called the Emoti-Chair, which turns music into vibrations on your back, and so, again, this is
an attempt to substitute things in one sensory medium to another. There's also been a lot of work on
mobility and wayfinding, including smart canes, smart wheelchairs, et cetera. But what about seniors?
Are these the crucial issues for seniors? And this question is interesting because seniors are getting
more and more prevalent in our population. And the most striking data point that I've seen since I got
interested in this issue was from the U.N. in a 2004 report in which they said that in 1950, 5 percent of
the world's population was over 65. In 2050, it'll be up to 16 percent, and by 2300, should the world last
that long, it will be up to 32 percent, which basically means that, assuming you're not a senior and
you're randomly distributed among other people, if you look to the right of you and you look to the left
of you, one of the three of you will be a senior citizen. So that's quite staggering, and in fact, the
percentage of seniors will pass the percentage of children, defining children, I believe, as under 15, and
seniors as over 65. That could happen in two or three years, okay, so the world is starting to look like a
very different place. The good news is that, of course, if we're healthy, and if we have things to enjoy in
our life, in principle, we can enjoy them a lot longer. And the bad news is that we have all sorts of
obstacles to enjoying life, ranging from things that are clearly defined as physical and neurological
conditions like Alzheimer's disease, aphasia resulting from stroke, difficulties in speaking, or Parkinson's,
shaking or things like that, and also, of course, blindness, deafness, poor mobility, et cetera, the issues
that the accessibility community has worked on very hard. But also things that you could view as
psychological or social barriers to leading a healthy life, such as isolation, loneliness, vulnerability, et
cetera, things that the medical community is not always willing to recognize is a disease or a condition
that the medical community needs to deal with. So the question is can technology help with any of this,
and if so, addressing what needs? I like taxonomies, and I like my style of work in new areas, and I've
changed fields several times in my career is to start dabbling in things, trying various things, and then I
try to pull back a little bit and say, "Is there some sort of systematic formalism or framework or structure
in which I can categorize pieces of work?" And a couple years ago, four years ago maybe, I ran across a
paper by psychologist Abraham Maslow written, I believe, in 1944. And Maslow argued that there was a
hierarchy of human needs starting at the lowest level with physiological needs, things like we need
oxygen to breathe, we need water to drink, et cetera. If we don't have that, we're going to die, and then
moving up somewhat higher to the level of safety, security, to the next layer he did was he called
actually need for love, affection, a sense of belonging, family, friends. The fourth layer he defined was
what he called esteem, the need to feel good about yourself and your capabilities, to be able to engage
meaningfully with the world. And the fifth level, which is perhaps the fuzziest, was what he called selfactualization. The need for fulfillment, a cause, some reason to exist, some higher purpose. And what I
want to do today is argue -- and I'm not going to argue at length, but I am going to assume it -- that
when I talked in my title about the mind, the heart, and the soul, I am going to equate the love layer
with things of the heart, the esteem layer with things of the mind, and the soul layer with the -- sorry,
the self-actualization layer with the soul and hence my title. So the opportunity for us is to identify
needs that somehow fit into this taxonomy or this space, and then come up with technological
solutions. And there are probably more than four categories, but the four I find most interesting are
diagnostic, prosthetic, rehabilitative, or preventative. And the easiest way to explain that is to use the
example, let's say, that you're out skiing on -- I don't know where people ski, ski here. Mt. Baker? Do
you ski at Mt. Baker or is it -- I don't know. There's places to ski. And so imagine you have an accident,
and you don't know whether, in fact, you've broken your ankle or you've just twisted it or sprained it.
So you might go have an X-ray. That's a diagnostic instrument, and that'll tell you yes, you broke
something. If in fact, you broke something, or if you severely you injured it, you will have a prosthesis.
In other words, something that compensates for this loss, whether it's a cast or a crutch. If then you
start to get stronger, you may see a physiotherapist, and that's a rehabilitative system to strengthen the
ankle. And then finally, you might think -- and particularly if the person who had the skiing accident is,
you know, a reckless 17-year-old male or something, their parents might say to them, "Well, you know,
if you had gone down the mountain a little slower and not been also talking on your cell phone, you
might not have had the accident in the first place." So and that's what one would call a preventative
intervention. And so those are the kinds of things we can look for in this area. And what's important to
realize is that any technology intervention that we develop should be and can be informed by medicine
and the health sciences. And in turn, as we do technological interventions, they often give us new
insights into the disease or the condition or the disability that one is attempting to deal with. And so our
lab, which is called the TAGlab, Technologies for Aging Gracefully lab mantra, research for the journey
through life, tries to design and develop technology to make seniors and their families smarter, more
capable, resourceful, and independent. And it's very different. I'm an HCI person, and this is different
from people who work in the same field who come from AI and computer vision who often try to make
machines smarter to watch over seniors, to ensure that something bad doesn't happen. And I'm not
here to argue that one approach is better than the other. In fact, we try to build as much smarts,
smartness, intelligence into the things we build as well, but we come to this, in my lab, from the HCI
approach. And so what we try do is to think where technology could serve human needs and to
envision, design, build, test, improve, and commercialize solutions for a variety of conditions, including
what somehow -- sometimes quaintly is called normally aging senior citizens. And so the outline of what
I'm going to do in the next 40 minutes or so is to go through these five layers and talk about a little bit of
work in each layer, in some cases, talk about the work of others, in some cases talk about work from our
lab, and then wrap up with some general thoughts that can be drawn out of all the layers. And I'm
certainly willing to take questions at any time. So starting with the layer of health, this is an area where
there's been lots of work. One phenomenon that particularly interests me is the increasing proliferation
of authoritative information sources or apparently authoritative information sources about health on
the web, and this is includes information from government-sanctioned websites such as PubMed Health
to very well-known and distinguished and authoritative hospital websites such as the Mayo Clinic and
more commercial websites such as WebMD. And you see here three snapshots of information about
prostate gland enlargement there. There's another phenomena which approaches this from a different
point of view, which is not the authoritative, summary pronouncements about certain conditions that
are defined by medical science, but the more anecdotal "here's what happened to me when I had my
brain surgery" or "here's what happened to me when somebody close to me passed away" or whatever,
"what happened to you?" and these are social media for health. Here's an example from
my.braintumorcommunity.org. And this all, in my view, is good for us as intelligent consumers of health
information. It leads to challenges such as, How do you decide what's correct, what to believe in? And
also how do you encourage the medical system and medical practitioners to spend some of their time?
And this is more of a problem in Canada than in the States, because in Canada, everybody gets
healthcare, but there's increasingly little time per everybody. Whereas here, if you have more money,
you can get, you know, you can get better healthcare, and you probably have a better chance here to
say to your doctor, "Okay, I need 10 minutes for you to explain to me why I shouldn't worry about all the
drug interactions that I saw on this Wikipedia site and why I should take this in confidence." In Canada,
you probably won't be lucky enough to find an M.D. who can do it, but you will certainly get -- you don't
have to worry about what happens with healthcare if you're poor. Another area -- another area is
monitoring physical fitness for health. These are slides about a device called Fitbit, which a lot of people
wear, which gives them information about motion in three dimensions and allows them to monitor their
activities. And then another thing that I think is particularly interesting is the number of seniors now
who are either keeping Wii fit or keeping Kinected and using some of these technologies for encouraging
exercise with things like Wii bowling, which I mean, I don't know how many seniors play Wii bowling,
but certainly that kind of activity is getting more and more common. So that's all I want to say about
physiological needs. I now just have one slide about monitoring for safety, and that's the next slide.
And I just want to mention two, what I think are, outstanding pieces of work. One is here from the
Seattle area, and in fact, I believe I saw Shwetak Patel wandering around the halls upstairs. So he's done
beautiful work on technology which you can either attach to your electrical circuits or to your hydraulic
circuits in the home, and they could be used for things like making inferences such as, well, for the last
two years, there have been electrical pulses around 9:00 every morning from two places in the kitchen
that could plausibly be the toaster and the coffee pot. Nothing like this has happened for two or three
days. Either the person has gone away, or maybe there's something wrong and we should take a look.
And similarly, for fluid flow, dealing with issues like sinks and toilets and things. And this, as I
understand, is now being commercialized by a company. I think it's called Belkin. So in the upper left,
you see some images taken from some of the Belkin literature. Work from Toronto that I find very
interesting and that uses state-of-the-art computer vision and machine learning, is from Alex Mihailidis
in Toronto Occupational Health Sciences as well as Toronto Rehab Institute. And that's designed for -- I
don't know to what extent in this area they sell the device where the television ad is, "I've fallen and I
can't get up.” Okay, and that works very well in the cases where you remember to carry it with you. It
doesn't fall out of your hand when you fall down, et cetera, et cetera, et cetera. But what Alex has done
is he's built some technology which sits on the ceiling and I think you can see it in this. Let's see -- okay,
actually I have a pointer there. You can see it up there. And if this is an actual scene down here, it
normalizes it and gets the raw image, and then it does some silhouette detection and is able to say, "It
looks like you're falling. Would you like help?" Et cetera, and so this is -- these are good examples of
where smart computer vision and smart machine learning and AI really can be greatly beneficial. Okay, I
now want to move -- and you'll notice at the bottom of some of these slides, I'm showing where I am in
the outline of the talk. So I'm now moving to the third level of the hierarchy, love needs. And so I want
to start with a question, in part because I hate a talk that's just a monologue even if I'm the monologuer.
So, can anyone here tell an anecdote? It need not be someone you personally know. It could be a
friend, a friend of a friend, something you read, but something about somebody who's isolated, lonely,
vulnerable, and what the implications of that for their life has been. Yeah, Mary.
>>: Well, it's usually severely negative and ending in death, right?
>> Ron Baecker: Yeah, yeah. So anyway, it -- these situations are clearly negative. There's an image
here of a happy couple Skyping one another, and not everyone is in a position to do that, and so, in fact,
there are large collections of cases in which this is an issue. So not every senior living alone is lonely and
isolated or isolated and lonely, but many of them are. Many people who are in long-term hospital care - I'll get back to hospice care towards the end of my talk. A large collection that's under recognized
perhaps and undervalued by society are 7/24 homebound caregivers. People, you know, like someone
who's 68 years old who's got a partner who's 74 years old and who has dementia. And you don't have a
lot of money, and you can't hire people to come in to take care of the person, and you're stuck there day
and night. You don't have kids who are nearby, et cetera, et cetera. People who are in long term -- it
doesn't have to be seniors. It could be people who are just hospitalized for a long term because of some
surgery or rehab or quarantine, or individuals with chronic pain. There's statistics which say that the
number of people with chronic pain in North America is something like 25 percent. This is pain that
persists for longer than three months, which is just a staggering number that I never believed was as
serious until I started talking with Diane Gromala of Simon Fraser, whose, essentially, life work is
technology for chronic pain. And so this often applies to individuals who may or may not live alone,
have relatively small and, perhaps, shrinking social networks, may also have sensory or motor
impairments, which makes it less, and also have little control over how they feel or their availability for
conversation at a particular moment. So, for example, if you're in a hospital bed and even something as
little as can you talk on the telephone. Many hospital beds don't provide a telephone next to you, and
you may have to summon someone on the nursing staff in order to talk on the telephone or do
something, so that can be a serious issue. So we've started -- we've gotten interested in this issue.
When we've surveyed the issue of video communications -- and being here has been very interesting as
it was four or five years ago when I was here -- there's sort of lots of technology around for synchronous
video chat commercially, Skype, Google Hangouts, et cetera. There's the "research frontier" and I put
"research frontier" in quotes, because, in fact, in the CSCW community, people at Bellcorp was doing
this 20 years ago with video windows. And I understand video windows have been quite controversial
at MSR, but the notion that you had a way to imagine as if you were living next door to your -- to your
children and you could -- and there was a window, literally, between the two buildings, and if both of
you agreed and pulled up the blind, you could communicate with video and audio interaction, so that's
with video windows. And a person named Tejinder Judge did some very nice work on this called Family
Window and Family Portals a few weeks ago. We've chosen not to start with synchronous or always-on
video, but with asynchronous communication or video messages, in part, because we've been very
influenced by the fact that people are not necessarily always available for conversation when you want,
and we thought the notion of just saying, "Hey, I'm thinking of you; get in touch" was very valuable. And
so what we've done is we've built some prototypes over the last few years. We've done a lot of
interviews, focus groups, and observations in a variety of settings: people living at home with chronic
pain, individuals in a complex, continuing-care hospital where they're in for very serious conditions and,
in fact, often don't leave, as well as we've been working with a company that owns retirement homes
and long-term care nursing homes, and we've been doing interviews and observations there. And we've
developed a set of design principles which, if the gods smile upon us and a random number generator is
-- is dialed in the correct way, you might get to hear at CHI 2014, but if not, we'll submit it somewhere
else. And as I said, we start with asynchronous interaction, and our goal is to build from there. And our
goal is to build something that looks like an appliance and that doesn't look like another program on a
computer. And so I am going to show you a few early prototypes. And this video does have narration,
so if I can find the start. [video begins]
>>: -- technology designed to aid communications by isolated individuals who may be lonely or
vulnerable. Families in Touch, by Elaine Macaranas and Thariq Shihipar, introduced the metaphor of a
digital communicating picture frame. Jessica David's Ringo advanced the concept further. Here, her
grandfather sends out a "thinking of you" message from his picture frame by touching it. Jessica
receives it and records a video greeting to grandpa, mentioning her dad, his son. "Hey, Grandpa, just
got out of class. It's really beautiful here in Toronto, and it's becoming fall. Dad wants to send you a
video of the dog later, so hopefully, you get that. Talk to you later, bye." Grandpa then receives and
watches both videos. "Hi, Grandpa, just got out of class. It's really beautiful here in Toronto as it’s
becoming fall. The leaves are really pretty." Jake, what do you want? What does Jake want? Huh?
Wanna go outside? Keith, by Mark Fema [phonetic] and Isma Khan [phonetic] explores a different way
of controlling the device and a different context of use. Isma's dad, here, pretending to be in a hospital
bed, commands the communications by gesturing to the other side of the room. He can send requests
to a nurse elsewhere in the hospital or send messages to family.
>> Ron Baecker: That last demo was done on a Microsoft Kinect. And the images on the right are
images out of the current prototype InTouch, which
does something very similar to Ringo, and we
hope to start testing with seniors in retirement homes and long-term care starting in January. There are
lots of -- this is a lovely domain to work on. There is lots of different research challenges or
opportunities. One is what's the metaphor you use? We started with a picture frame. Originally, my
idea had been to do this through television, because the work was motivated by visiting my sister in a
long-term care facility in the last couple years of her life, where she got increasingly isolated and lonely,
and I imagined the TV. Everyone in the -- everyone in a bed -- often two TVs in a room with twin beds -had their TV on, and I imagined the TV stopping and a voice coming out. "We interrupt this mindless
drivel from CNN with a message from your son. Hi Mom, it's Neil. Went golfing yesterday, shot three
birdies," et cetera, et cetera. So we've been looking at picture frames, TV screens. We believe that, in
fact, all three of the communication modalities, synchronous, asynchronous, and always-on, are
relevant and the question is how to use them together in some elegant way. And although we started
with touch, we are interested, as you saw in the Keith video, in gesture, but for the moment have put
that on hold because we wanted to explore voice as the second modality because we think that's
probably the most important one besides touch. Okay. Before I go on to the next topic, I would be
happy to take a question here or a comment if anyone has one, and then I'll continue. Okay. So the
next area I want to talk about I classify under esteem needs -- although, in fact, it sort of bridges
communication and thinking -- and that's having trouble to communicate and to speak. And we started
out working with technology that we developed for stroke survivors who had aphasia, but more
recently, we've started to think about another kind of impairment or disability that affects a lot of
people including seniors, particularly new immigrants, which is dealing with English as a second
language. Again, if anyone wants to share an anecdote or a story about someone they know who's had
a stroke or a child with, you know, autism spectrum disorder or someone you know who's an immigrant
who has trouble with English, I'd be happy to make this a little more participatory. And if not, I can
essentially go on, but I'll give you a few seconds to pipe up if anyone wants to say something because I
find that these stories often sort of bring it, bring it to home. Okay. So once again, there are
commercial products here that are making a difference. The fact that now you can get speech synthesis
that sounds almost human is -- is a big step. The image you see on the upper right of this screen is the
collection of materials that a stroke survivor named Bill used to work with to help himself communicate
as he wandered around Toronto. What's hard to see in this is -- I think it's -- I think it's this. Well, no. I
don't know where it appears here, but he actually has -- the most important thing are not all these maps
and business cards and things but a book that he put together that's essentially 2,000 words and
phrases that he finds the most useful for his communication. It includes things like stories of what his
kids are doing and stories, you know, things he wants to do in informal social interaction as well as
things like, I had my -- I had my stroke while being on the subway, et cetera. Now, you have to
remember that Bill is no ordinary stroke survivor. I mean, if you rated people in their intelligence and
their fortitude to try to say I'm not going be beaten by this, I suspect Bill is in the upper one percentile or
more. I mean, he's an engineer. He basically said, after he pulled himself together again and he realized
he couldn't remember what words and he couldn't barely articulate them, he said, "I'm going to beat
this," and so, your average stroke survivor isn't capable of doing that. And so my lab spun out a
company called MyVoice, which has an application called TalkRocketGo, which basically bring up words
that seem to be relevant in specific contexts and either allows you to see them on the screen or will
speak them out with a speech synthesizer. In terms of language aides, there are, of course, things like
Rosetta Stone. I don't know what TV is like here, but you can't watch television in Toronto without
seeing a Rosetta Stone ad. But none of this, none of the commercial products really deal with what we
think is the most serious problem or opportunity, which is supporting user needs in situ or situated
learning. In other words, how do you help the person by somehow zeroing on the vocabulary that might
be most relevant as opposed to forcing them to look through a whole dictionary? And so MyVoice
started working on that, and they have some of that in our commercially available product, but I have a
Ph.D. student, Carrie Demmans Epp, who's three quarters of the way through now, who's doing a
beautiful job, I think, with a technology called VocabNomad, which thinks about, like the MyVoice
technology did, in terms of vocabularies that are spatially relevant, and we call them "locabularies.” So
if you -- to understand the idea, if you're in a doctor's office, there's a particular set of words that are
more relevant for that condition than if you're at a movie theater, and that's going to differ, again, than
if you're, you know, in a Starbucks. That's not to say that in a Starbucks, you couldn't talk about the
movie you just saw or your forthcoming doctor's appointment, but still, the most critical things in a
doctor's office are words like nurse, appointment, blood test, and things, and the most critical things in a
Starbucks are things like black, double-double. I don't know, is that -- do you use that in the States, or is
that just a Canadian phrase, double-double? Donut, croissant, et cetera, et cetera, and so. And I don't
have a video of this unfortunately, and I decided to stop showing the MyVoice video. You see here -and it's hard to see -- three screenshots from VocabNomad. The upper left screenshot shows you some
of the interface for helping to tailor vocabularies on a desktop machine. And here, someone is working
on the milk vocabulary including things like -- or the liquids vocabulary. One percent milk, two percent
milk, 7-Up, et cetera. Down on the bottom left, you see the notion of chunks of vocabulary that are
relevant in particular situations, and so here, you see a portable vocabulary chunk -- or I think of them as
vocabulary books -- for numbers, for colors, for parts of the body. In the upper right, Carrie
demonstrated this at an artificial intelligence education conference recently, which was in Memphis,
and so she tried it. And it knew she was in Memphis, so it came up with American South, Baptist,
Memphian, et cetera, which wasn't bad. And I asked whether it would come up with Elvis, and she said,
oh, Elvis came up as the sixth term, et cetera, so that's not too bad. Johnny Cash, maybe, I don't know.
I've never been to Memphis. But so, again, in this area, we have huge research challenges, as do others
in the community. There's the classic problem from computers in learning, which is how do you track
learning, learner progress, and intervene appropriately? There's the automatic locabulary problem,
which we've barely scratched the surface of. There's what I'll call the social media for speech
production problem. Not only the idea that there could be vocabulary books that could be contributed
by people in the community, but the notion that, you know, you might be in a particular situation and
send out a message over the Internet to say, "I'm having trouble saying this. Could someone help me?"
and the question is, how quickly you could get an answer back. There's lovely work that was reported at
CHI by Erin Brady. Did she work with you? Yeah. Steve, Jeff Bingham's student. Of basically taking a
picture of things, like medications and things -- this is for blind people -- and saying over the Internet,
"Can you help me? What am I looking at?" And so the use of social media to help people with not only
visual challenges but speaking challenges, et cetera, sounds very interesting. Okay so I have two more, a
couple more things to talk about. Any questions or comments here before? Okay, so the next thing I
want to talk about is being able to read, and I won't -- I'll skip the exercise of trying to get you to talk
now. So I know it's Friday afternoon. It's been a long week. Did you have -- was Monday -- did you have
Monday off here also? No, you didn't have Monday off, okay. In Canada, we have Monday off for
Thanksgiving, although we ate so much that we've been sluggish all week. So we've gotten very
interested in the problem of helping people read. And this, I believe, is important or compelling,
because look at all e-books out there. I mean, you're sitting in Seattle here, and you've got Kindles, and I
forget -- does Microsoft have its own e-book yet or not yet? No. You have an investment in Nook, is it?
Yeah, Nook and whatever. So but anyway, most of the e-books don't seem to pay attention to people
with disabilities whether -- and it pays lip service to it, because you can enlarge type, but it doesn't do it
in a terribly sophisticated way. And the research community has started to look at this, but not nearly
enough. The most exciting piece of research that I've seen is by Hayes Raffle who's -- he was at Nokia.
Is he still there? Is he?
>>: They made a start-up company out of this.
>> Ron Baecker: Oh, they made a start-up company on this?
>>: A bunch of them left Nokia to do a start-up.
>> Ron Baecker: Oh, good, okay. Well, I need to find out about this. Anyway, I thought this was just
lovely work where you essentially combine desktop videoconferencing with a shared book that can be
synchronized at either end, at least two party. I don't know whether they've done multiparty, but twoparty is really the sweet spot. And this was very clever, but it also had an agent that would come in and
start making cute or intelligent comments, trying to get people to talk if they weren't already talking.
And, you know, I think this is lovely work. Anyway, so we've started to think about what we can do for
e-books for reading, and we came up with one of the world's worst names: The Accessible Large-print
Listening and Talking e-book, ALLT. And the idea was to be able to deal with -- the original idea was to
build sort of the equivalent of a talking book in a large-print book. Okay, so it could enlarge the type,
but it could also read it aloud, and those two technologies, talking books and large-print books, seem to
be different technologies. But also, we figured that it could be a little smaller than that. It could not
only talk, but it could listen, and that would enable it to be able so you could read for somebody. You
could record a play with people taking different parts, or you could read and then the person could
listen to it again. So I'll just show you a video we made a little while ago about ALLT. [video playing]
>>: ALLT is an app that helps people read by themselves and together with family members. Like an
electronic large-print book, type can be enlarged for individuals whose vision is poor. It can also be
shown on a high-def display or a TV for family viewing. Like books on tape, the text is read aloud for
people who are blind. ALLT is synthesizing the voice. That is not just for prerecorded books, but for any
digitized text. ALLT is accessible to people with motor challenges such as occurs in MS and Parkinson's.
It can be controlled by this mobile keyboard, or connected to a keyboard that also provides a stand for
iPad. [synthesized voice] It was not that he felt any emotion akin to love for Irene Adler. All emotion,
and that one particularly ->>: ALLT can also access a million books via the Internet archives, so let's pick the category Children's
literature English, and we find Alice's Adventure in Wonderland, and we'll download that. Family
members can read books aloud to ALLT users. Readings are recorded and can later be heard by the
user.
>>: Twas brillig, and the slithy toves. Did gyre and gimble in the wabe. All mimsy were the borogoves,
and the mome raths outgrabe.
>> Ron Baecker: And now you're going to play it back. So this is an area where we're just really getting
started, and the interesting opportunities are to think about sort of the old classic space-time
taxonomies from CSCW literature, and think about people reading together at the same time, at
different times, face to face or across space. And this, we believe, has a lot of opportunity, again, for
seniors who are isolated and lonely in order to embed social interaction in a cognitively interesting task
space. So the last part of my talk is now I want to go up to self-actualization, the soul. And we started
this work about eight or nine years ago. And I was working with a social worker, Elsa Marziali, who
worked at place called Bay Crest that has a lot of residents who have dementia. And she had done a
little project in which she had made a little 15-minute video of somebody's life, and it seemed to really
be valuable to them to look back at the video. And so we managed to get some Alzheimer's Association
money to look at it more systematically, and so our project was called multimedia biographies, and it
was really targeted at the very large number of people who -- at the moment, there's about 40 million
people with dementia worldwide. The most common form of dementia is Alzheimer's disease. About
85 percent of cases of dementia are Alzheimer's disease, and that's supposed to grow to 66 million or
more by 2030. And there's also a new condition that's only been recognized in the last ten years called
mild cognitive impairment, in which you start to have memory problems and some issues of confusion
and disorganization that doesn't detect or measure as Alzheimer's disease, but in 50 percent of the
cases, if you're diagnosed with mild cognitive impairment, within four years you'll be diagnosed with
Alzheimer's disease. So these are people who are very aware of the precariousness of life and anxious
to do what they can not to prolong it, but to enjoy it and to possibly preserve their history for their
families, et cetera, et cetera. So we worked with a dozen people over three years of the project, half of
them with MCI and half of them with moderate or mild Alzheimer's disease, and we sent out so-called
multimedia biographers, who worked with these individuals, typically, for about six months but very
intermittently to sit down and say "Well, what's the story of your life?" Now, in 12 of the cases -- in 8 of
the 12 cases, we worked with surrogates for the individuals, adult children, for example. In four of the
cases -- in four of the six cases with MCI, we worked with the individuals who were still healthy enough
to say, "I want to tell my own story." And so we had to work with them to figure out what is the story of
their life and then go out and capture and digitize all sorts of past archival material, you know,
photographs in a shoebox, old Super 8 cartridges, et cetera, et cetera, sometimes shooting new
material. Mike Massimi did a lovely project as an intern at MSR Cambridge five -- six years ago -- six
years ago, maybe -- in which he did a project like this also using SenseCam to help capture new imagery,
and so we did this. One of the things we wanted to show is that this was practical to do, to develop
something that would be valued by the individuals and the families without the expense of -- imagine -imagine Scarlett O'Hara coming to the end of her life and saying, "Okay, well, gee, I'd like Rhett, and
insofar as he still cares, or others to remember what my life is like. Okay, we'll make Gone With the
Wind as the story of my life." Okay, now, that, I think, would cost $300 million or something like that.
So we wanted to show that this could be done by undergraduates in various disciplines -- they didn't
have to be trained cinematographers -- with about a hundred hours of time. So let me show you two
examples very, very edited down. This woman was brought here from South Africa five years, roughly,
before this video was shot. This work was done by Tira Cohene who some of you may know. She works
at Microsoft in one of the product development groups. And Ms. F is sort of reflecting on her life while
watching, in some sense, the story of her life come in front of her. [video playing]
>>: I lived there.
>>: Yeah, you did.
>> Ron Baecker: So if I hadn't edited it down as much, you'd see numerous examples of her saying like,
"Oh, Table Mountain Top,” et cetera, et cetera. She recognizes names, places, faces, but she also reexperiences or remembers feelings. It was very good to live there. The real payoff of this proved to be
when individuals watched these with adult children, typically, or with partners. And so here you see Ms.
Z, also with midstage Alzheimer's, watching the story her life with her daughter, who's also the narrator
of the video, and you'll hear the daughter both recorded and live. [video playing]
>>: This is you in one of [indiscernible] suits. Really astounding suit. What an excellent couturier
designer he was.
>>: It's very important to me. Here you are looking so beautiful.
>>: Katherine Hepburn [laughing] ->> Ron Baecker: And she goes on laughing for another 20 seconds or 30 seconds. [video playing]
>>: My father used to sing it to us. You are my sunshine, my only sunshine. You make me happy.
When skies are gray. You'll never know, dear, how much I love you. Please don't take my sunshine
away.
>> Ron Baecker: So the power of this for reconnecting and encouraging conversation is really quite,
quite compelling. As with every one of these topics, there's still many, many research challenges. To
what extent can we use new technologies such as SenseCam, or I guess, the Vicon Revue now or Glass to
capture new video material? To what extent can we use social media to gather raw material together
and then somehow combine it into what's really important, which is transforming individual memories
and experiences into stories, which I believe is what all of this is all about. I now want to just say very
briefly, about -- so if you think about preserving your identity as you reach critical stages of life in terms
of your physical health and your mental health, what about the end of life? And I just have one slide
about this. My Ph.D. student Mike Massimi, currently in the second year of an internship at MSR
Cambridge, who's really been a pioneer near in thinking about this, actually coined a phrase
"thanatosensitive design." Thanatos is the Greek god of death, and the idea is to think about how to
design technologies and systems and environments that acknowledge the fact that we're not going to
be here forever. And so what you see here is a screenshot from a wonderful website developed by a
friend of mine, Kathy Kastner. It's called Best Endings, and the goal of the website is to encourage you
to think, talk, learn, plan, and share about what she describes there as "the shocking realization a
hundred percent of us are going to die." Other kinds of work in the area. And I don't have time to go
through the papers in detail, but in the forthcoming CSCW conference, 2014, there's a paper by
someone named Ferguson -- who I believe is a graduate student at McGill -- and Karen Moffat and Mike
Massimi and one other person, that talks about technology in hospice care. Hospice care is a setting in
which you basically acknowledge that someone is near the end, but you're trying to make them as
comfortable as possible. And we've been particularly interested in that in my group, is that the kind of
video connection technology that we're talking about with InTouch might be relevant in hospice care.
And then Mike, in CSCW2013, published a paper that talks about can technology be relevant after
there's been a death in terms of online bereavement support. So we've been doing this kind of work for
about 11 years now, only the last four or five under the name TAGlab. When I started, I thought we
were dealing with memory prostheses, because, in fact, one of the things that got me started was
reading a classic paper from -- I'm blocking the name of the fellow. He used to be at euro park and now
lives in Silicon Valley. Anyway, it was towards an electronic memory prosthesis. It was written about
1990, and I read it again about 2000 and I said, "Boy, we haven't gotten too far, have we?" So but I've
now realized as we've done this work together with collaborators from the health sciences, et cetera,
that it's not just about memory and cognition; it's about identity, efficacy, and self-worth. And it's not
just technology for individuals; it's collaborative technology for family, community, caregivers. And that
was illustrated very strongly be the Ph.D. thesis of Mike Woo who finished a few years ago, and that was
published in I believe in CHI and participatory design. The goal is not just about prosthetics but about
the rehabilitation of health and the preservation of health. And it's obviously multidisciplinary, and this
is not work that can be done in the lab. It's got to be done -- my students who have a huge burden
because I ask them not just to build compelling prototypes but to make it robust enough to take it out
and try it with real users in real environments, et cetera, et cetera. There are lots of other research
challenges in this area that haven't been in the topics I've discussed. Remembering important objects,
remembering to take them with you. And finding lost objects is one that one of Gregory Abowd's
students started to work on about 10 or 12 years ago, and I believe is still not solved. Recalling names is
one that we've dabbled at a little but we haven't solved. There's been a fair amount of technology using
things like Kinect and other things to help people who have walking conditions or -- or -- but the notion
of being able to monitor your walking in real environments and detect both trouble spots with a human,
like something wrong with the way your leg is working or a problem that's caused by the environment -bumps, things you can trip on -- is very interesting. And then finally, there's the phrase
"neuroplasticity." And I don't have time to go through it in detail, but neuroplasticity relates to the
notion that our brains are not, as they used to be thought, sort of pretty well set by about age four or
five, and then, yes, we could cram more information in, but, you know, if we had an IQ of 120, that's
pretty well where we'd stay. But there's increasing evidence that, even in late in life, if the brain gets
what some people call insults to it like a gunshot wound, and we've seen several cases of that,
unfortunately, in the last few years, where people have gunshot wounds and start to do remarkable
recoveries, and it's partly because of neuroplasticity, and the ability for the mind to increase its what
some call cognitive reserve to be resilient in the face of adversity. And so I believe there's -- I don't have
the time to go through a description of or my views of the so-called brain fitness industry, but there's a
brain fitness industry now that believes that you can build these little exercises and if you just sit in front
of your -- your Nintendo device or your whatever for a half-hour a day, you will get smarter, or you'll be
able to delay cognitive decline. And what I believe is the interesting challenge is to harness our abilities
to stay -- to stay mentally sound through conversation, through exercise, through normal everyday
activities, even though playing classical games like Scrabble or poker or bridge or chess will be better for
us than these same brain fitness exercises. A brief plug: As I faced 25 years ago, when Bill Buxton and I
started teaching HCI, we couldn't find teaching materials to teach from, even though Ben Shneiderman
assures me that if I had looked a little more carefully, his book was almost ready and we could have used
it. So that then led us to write some of the early readings collections in HCI. And now there's a new
approach to this developed by the same Mike Morgan, this time Morgan and Claypool, and so I'm
editing a series of synthesis lectures on assistive rehab and health preserving technologies. The upper
row are volumes actually out, and the bottom row are things that are in preparation. The one on autism
is coming out, literally, any day now. Why do we do this? Well, this is sort of a quiz. How many of these
eight productive, mostly still alive, although not all, senior citizens do you recognize? We can't all be
Casals or Picassos or Mandelas or Gorbachevs, but I contend that by doing work such as what I've
described from others and from our lab, we can enable lots of seniors to have additional years of happy,
healthy, contributing lives. Thank you for your attention, and I'm happy to stay as long as you want for
questions and discussion. [applause] And I encourage those of who are out there, if you are out there -and I wish I knew who was out there -- that if you know how to text any of the people whose names I
mentioned before, you could text in a question or email them a question. Yeah, Mary.
>>: You talked about a lot of technologies, but I didn't see a lot of results showing that these
technologies are actually having a positive effect.
>> Ron Baecker: Right and that -- that's -- that's a function of sort of the breadth-first approach I chose
to take. Let me talk about them. And in many cases, the work we've done, you know, is too new, so we
don't have really solid results, but I'll talk about the multimedia biography case. So in that case, with 12
people, we certainly -- first of all, we showed that we did have an efficient -- relatively efficient
production process. And then in the 12 cases, we saw lots of anecdotal evidence in the -- we went and
videotaped the people when they looked at the first time the video, the world premiere of the story of
their life, and then we had them watch regularly, and then we came three months later and six months
later. And we did see lots of instances, which we counted in many cases, of people having enjoyment,
having engagement, connecting with their family. Let me give you one specific example. There's
another group in another continent in another university that has worked on a similar problem about
how to use multimedia to engage seniors in reminiscing. And they have argued publicly many times,
and I've been in the audience several times and have seethed. They have argued that you don't want to
do individual multimedia; you should do generic multimedia that simply gives you scenes of your city as
a way to encourage seniors to reminisce, because, they have said, it might make you feel bad. It might
bring back unhappy memories. So we actually have a count. For example, in the -- the 12 times 3
videos, we talk of people watching this of how many times people seem to smile, give verbal
explanations of delight or pleasure or whatever as opposed to the "Oh, my god; I'm so sad," et cetera, et
cetera, and it was something like 15-to-1 in terms of joyful moments. So we do have some data, but -but not nearly enough, and again, it's very, very hard to do this work. And as you, you know, as you
probably recognize as you're starting to do this work with tracking emotions and effecting emotions,
how did you do ecologically valued field research in a way that doesn't disrupt behavior, is ethically
sound? I don't know -- do you have to go through ethics boards? Do you have an ethics board at
Microsoft?
>>: We have a minor IRB process.
>> Ron Baecker: Okay, a minor IRB process. Well, we have a major IRB processes, and when we do the
right thing by not just doing work from my laboratory but collaborating with people at any of the
teaching hospitals, then we have to go through ethics at U of T and also at the teaching hospital. So I
know students who've been held up or months and months and months with IRBs, so yeah. Jonathan?
>>: Are the ethics issues about how people react to being involved, or is it PII-type issues? Because the
PII, both for the research but even more for deployment, to the degree that I started to get into this,
was a real problem. You'd start putting video in places where other people might walk into rooms, and,
you know, you're showing videos where the faces are -- it doesn't have the same effect, right, when the
faces are blurred out?
>> Ron Baecker: Yeah. Um, I'm not -- what were the two choices again [laughter] in your question?
>>: Well, in research and in actual deployment. Also one of them is whether things will make people
feel depressed and so forth. The other is just revealing the fact that you're recording information.
>> Ron Baecker: Well there ->>: I was wondering what you were going to get to ->> Ron Baecker: Well, we're running into -- we're running into both. But for the moment, we're dealing
with research obstacles, and they're concerned about confidentiality and concerned about privacy, et
cetera, et cetera, et cetera. And the issue of, for example, video surveillance of seniors, what people at
Intel have reported that I've heard them report publicly is that very often, like if you want to, in so-called
smart home technology which I think has been a bit oversold, and I don't think -- I don't think there are a
lot of smart homes yet, which is why in the second part of my talk, I talked about little aspects of a smart
home, like monitoring water flow or possible falls, not an entire smart home. What people at Intel have
said to me and said publicly is that they've discovered that the senior citizens in whose homes
technology might be deployed are much less worried about invasions of their privacy than their adult
kids are worried and so. But there are absolutely critical issues here, and -- and as well as how do you
decide when something works well enough to deploy it? You know, pharmaceuticals, before they
deploy a drug -- and that's not that they don't have huge -- they don't kill lots of people nonetheless, but
they still first try it on a few dozen, and then they try it on a hundred, and then they try it on a thousand
and then try it on hundreds of thousands before. So what's going to happen with technology like this
when -- when the FDA or whatever wants -- starts to realize these are medical instruments that affect?
Will we ever be able to deploy anything because we'll never get to try this on thousands of people? I
mean, maybe social media you can, but not, you know, or let's say any of the devices that I saw today,
the video proxies, let's say. Let's say Microsoft decides to commercialize those or finds somebody to do
it. How are you going to run this on an actual randomized trial with thousands or tens of thousands of
people before deploying it? So this leads to interesting kinds of ethical and practical problems. And
anyway, other questions? Comments? Someone out there? Yes.
>>: So with the life stories thing, how does the multimedia production improve on or is it different from
just having photo albums?
>> Ron Baecker: We haven't done that study, and it would be interesting to do this, because we got
very little money and then never went for follow-on money. I believe or my speculation is that because
we're used to sitting around a TV screen and because more people can sit around it than around a photo
album, there's something -- there's an increased potential there, although the disadvantages is it's
costlier. And I mean, I'm not saying you should do one or the other. I think people should use
everything, but the -- and then the other thing is I think the -- I think where we've only scratched the
surface is the potential of social media and the Internet to help you gather material from families
dispersed over the world, around the world, and thus have a richer experience, including, of course, a
dynamic one with video and not just with photo albums. But, you know, I wish my parents had saved
more photo albums, so. Yeah.
>>: So going back to your point about multidisciplinary involvement and collaboration and to your
question about research that shows the effectiveness, I feel that that literature does exist outside of
HCI, you know, in speech pathology or rehab medicine or cognitive rehabilitation. I guess, having now
worked on kind of both sides of the coin, I don't see a lot of that collaboration that I would expect to
see. I see, kind of, HCI people doing their work, and then the people in health or education doing some
of the same work, but I'm just wondering if you know of examples where these kind of collaborations
are starting to take place more.
>> Ron Baecker: Well, I mean, we're -- I mean yeah, there are certainly examples. I mean, I think of on
the West Coast, Oregon Health and Sciences University, a group under Jeff Kaye, I think, really has a
superb collection of, sort of, technological and medical expertise. Alex Mihailidis' group does it. We try,
but it's -- it's -- it's hard to get people to often make the commitment to, you know, because if you're,
you know, if you're an assistant professor of neurology or something, you probably have to be pretty
courageous to say, "Okay, I'm going to go and spend a significant amount of our research time working
on some crazy new stuff with technology" as opposed to doing the kind of things that you know others
in the field have done before, so it's -- there are disciplinary boundaries. There are cultural differences
between the disciplines in terms of how the shared knowledge -- Jonathan has written a lot on this. I
mean, in CHI, you know, we go into these shootouts once every three months in which we know 80
percent of the papers will be knocked out, or maybe if it's CSCW, a more enlightened process now, only
70 percent, whereas most medical disciplines use journals for that kind of shootout and say, "Okay, we
want you all to come to the meeting of the Alzheimer's Association, and, you know, you'll submit
abstracts and as long as an abstract seems moderately coherent and on topic, we'll accept it." So one of
the reasons I'm editing this series for Morgan and Claypool is my hope is to bridge these disciplinary
chasms and get people across it. And the most recent one that's about to come out by Julie Kientz
locally and Jillian Hayes and Gregory Abowd, and Matthew -- I forget his last name -- in Boston, really,
looks to me, speaks to technology people and autism people and draws on the literature from all these
disciplines. And even though literature is in its own silos. I mean, there's the ACM digital library, and
there's PubMed and there's -- so it's really hard to bridge those.
>>: I feel like the field of health informatics is changing that a lot, though. That's where I'm seeing a lot
of possibilities. And doctors are very willing to work with health informatics students, and they're very
well trained in HCI.
>> Meredith Ringel Morris: Well, thank you all for coming. And Ron will stick around afterwards in case
anyone wants to come chat in a more one-on one environment and kind of hang out. Thank you.
[applause]
>> Ron Baecker: If anyone is still out there and not watching the Red Sox and the Tigers, send in a
question.