>> Eric Horvitz: It is an honor today to have Dr. Walter Greenleaf visiting us at
Microsoft Research. Walter Greenleaf got his PhD in the area of neural and behavioral
sciences at Stanford University, where I actually met him years ago, and Jerron as well
has known him for many years. His current research focuses on product development
efforts with use of simulation technology to improve neurological, cognitive and physical
rehabilitation. He has over the years jumped from academic scholarship into the world of
startups in this space. He is currently the founder of In World Solutions, a company that
actually specializes in using immersive technologies, augmented reality, virtual worlds
for behavioral healthcare. He also is CEO for Virtually Better, a company that helps
virtual environments for the treatments of phobias, anxiety disorders, PTSD and even
treating addictions. He is a pioneer in an area that was called, subfields called virtual
reality in medicine or health care, lots of technologies and ideas that came out of that and
even a conference or set of workshops. He is currently also a scientific advisor and
proposal reviewer for the US Public health service, the NIH, NASA…
>> Walter Greenleaf: Stop, stop.
>> Eric Horvitz: Well, lots of advisory roles and he has been in the media and he has
been communicating about the possibilities here for a long time. Walter, go for it.
[applause]
>> Walter Greenleaf: Thanks Eric. The minutia is too much. I really excited to be here
and one of the reasons I am excited to be here is that I think that Microsoft is in a position
because of some of the technology you guys are promulgating, to really make a
difference in healthcare and in the part of healthcare that I have been working so hard on.
And it is an important area. It is sort of a part of medicine that has been a little bit by the
wayside, occupational therapy, physical therapy, psychiatry, psychology; they have all
been under technologized over the years, for many reasons which I won't go into here.
But with the Xbox, the Kinect system, with some of the virtual world technology you
guys are pioneering, it is really making a big difference in my field. And I will go into
detail about that. But I do have a favor to ask of my audience. There is a substantial
amount of material that is interesting to cover and there's also enough time to cover it.
But it is going to be a little drab and boring if it is just me giving a spiel, so I am going to
say some things, but I want you guys to participate too. Give me some questions.
Object, object fiercely, criticize, comment, expand upon so that our audience is not just
listening to me give this spiel.
Okay, let's start. Here are the four main points that I'm going to try to cover. It is not
just, what I am going to talk about is not just something that recently sprung out of a
clamshell; it is something that has been founded on more than 25 years of foundational
research. And I will talk about some of the data, the FRMI studies, the longitudinal and
latitudinal studies that have been done, but there is a foundation of research. I also want
to talk about why it's an inflection point right now that the changes in technology and
changes of cost, more importantly is making this available to people both domestically
and worldwide. It is part of the breakthroughs that have been going on in telemedicine. I
will give some examples of how virtual environments are being used for a whole variety
of things, not just in behavioral medicine, but in physical medicine and as a disability
solution. And then I will talk about some of the things that are just around the corner.
Some of which are being pioneered right here At Microsoft Research. Actually a number
of the things--I give this spiel to other people and probably at least five of the seven
things I'm talking about that are just around the corner are being pioneered here at
Microsoft Research.
As you guys know virtual environments have been around for a while and they are
predicated on some foundational technology that came out of research labs more than 30
years ago, with general purpose simulators being a recent invention. They used to be
kind of a primary focus of virtual environments to be focused on one specific thing like a
flight simulator. But this gentleman here, Jerron, was one of the first guys to promulgate
the idea of having a general purpose simulation. Now this was back in the days of
computers being, to do the simulation being $400,000-$500,000 on a silicon graphics
machine. And Jerron I think one of the first times I think you elucidated the idea of using
a virtual environment to do programming was in this article from Scientific American
with the data glove using, to do coding.
But this concept of having a general purpose simulation was very important and it sprung
a number of people, including myself, getting involved. And for me it's been sort of an
exciting journey. Here's a picture of me back when I was in my late 20s I guess, when I
started on this endeavor to get involved in using virtual environments. What excited me
was the idea of using synthetic environments for stroke rehabilitation. I had known a
number of people who had had family members who had suffered a stroke and I really
thought the virtual environment would be a fantastic way for us to be able to help manage
the rehabilitation process. And finally with the advent of the technology that is present
now we are seeing this technology coming out and being used not just in the laboratory
but the technology is escaping the laboratory and making it into clinical care.
Of course, in medicine one of the areas where virtual environments have made a
substantial difference in our starting to become part of the standard of care, is in the 3-D
visualization, preoperative planning and data fusion in the middle of a surgical procedure
where a CAT scan or MRI image may be reviewed and used to plan a surgical operation
and then overlaid upon the operating field to help guide someone. It is also used for
training. So that surgeons can use the dexterous movements of new cools to be able to do
a complex procedure. And also now we are starting to see sort of a separation between
the end factor with the human and the surgical tool with robotic assisted surgery like the
da Vinci system.
Some people argue that there is a hype cycle that goes on with emerging technologies and
they would suggest that virtual reality is right through the trough of disillusionment and it
is starting to come out on the other edge to what is called the scope of enlightenment and
the plateau of productivity. I am not sure where virtual reality really falls on this, but
there sure were a lot of really bad movies back in the early ‘90s about virtual
environments.
>>: So right now cloud computing is the most overhyped thing.
>> Walter Greenleaf: That's what these people who have made this graph would, I guess
it's the Gartner Group, [laughter] well yeah, and e-book readers. You know, this is all in
augmented reality is in the advancing slope of being overhyped.
>>: Oh I see.
>> Walter Greenleaf: My point is…
>>: It doesn't help but that the 2009 picture, how much that picture has changed just
[inaudible] if you dangle on the graph, things are all different now.
>>: Tablets would seem to be main stream at this point not due to us unfortunately. It is
interesting. It's a cool slide. Where did you get--Gartner, huh, amazing.
>>: It's two years old now so I think it…
>> Walter Greenleaf: You know let's for the sake of discussion white out all of the
specific text here [laughter] and just say that at some point virtual environments are on
this curve and I would like to think that the phrase is no longer, just like artificial
intelligence at one point was a sort of an overhyped phrase, I think the phrase is now a
useful phrase. When I talk to clinicians about using a virtual environment as part of the
therapeutic process, they no longer look at me as if I'm crazy, and there is actually, we
will go into this later. But there are several international societies of cyber psychology
and society for international virtual rehabilitation. There are a lot of clinical journals and
research groups. So the phrase has made the transition from being a sort of an overhyped
phrase, a hyped concept. It spawned a number of lousy movies and a few bad television
shows. But now it is on the other side. We are starting to see it as part of the tools that
we use for clinical care.
Right now some of the virtual environments that are out there are incredibly realistic.
But they are missing something very important that will go into. They can be very photo
realistic with real-world physics and adequate sensory immersion but one of the
pioneering things that Microsoft Research is part of right now is the avatars don't often
have the right nonverbal behavior, facial expressions, body language and movements. In
order to seem real, the avatar--you guys have seen this in Second Life, I'm sure, where
everybody is sort of walking around like this and they have pretty blank expressions. For
my purposes for the use of, in clinical care, we need to have more natural body language,
natural facial expressions. So even though there have been great breakthroughs in VOIP
and better graphic processors, broadband, movement tracking, low-cost computers, all of
this has really helped advance the field but some of the late breaking things that are just
coming out now that are incredibly important are the use of nonverbal behaviors and
gestures as part of the experience.
Again to move along, there has been a foundation of more than 25 years of clinical
research, but just now this technology is escaping the lab and making it into the clinical
area. Let me talk a little bit from a neuroscience perspective. In order to change a
behavior, you really need to change your brain. This is often, you will hear people
talking about years of therapy, and it can take years, because right now--previously we
really haven't had the tools to break down that change into small finite steps. Also,
repetition is required and repetition can be boring. It is also important to involve the
brain's reward systems with positive feedback in order to engender the change. And that
is one of the things that has been lacking in both behavioral therapy and also in physical
therapy and occupational therapy, getting that feedback loop and enclosing it with a
rewards system, breaking things down into small steps. So a lot of what I am going to be
talking about is going to involve how do we engage the human cognitively to participate
in their own change and rehabilitation, in a way that is not boring, that activates the
reward systems and allows someone to do something that is repetitive but with a little bit
of change, and ability to monitor progress and get feedback. And I will show you how
we do that.
It is really critical to engage the brain's reward systems and here is a book that I just
found out about; I have not read it yet, but I like the title, the Compass of Pleasure, How
Our Brains Make Fatty Foods, Orgasm, Exercise, Drugs, Generosity, which is a social
experience, Alcohol, Learning, Gambling Feel so Good. These are the sort of things in a
gaming environment that we can leverage the brain's reward systems to motivate us to
change how our brains work. And that is through the basic principle of using virtual
environment in a therapeutic manner, is harnessing of the brains systems and the reward
systems in particular to help engender change.
Here are some examples, I am going to start first with behavioral health but later I will
talk about physical medicine. Here are some specific examples of how virtual
environments are being used clinically right now in behavioral medicine. We treat
phobias and anxiety disorders, posttraumatic stress, developmental disabilities such as
autism and Asperger's, conduct disorders, drug and alcohol abuse, anger, eating,
impulsive disorders; the list goes on, and I will show you how we do it. Let's skip a little
bit here. Here are just a few quick clips of some of the virtual environments that we are
using therapeutically right now. You will notice that there is a whole variety of graphical
resolution. There is no coherent platform right now and no standards for the virtual
environments that are used in the behavioral environment. It is just sort of all over the
map. Some are very cartoony, some are very photo realistic, but it seems to us, the
researchers in this field, that the degree of realism per se, the graphical realism is not as
important as the cognitive engagement, I will go into what is involved with that.
So the examples here, there is a virtual liquor store that we use of helping people with
problems with alcohol and substance abuse get familiar with what otherwise might
trigger a craving and learn to master the response to that craving. I will show you
examples how we use that in a social environment. Here is a World Trade Center virtual
environment that is used to help the still large number of people that are suffering from
posttraumatic stress from living near the World Trade Center or watching it on TV.
There are still a large number of people that are suffering from the cognitive problem of
that and years later you can still get help and make improvements. Actually I recently
talked to someone who was a Vietnam vet who had been suffering from posttraumatic
stress for I guess more than 40 years now. Recently he went into a center that used a
virtual Vietnam to help him overcome his posttraumatic stress and despite years of
therapy previously it was being able to use that virtual environment that helped him
overcome his posttraumatic stress. I will go into the details about how that is done.
There is another example here of using a virtual environment for what we call stress
inoculation. The Center for Disease Control is using virtual environments to help prepare
their first responders for disease outbreaks so that when they go to, if they have ever been
camping before and all at once they are of in an African village and learning to deal with
the jungle, you can prepare them cognitively so they don't have a stress reaction. Another
example is using a virtual environment to help with fear of flying. For fear of public
speaking we can use a virtual environment to reinforce-- it's amazing how well this
works. To have an audience that actually smiles and claps when you are speaking, gives
you positive reinforcement, can help you overcome any hesitation; this is your cue,
audience. So the point is we can use virtual environment to change how you respond to a
specific situation that otherwise might be very stressful.
And again we use a virtual environment to help with the social cues that you need to
master if you are having a substance or addiction issue like nicotine or alcohol or other
addictive substances. In some cases we use actual green screened individuals to put into,
as the avatars, mostly because we need to micro-expressions on facial expressions to do
this. But in other cases we just use very graphic avatars. And I will show you the
distinction of why we do that. So that is sort of my smorgasbord in behavioral medicine
of some of the areas.
>>: I have a question. In the previous slide what was the application of the people
around the pizza boxes?
>> Walter Greenleaf: That is for people that…
>>: The other ones I get public speaking, flying, what is that one?
>> Walter Greenleaf: That one is for, it is a party scene. And a clinician can use it for
someone who is having trouble walking into a party, being offered a cigarette, social
phobia. It can also be used for just if you have nicotine issues, alcohol issues. Often
what happens to people is that they are very good at maybe at home not reaching for a
cigarette or reaching for a beer, but they will go out to a social scene and sensory clues in
the social clues will trigger the behavior that they are trying to avoid. So what the
clinician will do they will have the person practice going to a party, and it works. You
might not think it does, but it really does. You go to a virtual environment, you practice
saying no to the beer that's offered to you. You do that over and over and over again, and
eventually you get it down. Why can't you just do that using your imagination in the
clinician's office where you are sitting there and saying okay, now imagine that you are
going into a bar and somebody is offering you a beer. What are you going to do? You
might think okay, I am not going to do it. Again unless we engage the brain’s--the more
sensory systems we can involve, the better the response we get. And it can make the
difference between years of therapy, years of counseling, versus just a few sessions. And
to that effort we do have some olfactory stimulation systems that really seem to help a lot
for some of the things that we are doing.
I am going to pause for a second because I have given 10 minutes worth of the spiel here
to say hey, what do you guys think so far? Is this lining up for you any questions at this
point? Okay, all right. Let's push on then. I break down what is going on right now in
therapeutic healthcare with virtual environments into sort of the four zones. There are the
very immersive virtual environments where we use an HMD. You may have a tactile and
olfactory stimulation that goes on hand-in-hand with that. These are typically used for
the posttraumatic stress, the phobias and anxiety disorders. Another one is the interactive
social virtual environments. These can be done with a flat screen. They can be done
over a network. But they emphasize the social interaction between individuals and this is
where we need to have the facial expressions and body language et cetera in order to have
it work. These environments are used for addictions, for learning social skills, dealing
with problems such as autism and Asperger's, anger management, impulsive disorders,
attention disorders like ADHD.
The next zones are the zones for health, for wellness and prevention, where you are
trying to learn better eating habits, better exercise habits, smoking cessation and et cetera.
These often are done on the flat screen game like environments which we described, but
they have a completely different paradigm. The condition is not involved; these are often
things that can be done on your own with the help of autonomous coach or an algorithm
to help you manage your progress in the therapeutic procedure. And the final area is
mostly dichotomized because it uses other sensors. And this is the zone of physical
rehabilitation and injury recovery for stroke, traumatic brain injury, et cetera. Has
anybody in this room ever gone through physical therapy from an injury? Okay it is
really boring, isn't it? You have to have the personality of an athlete to do it. And one of
the things that is coming through right now is we are using virtual environments to make
it both social, you can do your rehabilitation over a network with other people, and see
your progress and make it engaging. So no longer does it have to be something where it
is painful and boring, it can be painful and interesting.
Let me show you some examples of the immersive treatments that we are doing for
posttraumatic stress et cetera. I should mention that our collaborators which I think folks
here at Microsoft Research are probably doing some work with, Mark Bullis at IC2 is
doing a lot of work in this area and Skip Rizzo is one of the early promulgators of using
virtual environments for a variety of problems such as posttraumatic stress and cognitive
rehabilitation. So I will go into detail a little bit on this one. There are a huge number of
unfortunately servicemen and women coming back from extended tours of duty in
combat zones and they may have a traumatic experience that is acute, where they are
driving in the Humvee and there is an explosion and they see something very, very
disturbing like a friend being killed. But there is also just a chronic stress of being in a
war zone, constantly worrying about what is going to happen, being hyper vigilant for
years. And this results in long-term disabilities. And I will give other examples from
earthquake posttraumatic stress et cetera but we use combat as an example here. So
consequence might be you come back, you are no longer in the military, you are driving
on I-405, yet you see some trash on the side of the road and it causes a panic response.
Or you go under a bridge and you are worried somebody is going to drop a grenade down
on you. It is very hard to unlearn this learned fear response. And it can haunt you for
years.
In the example I gave earlier about someone from Vietnam suffering from problems 40
years later, yet, there is a way to treat it. Basically you have learned a limiting system
fear response that is triggered by visual and auditory and olfactory stimuli and it is a
survival response, a mammal to have that sort of instinctual reaction to something that
almost killed you before. To unlearn it, you need to have, to go back, stimulate those
parts of the brain that have learned that response. You do it in a controlled way so that
what we call our higher functions start kicking in and saying oh, that piece of trash on the
side of the road, I know what that is. I am no longer back in a war zone; it is just trash. It
is very hard to do that if you're just using your imagination. Your brain, those are painful
memories, painful images. And besides not all of us are very good at reimagining things
anyhow, only a handful of people can really have the vivid visual and olfactory auditory
memories anyhow. And your brain doesn't want to go there if you have had trauma over
it. We can take people there in a graduated manner, do the exposure and immersion,
cognitive immersion to habituate that feared response and it works.
So here are some clips of the system being used. Currently we have recently deployed in
10 Air Force bases and I think we have probably 30 systems out at VA hospitals are now.
It is a great success story. A clinician will work with a patient. They use an HMD. And
they go through, and I will show you some video clips of this, they go through in a
graduated manner. You might start with something that is non-traumatic at first and just
get them used to the system, returning to the scene of the stress. And the counselor helps
them manage their anxiety. They practice breathing and relaxation techniques. It may
not take several sessions; it may take 10 sessions. But after a while, the brain gradually
habituates and you can recover from posttraumatic stress. And again the same paradigm
can map to fear of flying, fear of heights, early trauma of childhood abuse, et cetera.
There are lots of stresses out there that cause those problems for people later in life. And
the same paradigm can be used to help them overcome them. Jerron?
>> Jerron: No go-ahead.
>> Walter Greenleaf: Let me show you a clip on how we do it. This is from an Xbox
game actually. [video begins] This is from Skip Rizzo's virtual environment, and a
clinician might just start out with this or a more neutral scene where they are just driving
along in a Humvee but nothing necessarily traumatic happens other than just been reexposed to the scenes of war.
>>: The basic principle here is exposure with some notion of relaxation or distant seeing
from the negative aspects again and again and again [inaudible] is a healthy thing to do to
get beyond the fear of a single incident or a scary incident? Is that the principal?
>> Walter Greenleaf: Yeah, you trying to reassociate those sights and sounds and smells
with a more relaxed situation and you are trying to let people know that they can think
about these things, be exposed to them, and not have the panic response. So in a way you
can just simply describe it as habituation, it is called exposure therapy.
>>: So as you expose, how are you relaxing?
>> Walter Greenleaf: Well the clinician…
>>: This is not exactly relaxing [inaudible]
>> Walter Greenleaf: Right. Well the clinician is there with you in the room. You know
it is not real. At any point you can take off the HMD. The clinician is instructing you to
relax, to remember that it is not real. The clinician is having you practice breathing. If
they see that you are getting a little bit overstressed, they will say, why don't we stop for
a moment and think about this. And it works. The biggest problem we have is actually
having people volunteering to be part of this. This is something they are very fearful of.
And unfortunately in the military, too, there is a huge stigma about posttraumatic stress.
It is getting better but my understanding from talking to clinicians is that once they get
someone to say I have symptoms of posttraumatic stress. I am having suicidal thoughts.
I am having a lot of stress and anxiety. There are processes like this we can put them
through to help get them back. But because they might lose their job, because it is not
macho to admit that you're having this, it is hard to get people to participate. And that
brings up another point I want to make which is one of the advantages of using avatarbased therapy is that we can do anonymous counseling, and anonymous help. So maybe
help people start taking those first few steps where they don't actually have to go to the
clinic, where they can show up as a virtual avatar. And it is not just for posttraumatic
stress. If you have a sexual addiction, if you have a drug or alcohol or substance abuse
problem, you might want to show up for your sessions to start the process as an
anonymous avatar. And I will go into that a little bit later.
>>: I was just going to mention when these applications of treating PTSD with virtual
reality first were proposed, I was one of the biggest skeptics. I thought it was just too
cute and gimmicky and what persuaded me about it were really just the clinical results. It
was one of those things where I really came to it with a biased feeling that this couldn't
possibly be for real and it turned out to be so, I was actually chastened.
>> Walter Greenleaf: Well I will show you some of the data if we have time. It actually
is pretty amazing. You can see a nice stepwise improvement that you don't see when you
look at Wait List and Controls or you look at other ways of doing it. Now I should point
out for some of these things like fear of flying, exposure therapy has been done for years.
But they would actually have to take someone to the airport. Like for fear of heights they
would actually have to take someone to a bridge, or the elevator and they would have to
go with them, very expensive, very time-consuming. And for combat or an earthquake
type of posttraumatic stress, it is very hard to take someone to re-experience that in a
controlled manner. So the virtual environments give us the ability to do something costeffectively, possibly over the internet, and in a controlled manner with measurable data.
And that is one of the keys to some of these things I am going to be talking about too is
not only are we getting new tools for therapy, we are getting new tools for monitoring
and measuring progress. And one of the reasons cognitive behavioral therapy, physical
therapy and occupational therapy have been sort of under technologized aspects of
medicine, is that they have not had good measurement tools. So one of the ultra
messages I would like to portray here is that the technology that is been worked on here
and other places is going to bring out a new ability to measure and index progress in a
part of medicine that has largely been analog and subjective. These rich environments
can be used to measure objectively things that so far have been subjective, but that is for
another part of the talk.
I am going to push on because I have a lot to go over and not a lot of time. So, very
quickly, research has shown that it is more effective than just plain imagine exposure. It
is as effective as the in vivo exposure and of course it is more cost-effective. You do not
have to have an exact replica of the environment to have a therapeutic effect. You just
need to have enough cues to stimulate that sort of anxiety response. But it does not
supplant the condition; it is just a tool that allows them to do things that otherwise are too
difficult or impossible to do.
I am going to skip ahead to some of the work on fear of flying. This is an area where
there has been a lot of research showing the clinical mechanism and validating it. And of
course now we are starting to have much more high-resolution airports that we can take
people to put them through. Here is an example of one of the studies on fear of flying.
On the left hand of the graph you see virtual reality exposure therapy both pre-treatment,
there is an index of fear of flying. That is a general scale, the post successions of
treatment and then the six months follow-up. So with the virtual reality exposure you see
a nice stepwise decline. Comparable to the standard exposure, but with the Wait List and
Control they don't get better over time; they actually get worse. And this is a typical
pattern we see for other areas where there has been research. Here is a study on Virtual
Rack which was done at Camp Pendleton with the Marines that are coming back. Same
type of progress that pretreatment, high levels of measurements of posttraumatic stress
including indexes of anxiety and depression. Post treatment there is a decrease and then
with a three-month follow-up, there is even more of a decrease. And it shows up not just
in the advert symptoms of posttraumatic stress, but other measurements of anxiety and
depression. We have looked at where is this going on in the brain, looked at it with
interest with fMRI that mapped to neural plastic changes. And we can track the changes
that go on looking at the MIgda activation, pre-and post treatment to using this approach.
So this is not just a subjective index of improvement that we are getting from self-reports
that we are getting from the patients going through but we are getting a subjective
measurable changes correlating with the improvement.
And there've been a number of meta-analysis that has looked at the value of this for
treating exposure therapy. I should mention though that despite the clinical validation
studies have been done, despite the almost 20 year’s worth of research in laboratories on
this, it is just now making the transition from the universe of laboratory to clinical care.
There have been barriers to entry on a market basis. A lot of which has been cost. And
that is one of the reasons I am excited to be talking to this group is that the Xbox, the
Kinect and some of the other technologies that you guys are pioneering are making this
affordable and that will make the difference. If it is just an expensive way of having a
therapeutic approach, the clinicians won't be able to take the time or afford to do it but
making it affordable makes all the difference.
Okay let's talk about just generally. I was talking about one specific area, posttraumatic
stress. But let's talk about a general cognitive behavioral therapy. I won't ask how many
people in the audience have had to go through counseling or psychiatric or psychological
care, but I will point out that for most people it is a long drawn out process that is often
very low yield. I am sure the clinicians out there that will probably take issue with it but
it takes time and it doesn't always work. And part of it is, in my opinion, is that you have
to recall past events. You have to convey them verbally to the clinician you are working
with. You have to do some role-playing and imaginative work on how you are going to
handle the situation. Maybe you are dealing with getting into a fight with your spouse all
the time or maybe you are dealing with an anger issue that you have. Or maybe you are
dealing with an addiction that you have. It is very hard to just talk about these things.
But if we use the virtual environment as scaffolding for these discussions, it can make a
big difference.
I will show you how we use it. Again with what goes on with just talk therapy, it is very
hard to change the neural system behind the behavior because you are not activating
enough parts of the brain, at least that is my biased perspective on it. I feel that in order
to change the brain system, you have to activate it. And if you're just using talk therapy
alone, you are not engaging the patient and activating enough parts of the brain to do it.
So what we are using right now, interactive virtual environments in a variety of
situations. I will show you a few examples. Now in this case we are stepping away from
using the fully immersive, with 3-D audio and 3-D visual and sometimes olfactory
stimulus to just using a standard keyboard and visual display. Despite the less immersion
we are finding that it works very well especially with younger patients. For example, we
are using it right now at a clinic in Newark New Jersey with kids who are 14, 15 years
old, both girls and boys who are in a locked residential care facility because of their
behavioral problems, conduct disorders, violence. These are kids that are there because
people believe that they don't have to spend o the rest of their lives in prison. But unless
they learn how to control their impulses, these are kids who have killed other kids, or
other people. And these are kids who burn down buildings and I won't go into lists but
there is hope. We are using the virtual environments and the clinicians tell me that it is
working to for them to learn to control their impulses and modify their behavior. So in
this case we use a 3-D interactive virtual environment to do role-playing and interaction.
So the clinician may take on the role of another teen and they may discuss or do some
role-playing as to how to deal with a very frustrating situation. And they will practice
how to respond to it. Now the clinicians tell me that if they just sit down with the kid and
try to do some role-playing in two chairs sitting and talking, these kids won't stand for it.
They will pick up the chair and throw it across the room. They will not pay attention.
They get maybe 5 minutes at most of engagement of these kids. But they will sit down
and spend an hour in the virtual environment. Part of it is because they are little bit more
anonymous. Part of it is that they don't have to worry about what their body is doing.
Part of it is that the clinician is no longer the man, an older adult; he's taken the role of a
teenager.
And so we have had great progress in having these kids learn to control their impulses.
That is just one example. I will give you some examples further downstream in my talk
about how it is being used for kids with autism and Asperger's to learn social skills. But
these flat screen interactive virtual environments have proven to be some breakthrough
ways of helping reach people with cognitive behavioral problems that otherwise cannot
be reached through therapy.
The other advantage to using a virtual environment for cognitive behavioral therapy and
other forms of rehabilitation is you can create a community. Often the therapy is in
isolation, but if you are doing family therapy if you are doing things that involve, for
example, a service man our woman coming back with an amputation, what we are doing
right now is we are working with the Air Force to help people who are deployed stay in
touch with their families through a virtual environment where it is not just
videoconferencing where your are doing postage stamp look at somebody else and you're
talking with them. If you have a five-year-old kid this videoconferencing only works for
a few minutes. They want to do something with you. In a virtual environment you can
go for a ride, you can go for a walk and you can go play a game or you can go for a
scavenger hunt. Then later if there is an injury and dad is returning and he is making his
progress through the health care system before he is coming back, a clinician can meet
with the kids in the family and say look mom’s coming back, dad’s coming back, he's
going to be a little bit different. He had a brain injury or he lost an arm, prepare them
psychologically and also prepare the serviceman or woman of what is going to be like to
be--we can build a community and we can help people make transitions through the use
of the virtual environment. I'm going to pause for a second because I have been talking
uninterrupted for a while, what do you guys think so far? Any questions?
>>: One question I have is in counseling have people thought about, I am curious about
various studies that integrate the fidelity in characteristic ways and see what the loser
gained [inaudible] interactive flat screen presentation of [inaudible] 3-D world and you
go to really beautifully rendered sets of pages of scenes, how much does that get you to
[inaudible] versus Norman Rockwell homecomings for service members [inaudible] there
must be some effect [inaudible]
>> Walter Greenleaf: I will tell you a couple of thoughts. I think there have been some
studies mostly looking at the difference of HMDs with 3-D sound versus flat screen. But
the need has been so acute and the funding has been so miniscule that there really hasn't
been a lot of research in this area. I can tell you some of the clinical impressions I get
from people who use the virtual environments. For things I do like the phobias, the
posttraumatic stress, a little bit of a clue goes a long way. People's brains take them there
if they just get a few little stimulus. It doesn't matter how high resolution it is. For the
adults, I mean older adults, let's say, who are using virtual environments, they haven't
spent a lot of time playing video games. At least not most of them, so it takes him a little
while to get familiar with it. But there the fidelity doesn't make a big difference. But for
some of the teenagers that we are working with, if you put them in a bogus virtual
environment which is just like lame like two-year-old technology they don't want to
participate because it's old. So I think in a way you are asking two questions, Eric. One
is what is the level of fidelity we need to get a clinical effect? And two is sort of the
social context for doing this. What is going to make somebody want to participate and
comply with participating in it.
>>: [inaudible] I’m not sure if I should ask this or even bring it up, but it seems kind of
scary thinking that there is a positive therapeutic effect in going about the [inaudible]
version of warfare [inaudible] modern teenager [inaudible]
>> Walter Greenleaf: It's important question and I am glad you brought it up. For years
some of the videogame people have been saying, you know, these games that have you
stabbing, shooting and blowing up things all day long, some kids will spend 12 hours a
day doing this, doesn't transfer over. They don't learn from it. But now that we are
starting to have some healthy games come out that are using these virtual environments,
some of the video companies are saying, you know, it can transfer over in some contexts.
You're absolutely right. I think that we do reprogram our brains all the time by how we
spend our time and what the context is. And I would like to see that kept in mind as
games are designed. I think that that would be important. My role is to try to leverage all
this game technology and all this military simulation technology and move it over for
therapeutic purposes. But if you're asking me do I think some negative things can happen
because of being immersed in violent video games, sure. Yes, it does happen. Yes?
>>: You see a future for Xbox is that there is a new conferencing app called [inaudible]
avatar. Playing in this…
>> Walter Greenleaf: Avatar Kinect.
>>: You have to worry about the [inaudible] because they are all avatars.
>> Walter Greenleaf: Right. I have got that a little bit further down in the spiel here.
But yes, it is going to make a big difference in what we do.
>>: Especially if there's physical attributes to the therapy and may join in groups. That
might seem like it would help.
>> Walter Greenleaf: Absolutely. For group therapy, for networks therapy, sometimes
having an anonymous presence, especially--I'm finding that for the work groups that we
do for kids with autism and Asperger's who are learning social skills, somehow not
having to worry about what your body is doing, somehow not having to make eye
contact, that puts, and also having control of the world to a degree and complexity allows
the clinician or counselors working with them to staircase them up into learning some
social skills otherwise that they might not be able to do. There are some really great
things we can do with the technology that you just described. For example part of the
miss-development in the social brain that often happens with autism and Asperger's is the
ability to recognize correctly facial expressions and other nonverbal clues such as body
language and gestures. We can exaggerate those in a virtual environment and have
someone practice recognizing them until they get it. And then we can reduce the
exaggeration until they can recognize it with a normal level of realism. So we can finally
take social interaction and adjust the degree of fidelity for therapeutic purposes. And it is
really going to make a big difference, and the ability to measure movements, well that
has great implications not just for the therapeutic aspects and behavior but also for stroke
rehabilitation and traumatic brain injury rehabilitation et cetera. I will get to that in a
moment too.
Let's keep moving unless there are any other questions right now. Usually at this point I
would invite my colleague Yvonna Steigman who is out there doing a lot of clinical care
research right now to join us. This is, but she is not available right now. But this is the
user interface that we have developed for using one virtual environment for online
interactive therapy. The avatar is pretty good. If she was moving around and talking
with us and giving a tour right now you would see that she has this natural bio kinetics
and movement is pretty natural. There are great facial expressions which are linked to
the tone of her voice. There is lip-synching so when she talks her lips move and the
facial expression maps some aspects to the tone of her voice. We have buttons the
clinician can use to not just give certain gestures but to convey a motion. There is
another set of gestures that we can pull up for different emotions like happiness, sadness
et cetera. So Yvonna uses this as a therapeutic environment either co-located where she
is sitting on one computer and the person she is working with the sitting on another, or
for group therapy where they are all sitting together on computers in one room but
meeting in the same virtual environment, or distributed across a distance. An example is
a group of social skill work with for kids with Asperger's, where they will meet
Wednesday nights at the clinic and they will all participate in the virtual environment at
the clinic. But then on Thursday night and Friday night they meet from their homes and
practice the socials skills learned in a moderated group environment.
Part of the problem we have with this virtual environment right now is exactly what I
described. We use a button to say happiness, a button to say anger. So we convey the
emotions to the avatars here in a very binary manner without very much finesse. So one
of the things I am very excited about with the Kinect technology is to be able to map
facial expressions to our avatars. I can't wait until we can do that. We don't have it in a
clinical environment yet, but you have created the underlying technology for that.
So this is the virtual environment that we are using right now for substance abuse issues,
for social skills training, for anger management, for addressing issues of family therapy, a
whole variety. And I will show you some examples of that. Here is an example of using
this for family therapy, teaching somebody in a group therapy situation how to deal with
a family issue. Here is an example of [video begins]
>>: Okay [inaudible] are you ready to role-play? I will play your mother and you can
just be yourself, right?
>>: I think so.
>>: Okay let's all report to the house while I change my avatar to look more like
Debow’s mother.
>> Walter Greenleaf: The wallpaper is horrible in this example.
>>: So son, what did you want to talk about?
>>: I am not really sure, but I know that I let you and dad down and became a great
disappointment.
>> Walter Greenleaf: One advantage of doing some of the role-playing in a virtual
environment is you can capture the interaction and then review it for sort of a debriefing.
So that was an example of how they might have done some role-playing of what the son
is going to do to talk about it to his mom about a difficult subject. But later after doing
the role-playing you can save it on smart phone or on some other pad like device to
review and practice. If you are dealing with how to deal with a bully at school, or if you
are dealing with how to deal with a difficult coworker, you can do some role-playing and
then review it later. I will give some examples of some social skills work too.
[video begins again]
>>: Okay, thank you all for teleporting to the school.
>> Walter Greenleaf: Of this is from role-playing that we did with a social skills group
with kids with autism.
>>: And today your [inaudible] is inviting your friends to the movies. Remember how
we were learning earlier this morning in our circle about how to start and maintain a
conversation and how to end a conversation? So I would like to know who would like to
volunteer to practice those skills.
>>: I would.
>>: I would.
>>: I will.
>>: Okay, how about we have Latoya and Linda. If you guys could all, the rest of the
kids if you could all circle around and let them be the center so they can practice their
skills. And please remember that while we are practicing here that we are going to give
feedback to you and this is only to help you not to hurt or make fun of anybody. So I just
want you to know that we are all here to support you, okay? So go ahead, let's start a
conversation.
>> Latoya: Hey what's up?
>> Teacher: How about their body space? Remember we talked about body space?
>> Walter Greenleaf: Okay you get the idea. Go ahead Eric.
>> Eric Horvitz: Are people interacting live or if so, what are the controls they have?
>> Walter Greenleaf: It was that user interface that I showed you earlier. Moving the
avatars is by mouse and keyboard movement and that is why they look sort of jerky.
>> Eric Horvitz: So they are trained in navigation [inaudible]
>> Walter Greenleaf: Yeah, well with kids, you don't even have to tell them. They just
sit down and they know that the W in the S and the keyboard are for certain movements,
you know, old-school. So in that case, it was that paradigm that I described before where
they are all meeting at a clinic. They all have their own computer. They are meeting in a
virtual environment. And then later they can meet off-line from their home to practice
with the moderated sort of chat room way of practicing social skills.
>>: What type of platform are these examples running on?
>> Walter Greenleaf: That was from Olive.
>>: That was Olive, okay.
>> Walter Greenleaf: Since we are online interactive virtual environment it is 4 Terra,
out of San Mateo developed it; it was recently acquired by SAIC. We use it because it's
HIPPA compliant, which is very important for what we are doing. These become
medical records, which is another whole issue of discussion we should talk about. But it
is important to be able to, you know, keep the medical record confidential and the voice
is identifying for the individual. One could argue also that the body language, once we
start mapping back, and the facial expressions that is going to become uniquely
identifiable too. So you have to treat these as medical records even if they are in
anonymous avatar, if you are capturing the person's behavior, you are making a medical
record. So it is something to keep in mind.
Okay here is just one more quick example. I want to push onto some other things. This
is like using this for an anonymous AA meeting.
[video begins]
>>: Good morning everyone and welcome to the [inaudible]. My name is Barry and I
am an alcoholic and I chair this group.
>>: Good morning Barry.
>>: Hi Barry.
>>: Good morning Barry.
>> Barry: Good morning everyone. The focus of this meeting is to share our strengths
and help each other so we can solve our common problems with alcohol addiction. Let's
go around the room and introduce ourselves…
>> Walter Greenleaf: So it takes on the format of a standard clinical therapeutic
environment. It doesn't have to be the usual AA structure. But it allows people to A,
participate in a telemedicine manner and that is really important to keep costs down. And
B, for people who might be shy about talking about a psychological problem, it breaks
the barrier when they can show up as anonymous avatar and get help. It is going to open
up therapy to people who otherwise wouldn't through cost, distance barriers or their own
hesitation to take that first step. They will be able to start participating in the therapeutic
process.
I didn't bring statistics. I'm sure you guys have already heard about it about the percent
of mental illness problems in our culture. But it is legion. And as we are getting better
with dealing with disease and the physical aspects of aging and so on the cognitive, the
behavioral and especially in the stress involved world will start to become more and more
large. The technology that is here is just coming out in time. I think it is going to be
really important especially in a telemedicine manner to get it out to people. We will talk
about that later.
I will skip forward a little bit. We also use virtual environments for what we described as
refusal skills training, practicing saying no. So we have virtual environments for, you
know, someone comes up and asks you for your drink order, or you go into a liquor store,
we have--For a clinician to induce a craving for someone who has an addiction problem
is a big challenge, because you can't exactly show them marijuana. You can exactly
show them cocaine in a therapeutic environment; it is illegal. Yet you need in order to
get them to master their cravings, you need to stimulate that response. So we can do it
very effectively with virtual queues. Again the telemedicine model makes a big
difference. One of the things I am really excited about is behavioral medicine is really
devoid of a lot of objective data. It is a very subjective world right now with very
subjective measurement scales that are often just collected in an analog manner. Data
piles up in files but it is never really--algorithms aren't optimized. While using a virtual
environment in a telemedicine manner we can start collecting and anomatizing data at the
backend to start defining protocols and this is really important.
>>: That little thing you put on the left there that's a Kinect?
>> Walter Greenleaf: You're absolutely right. It is an early version of a Kinect. And the
snowman, who knows where that came from? Okay, this is an illustration that I did 20
years ago. It is just now coming out. This is a really important thing. And here's
something else I wanted to talk about. The tools that clinicians have right now in
behavioral medicine for measuring psychological state are limited. You may not have
taken some of these tests. You probably have taken intelligence tests; I'm sure the people
in this room have. Some of the psychological inventories like the MMPI test et cetera are
very old. The answers are out there on the internet. So if you are using them to screen
for employee issues et cetera, the answers can be found, people can break the system. So
there is a need to come up with some better ways of measuring mental status, mental state
that are ecologically correct and are variable. So what I want to do is use virtual
environments as a better way of doing psychological assessment. Instead of a subjective
assessment, to have some trials. You know, we have somebody be challenged. Maybe
we have them, if we are looking for an attention issue, attention deficit disorder, why
don't we have them go for virtual bicycle ride and have a bunch of distractions come up,
and how you respond to the distractions, and how long attend to them, how quick can you
get back on task? But do it in a functional manner, rather than the silly tests that we are
doing right now.
For more complex cognition executive functions, same thing, as we age we are going to
want--there is going to be more therapeutic options out there. But unless we have the
underlying data to optimize protocols, define the drug and therapeutic interventions, we
are not going to be able to refine the response. So this stuff is really important in terms of
improving the next generation of behavioral healthcare, because we will finally have
data, data that we have not had before.
Okay I am going to talk little bit about, there is a business model behind this. We can
show how it increases revenue; how it reduces cost, how there are a number of value
points for state and local governments. I think I remember reading recently that the state
of New Jersey has spent 2 1/2 billion dollars on mental healthcare services alone, in one
year. Mental health care and some of the subsidiary problems like violence and
addictions and so on is extremely expensive. So the state and local governments have an
interest in this. They are already spending a lot of money on not very effective
therapeutic programs such as prisons. But patients themselves have an interest because it
is less boring. There are better results. There is reduced stigma. There is a business
model for the clinicians and the practice owners. So later if you guys have questions
about the business model behind this I can go into that in detail too.
Another example of where we are using virtual environments this is a very beautiful
game environment that Michael created using for kids going through painful procedures
like recovering from burns or having a bone set et cetera. So we create an engaging
environment that is specifically designed to distract them during a painful procedure. It
is just sort of a modest example of how this technology can be applied to a particular
problem but it is one that is finally getting out there. And it is nice to have the graphical
tools to be able to finally do this. And in this case we have a controlled breathing that
goes along with the distraction. And in order to progress through the virtual environment
you have to regulate your breathing. And that helps distract them from the pain.
We are also using virtual environments for stress inoculation. This is a project we are
doing with the Centers for Disease Control, to help them train first responders how to
deal with going into an earthquake zone. I would also like to use the same environment
for the posttraumatic stress from earthquakes. As you guys know there is Haiti, there's
China, there is Japan with the tsunami. There is a huge amount of posttraumatic stress
that will go untreated unless we can come up with a cost-effective way to do it. So again
we use the virtual environments and with the same paradigm that I showed you before we
do the therapeutic process. But you can also prepare people who are likely to go through
this stressful process by having them go through it in a virtual environment and be
prepared cognitively for it. It helps reduce the risk of posttraumatic stress.
Here's another interesting application. This is training a clinician on how to present a
terminal diagnosis. You would be surprised, and Eric maybe you remember this from
your medical training, you would be surprised at how ill-prepared clinicians can be for
delivering bad news. And if they don't deliver it very well, it can be a horrible experience
for the patient. To make what otherwise could be a manageable issue become very
difficult to manage if the presentation of the problem is not handled well. So that we are
working with MD Anderson clinic, we have developed this system here for the clinicians
to practice how to deal, sort of like a flight simulator, a flight simulator for delivering a
terminal diagnosis. In this case we are using a green screen to deliver [video begins]
>>: Hi. It is nice to meet you. I hope my results are ready. I was expecting to see
someone else.
>> Walter Greenleaf: So the user is typing in commands here. I would love to use some
of the NLP work that has been done here et cetera to help refine this.
[video continues]
>>: I came alone. No one will be joining us.
>> Walter Greenleaf: So there are a variety of responses that can be engendered. And
angered response, a tearful response, a violent response, but the clinician can practice and
prepare themselves on how to do this in an empathic way. Some of the users have been
using this to teach something that is very important but very hard to train somebody on
which is how to be empathetic to a situation like this.
>>: [inaudible] there are two physicians in Seattle; they are the two NW. referrals for
Lou Gehrig's disease and they work together to break it to the patients slowly even when
they are both certain that. Lou Gehrig's is easy to diagnose, well were not sure here
exactly, but it's something that is not great. I will tell you what wanted to go to and so
they send them for a second opinion and they actually coordinate completely on the
phone behind the scenes on how to break the news. It is such a difficult diagnosis for the
family and they try to [inaudible] in the right places and so on.
>> Walter Greenleaf: Well that is good that they have worked it out. From what I hear
from my lot of the medical schools that often people just sort of get thrown out of there
without a real chance to learn how to do this difficult thing which is part of being a good
clinician.
>>: [inaudible] experience I asked the medical students in the medical students to tell
them, the young mother of a new baby that was having [inaudible] problems. They
discovered that the mystery loss of platelets was caused by HIV and to tell her mother
that her kid had HIV and by the way so did she. And it was all [inaudible] shock and had
to be delivered in parallel with the family and the medical student on the case [inaudible]
of doing this was [inaudible]
>> Walter Greenleaf: Sure. And it is the sort of thing that you don't necessarily want to
have on-the-job training. You want to know how to do it right the first time. Virtual
environments have been used to train people how to do spinal taps effectively the first
time so that they are not practicing on a real human. Now we can finally use it to teach
some of the…
>>: [inaudible] spinal tap [inaudible] see one, do one, teach one. Is the standard model
in medical school on the [inaudible] patient.
>> Walter Greenleaf: Right. And I don't want to be the first do one. Anyhow, one of the
things that I think is going to be great is what is going on with using virtual avatars now
is that it is expensive to do the green screen. It is hard for us to come up with a variety of
therapeutic teaching environments, but we do it to get the micro-expressions of the face.
So with progress of the facial recognition algorithms that you guys have, I think that we
can eventually get that onto, so that we don't have to use the green screening which is
very expensive. Just quickly here is an example of using a virtual environment for
leadership training.
[video begins]
>>: The scenario that we are going to show you here is a reduction in force exercise so
we gathered managers together from a big bank which has just announced its earnings for
the most recent quarter which unfortunately are…
>> Walter Greenleaf: Reduction in force means firing. So this is using a virtual
environment to teach someone how to correctly lay off someone in a nontraumatic way.
>>: [inaudible] that we can just donate to this project. [laughter]
>> Walter Greenleaf: Well anyhow. This has been used by the business school at UC
San Diego as part of their executive coaching is teach managers how to humanely hire
and fire people and so on. Why would you use a virtual environment to do this? Well, it
is less expensive. You can practice it off-line. And you can keep notes on how to do it
right. And more importantly, and again this gets back to how do brains learn, is you can
practice instead of just doing it on-the-job. I'm not going to go into detail on this one. I
can show it to anybody who is curious later.
Okay some of the work we are doing with ICT right now is using virtual environments as
cognitive assessment tests. This is for being able to, a next generation of mental status
tests for the Army. And again using the virtual environments instead of using paper and
pencil tests to index memory and recall, why not use a virtual environment where you are
asked to walk down an alleyway and recall what you see in there. And later you might be
asked did you see a car? There is a lot of data you can get by just looking at how
someone navigates and how they respond to stimuli and what their recall is. It is much
more effective than the analog tests that are currently being done. They are hoping to use
this to get some baselines from people before they are deployed so they can do an
inventory after they come back to see if there has been an impairment because of
concussion et cetera.
Okay I am about to segue over, this has been a long spiel. I hope you guys are doing
okay. I am about to segue over to talking about this for physical therapy, occupational
therapy et cetera. But before I leave cognitive behavioral stuff are there any questions
about that? Sure.
>>: I was wondering what kind of [inaudible] the person would get when they're using
the virtual environment. So for example if they are practicing in the real world if there
are other people that can teach them about their mistakes versus when they are in the
virtual environment. So what kind of [inaudible]?
>> Walter Greenleaf: Well in the virtual environment they can get the same sort of
feedback. If you are doing group therapy in the virtual environment, there are other real
people there with you. And the clinician is there with you too. So they, it is the same
level of group or expert feedback that you can get too. We would like to move some of
this off-line so that you could learn some of these, practice some of these things like
anger management for example, practice it on your own with a virtual avatar. And I will
show you examples, some simulations that are being used for that. But we are not at that
level of sophistication yet. Right now the interaction we have involves real humans
interacting with real humans. We just use the virtual environments as a scaffolding to
improve the therapy process.
Outside audience, any questions? Okay. I asked you guys earlier if you've gone through
physical therapy, you know it is pretty boring and often the tasks that they have you do
are sort of simulations of functional things. It's like you might in order to practice, you
know, manual movement they might have used stack colored rings on a pole. You might
have stretchy bandages that you pull. One of the big problems is that it is boring of
course, but also when you get home you are often sent with a Xerox sheet of paper with
the exercises that you are supposed to do with maybe some stick figures showing you
what to do. You often forget how to do it right or you don't quite understand the figures
on how to do it right or you're not moving the right way and you end up doing damage.
And there is no feedback loop. You come back to the clinic a week and a half later or
two weeks later, you have been overdoing the exercise or under doing the exercise or
doing it the wrong way. The clinician can only get several sample points and they are
often just very subjective sample points on how you're doing.
So it is very hard to come up with a trajectory for rehabilitation. It is very hard to
dynamically adjust that protocol. So what is great about virtual environments is that we
can make it less boring, more engaging, we can do it in a telemedicine manner so that we
can get feedback on a daily basis on progress, send an alert to the clinician if the patient
is not doing it right or send an alert to the clinician if they are making progress and the
protocol needs to be stepped up a bit. Let me show you some examples. The stuff that is
used for rehabilitation right now is usually pretty old and boring and not very engaging
and again results are mixed. People can have an identical injury and someone might
recover perfectly with one month worth of work and another person might never recover
function. And the difference seems to be understanding what the process is and
complying with it and that is something that is very difficult to do without the proper
guidance. And we can use the virtual environments to provide the guidance.
This is the study that I did with Stanford Hand Clinic. We took the people who were
recovering from upper extremity injury and we hooked them up to a data glove and some
other measurement tools. We hooked up some exercise programs that gave feedback in
real-time. But more importantly sent the data to a server and allowed the clinician to
check in and see how the patient was doing on a basis and get an alert if they had fallen
off the correct protocol. This is something that we did about 10 years ago and again it is
a tribute as to how hard it can be to get things once they have been proven in the lab out
to the clinical community. We have still to get this out and commercialized. But it is a
good example of the right way to do things. We set up an online chat group so that
people could play an interactive game, and we found that with some of the elderly people
that were going through recovery that they liked doing things that were more like a group
like swimming together as you did your exercises as opposed to shooting aliens. And
what was important was to be able to have them get feedback with a little video of what
to do for their exercise, give them some real-time feedback of what their progress is and
how they are doing, allow them to see their progress over time so they are motivated and
again close that feedback loop so that the clinician can see what they are doing.
And this allows you to have a tele-rehabilitation system too where if you are doing
remote therapy, trying to reach underserved areas, it allows you to do it. Some of the big
breakthroughs in, you know people collect rehabilitation now; I would like to change that
to connectabilitation. [laughter] a lot of people are using--now this is actually a problem,
as far as I'm concerned. A lot of clinicians are so eager to motivate their patients that
they will say go home and use the Wii to do some exercises. And here is a game. But
there is no managed protocol. It gets them moving, you know, they might move their
upper extremity, but you really want to manage the protocol. And my friends who are
doing research in this area say that often just sending somebody home with a game to
motivate them does more harm than good. You really want to have a managed protocol
where you are doing the right exercises the right amount, not overdoing it, not under
doing it.
So the answer is to have something that is designed to be a therapeutic tool rather than
just sending somebody home with a game to do the process.
Okay. I am going to talk about some of the things that are just around the corner, a lot of
the things that are being done here. Well actually, before I do that, does anybody have
any questions about--I skipped over pretty fast using virtual environments for physical
therapy and stroke rehab et cetera, any questions about that? It is pretty self apparent that
the value of the technology here. And trust me guys. We want this technology to come
out because in 10 or 15 years when we are older and we might have a cognitive injury or
a stroke or something like that, you don't want to be using those stretchy bandages for
your rehabilitation. It is boring. And the good news is especially with some of the new
pharmacology that is coming out that will help with rehabilitation that recovery from
even a pretty significant injury is possible. But it is painful, it is hard, and it is slow. We
want to have systems like this to help us do it, to help our parents do it, to help our
friends do it, and by the time we need it.
Here are some of the things that I am excited about. Here is something that I think I
grabbed from Microsoft Research thing using texture mapping of faces to avatars. [video
begins]
>>: [inaudible] demonstrating a new approach towards making a photorealistic avatar
using a regular 2-D video of someone as a reference and a simple 3-D model that can be
created with Kinect. The research team synthesized a 3-D avatar event can be animated
in real-time. This process of pasting on a 3-D video reduces the computation required for
photorealism in real-time. As a 3-D talking head, I have many useful applications.
>> Walter Greenleaf: I hope whoever is behind this is listening, because this is really
great. This is really going to help what I do and my colleagues too so much, to be able to
have the real-time texture mapping and facial mapping and the voice synthesis. This is so
important. And something else I am excited about here. This is something with ICT. I
think Microsoft is probably sponsoring some of this research too. This is the work of
using simulations that serve the evolution of Klipee to be able to guide people through
the morass of website information that the military has out there for helping families and
people with posttraumatic stress. There is a huge amount of information out there. What
they want to do is have a coach that you will bond with and trust a little bit, even though
it is an avatar, even though it is a computer AI to help guide you to start the process of
getting help for your problems with suicidality or your problems with substance abuse et
cetera. So let's see if I can find it here. [video begins]
>>: Well it sounds like you are experiencing some signs of depression. You know,
things just aren't as satisfying as they used to be. Now this is pretty common for people
who are going through major changes. And I know it is rough, but you need to be easy
on yourself. Here are three things that can help, having someone to talk to, reading more
about this thing you're going through, and getting back into the swing of things. Here are
some links to get you started.
>> Walter Greenleaf: So this character, they have done a lot of social engineering trying
to figure out who is going to look comfortable for, you know, military servicemen and
women to share information with. They have come up with, you know, other ethnic
groups and other genders in case you choose to relate to that person. But this was, for
example, a DARPA initiated project called Healing Heroes. Where they are going to
build a social media site, I think that is not the operative name anymore, to study and
provide the analysis of what people are typing as they interact to help guide them to find
the right information. And it is done in a bit of an anonymous coaching type of manner
where you get people to actually start sharing information with somebody that otherwise,
you would never share any information with, a computer.
I want to show you some other examples here. You guys have seen this one I am sure.
This is extremely exciting for us. [video begins]
>>: [inaudible] your voice and track your body.
>> Walter Greenleaf: You guys have heard this 100 times.
>>: [inaudible] facial expressions. As you can see now can track features like your
smile, your laugh, and even the raise of your eyebrows.
>> Walter Greenleaf: Since you guys have heard it 100 times I am going to cut it short.
But that is incredibly important to what I'm doing, to be able to capture the facial
expressions, to be able to capture the movements, to bring that onto an avatar. It is like
the fifth dimension of interactivity and believability that we need for the type of care that
we are doing. So it is very important and I am very excited about it. But that is part of
my standard spiel that I give to audiences about what is just around the corner. Keep it
coming please. Let's see here. Oh, some of the other things that we are working on now.
We don't have things like hugging and shaking hands and crying and in the standard
video games in virtual environments that are out there, there aren't. They are mostly
pretty perfunctory in terms of being able to drive a car or been able to stab someone or
being able to blow up something. So one of the things that we are putting into our virtual
environments are more social things such as being able to have the bio kinetics right for
hugging, for shaking hands, things, things that are really important and not part of the
standard repertoire that we are seeing in virtual worlds now. So whoever is out there
listing we want to encourage you to make sure that you build it into your virtual worlds
all aspects of human behavior, because for therapeutic purposes we need it. Again for the
realistic avatars, one of the things I am starting to see are algorithms for cultural specific
nonverbal behaviors. For example, you know, in many of the virtual environments we
are using for therapy right now, I notice that the women in burqas are standing around
with their hands on their hips like this, and they have this typical body language of an 18year-old boy. And it is because it has been modified from video games, of course. But
we need to start coming up with more cultural specific, gender specific nonverbal
behaviors for our therapeutic use, so we are looking forward to that.
And this may be non-germane now that some of the other breakthroughs have come
through but there are groups out there working on really making mapping of facial
expressions to emotions so that we can do a better job of conveying emotions on the
avatars and have the right algorithms for doing that. So it doesn't just have to be in real
time for catching a facial expression, but then we can generate them as we need them for
therapeutic purposes. All right, well, later contact me and I can give you a list of, if you
are interested in this area, there are a number of research groups and societies and
companies that are doing work in this area. It is really a rapidly emerging area which I
think is going to be catalyzed by some of the new breakthroughs in technology that are
occurring here. And that is it. That is how they get in touch with me.
[applause]
>>: I have a question. In the range of applications that you worked with if there was one
type of sensing or analysis that you don't have access that you would really like, some
part of the body or some particular signal that you would really want to get can you talk
about what that would be?
>> Walter Greenleaf: Sure. From a physiological viewpoint, I need heart rate. From
heart rate I can extract respiratory rhythm. From heart rate and respiratory rhythm I can
get sympathetic nervous system tone. From sympathetic nervous system tone I can get
attention and I can get anxiety. And if I am going to be doing any telemedicine aspects
of this I need that. So I know there is work been done at Media Lab and other places of
extracting noninvasively heart rate, but that is something that is very important for all
aspects of what I am doing. The other thing that is really important, I have talked about
but I don't think it's out there yet, is being able to get facial expressions mapped to
avatars. I need a virtual world with train stations and cafés and alleyways and houses that
I can have avatars in, but I need those avatars to have proper body language and I need to
have proper facial expressions. So it is the nonverbal aspects of human communication
that needs to be put on to avatars. What is going on in Second Life right now where
people walk like this and their faces like that, just won't do it for therapy. So you asked
me for one thing, I gave you two.
>>: What about GSR which is important for [inaudible].
>> Walter Greenleaf: You know some people will disagree with this but I am not a big
fan of GSR. I think it is, to me it is sort of a noisy signal and it is, some people love it
and use it in their research. I always preferred more pulse amplitude and other signals as
opposed to GSR. But I wouldn't say no if you could bundle it with some of the other
signals.
>>: [inaudible] whereas respiration and heart rate don't need body contacts.
>> Walter Greenleaf: That's right. I would like to avoid skin contact because clinically I
have to do a lot more care with that too, in placing the electrodes and also some people
have thicker skin and it is a relative change.
>>: Yeah, and it's another signal.
>> Walter Greenleaf: Yes it's another signal. Well look, if I get them, I need heart rate; I
need respiration; I need GSR; I need EEG; I need eye gaze; there is a lot I need but Jerron
only gave me one. But the more psycho physiological signals you can give me, the
better. You know, I would love to have like a ballistic pneumograph that somebody
could sit on or something on your mouse that has a photo methisnograph that can get
pulse amplitude; that would be really great. But I will take anything right now. Yes?
>>: So it is pretty clear that these developments work very well in a [inaudible] do you
see any application inside of a hospital environment, instead of at home [inaudible]?
>> Walter Greenleaf: Well sure, for acute stages of rehab the virtual environments can
be really important for motivation. If someone is recovering from a stroke or a traumatic
brain injury, that first few steps when they are still in the hospital are really important and
the virtual environments can really help with that. It can also later be part of the
continuity of care. If somebody is injured over in Iraq, and they go to Germany, and then
they go to Walter Reed and then they go to the local hospital, it is nice to have the
clinicians be able to track them, and virtual environments can be used for that. I guess
also in a hospital--well right now it is used quite a bit by radiologists and surgeons, but I
guess I would use it for maybe, well here is one example, if you are taking your son into
the hospital the next day for a surgical procedure, you might want to be able to go to a
virtual representation of that hospital, walk through the clinic, get familiar, it's stress
inoculation, get familiar with what's going to happen, talk to maybe an avatar clinician
who is going to give you a tour, prepare you for what's going on. So it is not exactly in
the hospital, but it is part of the hospital experience. I think that would be a really great
thing. The other way it can be used in a clinic is for, well all I can pick up right now
pretty much is neural rehab and stress inoculation but I am sure there are others, pain
management, you know. That is an important area too. You must be thinking of one
too?
>>: I was just curious. So we had a sample [inaudible] duty doing possible discharge, so
education.
>> Walter Greenleaf: Education is really important I think, again, for clinicians, it's the
behavior of being a clinician that is very important. And if you are just using actors to
train the clinicians, it is very expensive and they often don't get enough. Also for
psychology, it is hard to supervise what goes on in clinical psychology because unless
you are using one-way mirrors with volunteer patients, it is very hard to be there. If we
start using virtual environments as a therapeutic environment, then a supervisor can be
watching as an anonymous avatar, and again with permission, and can observe and
maybe even give coaching, we do this right now for our group therapy where the kids
might be interacting with each other, but one kid isn't getting it, or one kid is being
unruly, so even though they are in a group, the leader, we call it, can talk to the one kid
and say look, you're coming on too strong or you are reading that person the wrong way,
so even if they are in a group, they can have a private conversation. So the virtual
environments involve I guess better group interactions in a therapeutic manner. And that
can be done in a hospital setting too.
>> Eric Horvitz: I think that's about all the time we have Walter so thanks again.
>> Walter Greenleaf: Yeah, thank you guys. [applause]