Augmented reality based affect
adaptive neurocognitive rehabilitation
Intermediate report of Msc. Thesis project
A.S.Panic
• Student Media & Knowledge
Engineering (EEMCS) - Man
Machine Interaction
• 5 months project work at ETH Zurich
• Financial support provided by
Imperial College, London, TU Delft, ETH Zurich,
RWTH Aachen, ParisTech
http://www.idealeague.org/
In this presentation
Thesis introduction
Literature survey
Experimental
study
Augmented reality based affect adaptive neurocognitive rehabilitation
Thesis position
1. Neurocognitive rehabilitation is possible to a
higher extent than previously assumed
2. Therapeutic exercises can be made more
motivating for the patient
3. In the near future more people will need
cognitive (re)habilitation than have access to, or
can be treated in the existing clinics
4. Human factors play a significant and
undervalued role when designing technology
Augmented reality based affect adaptive neurocognitive rehabilitation
Surveying the literature
• Three questions drive the literature survey:
1. Which concepts are used to understand cognitive
processes and neurocognitive rehabilitation?
2. How can neurocognitive rehabilitation be
supported by virtual reality applications?
3. How can therapeutic games be created so that they
promote a motivated learner?
Augmented reality based affect adaptive neurocognitive rehabilitation
Main topics for literature survey
Cognition
Brain plasticity,
Cognitive reserve
Cognitive
rehabilitation
Affect, Learning,
Games
Augmented
Reality
Augmented reality based affect adaptive neurocognitive rehabilitation
Basic stages of cognition
• Lack or loss of language
severely impairs cognition
• Sound and music can
significantly influence
emotion
• Emotion significantly
influences cognition
Source: Groome & Dewart, 1999, An introduction to cognitive psychology: processes and disorders
Augmented reality based affect adaptive neurocognitive rehabilitation
Emotion and cognition
One example of cognitive influences of emotion:
• In a study at Tufts University:
1.
Test subjects were blindfolded and
physically moved a random
distance between two houses
2.
The test subject were then asked
to which of the two houses he or
she was closer to
The conclusion: the reported distance estimates were
significantly different if the two houses were religious
in nature (e.g. church, mosque) versus neutral
Source: Qui Wang at the Psychology department of Tufts University
Augmented reality based affect adaptive neurocognitive rehabilitation
Stages not easily summarizable
• Stages of visual perception
include processing horizontal
lines, vertical lines, arcs, edge
detection, shape detection,
• Stages of auditory perception
include pitch, timbre, rhythm,
melody, localization, so…
A multi-stage model of memory
• In reality cognitive processes are not identifiable by
independent and distinct stages, but composed of many
(overlapping) substages
• Cognition operates at the levels of basic skills, executive
functions and general intelligence (fluid and crystallized)
Source: Groome & Dewart, 1999, An introduction to cognitive psychology: processes and disorders
Augmented reality based affect adaptive neurocognitive rehabilitation
Loss of cognitive skills
• A loss of cognitive skills can be caused by:
– Typical aging process
– Neurodegenerative
diseases (see picture)
– Traumatic brain injury
• Similar symptoms exist in young and old
people, but with different names
• Mostly pathological causes
Source: Fisk et al, Designing for older adults
Augmented reality based affect adaptive neurocognitive rehabilitation
Cognitive (behavioral) therapy
• Cognitive therapy (CT) aims at rehabilitating any of
the basic cognitive skills
• Cognitive behavioral therapy (CBT) aims at
rehabilitating one or more Activities of Daily Living
(ADL)
• Issues with CT and CBT:
– Treatment or assessment can be dependant on
administrator (therapist)
– Lack of standardized stimuli or treatment content
– Therapeutic exercises can be boring
Source: Buschert, 2009, Kognizionsbezogene Interventionen bei Alzheimer-Krankheit
Augmented reality based affect adaptive neurocognitive rehabilitation
Cognitive rehabilitation strategies
Cognitive rehabilitation can aim to:
1. Reinforce, strengthen or reestablish previously
learned skills or behavior
2. Establish patterns of compensatory cognitive
activity to cope with impairment of cognitive
systems
3. Establish environmental compensation that allow
new patterns of activity
4. Enable the patient to adapt to their cognitive
impairment in order to improve overall level of
functioning and quality of life
Source: Cicerone, 2000, 2005, Evidence based cognitive rehabilitation, recommendations for clinical practice
Augmented reality based affect adaptive neurocognitive rehabilitation
Brain plasticity and cognitive reserve
• Since the 1980s two (theoretical) constructs
became adopted when explaining how a loss of
cognitive skills can be rehabilitated:
1. Brain plasticity = the changing of neurons, the
organization of their networks, and their function
via new experiences.
2. Cognitive reserve = the brain’s resilience to neuro
pathological damage
Augmented reality based affect adaptive neurocognitive rehabilitation
Gaming stimulates brain plasticity
• MRI assessments show
that structural
engagement with the
Tetris game increases
formation of ‘gray
matter’
• Lack of conclusive evidence for other games, e.g. Dr
Kawashima’s “Brain Training” for Nintendo DS.
• However many of these games are based on
standard instruments of cognitive assessment.
Source: Haier et al, 2009, MRI Assessment of cortical thickness and functional activity changes in adolescent girls,
following three months of practice on a visual-spatial task
Augmented reality based affect adaptive neurocognitive rehabilitation
Cognitive reserve
• Cognitive reserve is based on two mechanisms:
1. Neural reserve
2. Neural compensation
• Evidence for cognitive reserve:
Nuns have been diagnosed with
Alzheimer’s Disease
(neurodegenerative!), but without
any noticeable loss of cognitive
skills
Sources: Stern, 2006, Cognitive reserve and Alzheimer’s disease
Snowdon, 2004, Healthy aging and dementia: findings from the Nun study
Augmented reality based affect adaptive neurocognitive rehabilitation
Case studies and explanations
Brain plasticity, Discovery channel clip (5 min)
This video clip, from the Discovery Channel and McGraw-Hill Higher Education,
details how the brain can "repair" itself. It introduces a case study of young Jody
Miller, who had half of her brain removed, in an attempt to control her epileptic
seizures.
http://www.youtube.com/watch?v=TSu9HGnlMV0
Re-wiring the brain, 2008 TED talk (20 min)
Neuroscientist Michael Merzenich looks at one of the secrets of the brain's incredible
power: its ability to actively re-wire itself. He's researching ways to harness the
brain's plasticity to enhance our skills and recover lost function.
http://www.ted.com/talks/michael_merzenich_on_the_elastic_brain.html
Augmented reality based affect adaptive neurocognitive rehabilitation
Brain plasticity – a case study
stroke of insight, 2008 TED talk (20 min)
Brain researcher Jill Bolte Taylor studied her own stroke as it happened. She has
become a spokesperson for stroke recovery and for the possibility of coming back
from severe brain injury stronger than before.
http://www.ted.com/talks/jill_bolte_taylor_s_powerful_stroke_of_insig
ht.html
Augmented reality based affect adaptive neurocognitive rehabilitation
Stimulating cognitive reserve
• At any point in one’s lifetime, cognitive reserve
results from a combination of exposures:
– Socio-economic status (e.g. occupation)
– Educational attainment
– Leisure activities
• Physical activity stimulates neurogenesis
Source: Stern, 2009, Cognitive reserve
Augmented reality based affect adaptive neurocognitive rehabilitation
The virtuality continuum
• Allows the classification of applications that
create an illusion of an alternate reality
Mixed
Reality
Reality
Augmented reality,
more real than virtual
(e.g. HMD)
Virtual
reality
Augmented virtuality,
more virtual than real
Non or partially-immersive VR
(e.g computer monitor)
Source: Milgram & Kishino, 1994, A taxonomy of mixed reality visual displays
Augmented reality based affect adaptive neurocognitive rehabilitation
Creating a fake reality
Sensorama (1957) by the ‘godfather of virtual reality’
• One to four simultaneous users
• 3D motion picture, smell, stereo
sound, vibrations, wind
• The holy grail of virtual reality is
to fool all the senses and create
a ‘suspension of disbelief’ for
the virtual environment
Source: Morten Heilig, 1962, Sensorama patent
Augmented reality based affect adaptive neurocognitive rehabilitation
VR based cognitive rehabilitation
• When using virtual
reality for
neurocognitive
rehabilitation, what are
the:
–
–
–
–
Strengths?
Weakness?
Opportunities?
Threats?
• Assumptions:
– Administration in home
environment
Source: Adapted from Rizzo & Kim, 2005, A SWOT analysis of the field of virtual reality
rehabilitation and therapy
Augmented reality based affect adaptive neurocognitive rehabilitation
SWOT of particular interest
• Strength:
– High fidelity training and assessment environments can be
created
– Economy of scale
– Gaming factors can be used to increase patient motivation
and adherence
• Weakness:
– Interface challenge: wires, displays and peripherals
– Side effects of exposure to virtual environments may not be
unavoidable
– Engineering challenge: proper support for data mining,
extraction & analysis (for therapists!)
Source: Adapted from Rizzo & Kim, 2005, A SWOT analysis of the field of virtual reality rehabilitation
and therapy
Augmented reality based affect adaptive neurocognitive rehabilitation
SWOT of particular interest
• Opportunities:
– Emerging technologies:
• Unobtrusive and natural interaction devices, wires
• Wearable computing, sensor networks
– Situated cognitive rehabilitation / telerehabilitation
– Preventive training during preclinical stage increases
cognitive reserve and resilience to pathology
• Threats:
– Privacy issues with telerehabilitation and medical data
– Potential for lawsuits because of side-effects
– Ethical challenges
Source: Adapted from Rizzo & Kim, 2005, A SWOT analysis of the field of virtual reality rehabilitation
and therapy
Augmented reality based affect adaptive neurocognitive rehabilitation
Motivation in instructional games
Instructional
Content
Process
S
fee ystem
db
ac
k
Input
Outcome
b e U se
ha r
vi o
r
Game
Cycle
debriefing
Learning
outcome
Game
Characteristics
User
judgement
• The goal is to create a game cycle which:
– fulfills learning objectives and
– creates and sustains a motivated learner
Source: Paras, 2005, Games, motivation and effective learning: an integrated model for educational game
design
Augmented reality based affect adaptive neurocognitive rehabilitation
Affective gaming
• Affective computing: computers and programs which
take the player’s affective (emotional) state into
account
• Affective games: affect-adaptive games adapt their
content based on the player’s emotional state. E.g.
adapt gameplay difficulty, offer more or less
feedback and support on progress, offer tips and
suggestions on how to continue, increase or
decrease number of concurrent tasks etc.
• High level design heuristics: “Emote me, assist me,
challenge me”
Sources: Picard, 2000, Affective Computing
Gilleade et al, 2005, Affective videogames and modes of affective gaming
Augmented reality based affect adaptive neurocognitive rehabilitation
Affective game engines
• Core requirements:
1. A knowledge base which (hierarchically) describes emotions,
used for emotion generation or expression.
2. An affective user model (AUM) of the player, which facilitates
recognition of emotion and transitions between them.
3. Modeling of game characters’ emotions.
•
No computational model of cognitive processes required, a
few simple rules can already have a measurable effect
Source: Hudlicka, 2009, Affective game engines: motivation & requirements
Augmented reality based affect adaptive neurocognitive rehabilitation
Thesis position (revisited)
1. Neurocognitive rehabilitation is possible to a
higher extent than previously assumed
– This may be explained by theories of brain plasticity
and cognitive reserve
2. Therapeutic exercises can be made more
motivating for the patient
– This can be achieved by designing therapeutic games
using criteria from affective computing and affective
gaming
Augmented reality based affect adaptive neurocognitive rehabilitation
Thesis position (revisited)
3. In the near future more people will need cognitive
(re)habilitation than have access to, or can be
treated in the existing clinics
– Using telerehabilitation and commonly available (cheap)
computing technology, virtual reality based therapy can
be offered to a larger part of the patient population
4. Human factors play a significant and undervalued
role when designing technology
– Human Computer Interaction design should target
natural and embodied interaction as much as possible.
– Differences in technological aptitude should not prevent
people from accessing healthcare
Augmented reality based affect adaptive neurocognitive rehabilitation
Experimental research questions
• Principal research questions:
– Does the use of affect-adaptive training influence the
rate of learning and motivation on a commonly used
instrument of cognitive performance, when
compared to a standard computer based training?
– Does the use of human computer interaction devices
which support more natural and embodied
interaction lead to a higher rate of acceptance by the
targeted population?
Augmented reality based affect adaptive neurocognitive rehabilitation
The mental rotation task
• Research vehicle: the mental rotation task (MRT)
Are these two objects the same?
• Response time depends on:
• Difference in rotation between object pair
• Object complexity (but this diminishes with
practice!)
Sources: Shepard & Metzler, 1972, Mental rotation of three dimensional objects,
Bethell-Fox & Shepard, 1988, Mental Rotation: effects of stimulus complexity and familiarity
Augmented reality based affect adaptive neurocognitive rehabilitation
Why use the mental rotation task?
Rationale for using the MRT in this experiment:
– As a widely used instrument for cognitive
assessment it is particularly suitable for investigating
gaming factors that influence motivation
– As a visuo-spatial task it is particularly suitable for
virtual and augmented reality
– Stimulus complexity can be controlled precisely
Sources: Rizzo et al, 1998, the virtual reality mental rotation spatial skills project
Augmented reality based affect adaptive neurocognitive rehabilitation
Experiment setup
IR LED glasses allow tracking of
head position & gaze direction
Wii remote allows gestural
interaction with virtual
objects (pointing, dragging,
selecting) and tactile and
auditory output
Training,
Assessment
With headtracking support, a
TV screen allows virtual
objects to appear in front
or behind of it
Augmented reality based affect adaptive neurocognitive rehabilitation
Experiment protocol
• Targeted population:
– Elderly people with mild cognitive impairment
– Vision corrected to standard, no history of
postural instabilities, no motion sickness
• In a clinical session the participants complete:
1. A training round (using 1 of 3 game modes)
2. A feedback round (motivation questionnaire)
3. An assessment round (of task performance)
Augmented reality based affect adaptive neurocognitive rehabilitation
Project status
• Dec 2009 – Mar 2010
– Project work @ ETH
• Mar 2010 – Jul 2010
– Finish writing Msc. Thesis @ DUT
– Thesis defense
– Complete 2 remaining human factors in aerospace
courses from my honors program
Augmented reality based affect adaptive neurocognitive rehabilitation
Do you want to know more?
Y. Stern,“Cognitive Reserve”,
Neuropsychologia vol 47, 2009
http://www.cumc.columbia.edu/dept/sergievsky/cnd/pdfs/sdarticle-1.pdf
A. Rizzo & G. Kim, “A SWOT analysis of the Field of Virtual
Reality Rehabilitation and Therapy”, Presence vol 14, 2005
http://vrpsych.ict.usc.edu/PDF/1Rizzo_2005_A%20swot%20analysis%20of%20the%2
0field%20of%20VR.pdf
E. Hudlicka, “Affective Game Engines: Motivation
and Requirements”, ICDFG proceedings, 2009
http://www.imgd.wpi.edu/speakers/0910/Hudlicka_ICFDG_09_Dist.pdf