Chapter #
Gaming and Social Interactions in the
Rehabilitation of Brain Injuries: A Pilot
Study with the Nintendo Wii Console
R.C.V. Loureiro, D. Valentine, B. Lamperd, C. Collin
and W.S. Harwin
Abstract
Physical rehabilitation of brain injuries and strokes is a time consuming and
costly process. Over the past decade several studies have emerged looking at the
use of highly sophisticated technologies, such as robotics and virtual reality to tap
into the needs of clinicians and patients. While such technologies can be a valuable
tool to facilitate intensive movement practice in a motivating and engaging
environment, success of therapy also depends on self-administered therapy beyond
hospital stay. With the emergence of low-cost gaming consoles such as the
Nintendo Wii, new opportunities arise for home-therapy paradigms centred on
social interactions and values, which could reduce the sense of isolation and other
depression related complications. In this paper we examine the potential, user
acceptance and usability of an unmodified Nintendo Wii gaming console as a lowcost treatment alternative to complement current rehabilitation programmes.
Introduction
Although an increased effort is placed on the recovery process of patients
following a brain injury such as a stroke, economic pressures, and lack of available
human resources means that patients generally do not reach their full recovery
potential when discharged from hospital following initial rehabilitation (Broeks et
al., 1999). Although there is already evidence suggesting that the damaged motor
system is able to reorganise in the presence of motor practice, optimal training
2
R.C.V. Loureiro, D. Valentine, B. Lamperd, C. Collin and W.S. Harwin
methodologies promoting such reorganisation remain unclear due to discrepancies
of current rehabilitation therapy, quantification of dosage and rehabilitation type
(Shadmehr and Mussa-Ivaldi, 1994). The recovery of upper limb function is
particularly affected as the initial challenge is to stabilise the trunk and relearn
minimum independence levels through gait re-learning. Robotic machines have
been identified as a possible way to automate labour-intensive training paradigms,
to improve patient access to therapy and to provide new tools for therapists.
Several authors have already proposed the use of robots for the delivery of this
type of physiotherapy (Krebs et al., 1999; Lum et al., 1999; Johnson et al., 1999;
Reinkensmeyer et al., 2000; Loureiro et al., 2003; Hesse et al., 2003; Nef and
Riener, 2005). One of the challenges still present today, is to how best use robotic
technology to augment the physiotherapist’s skills (Harwin et al., 2006). On the
contrary to public perception, robotic technology aims to be an advanced tool
available to the physiotherapist and not a replacement. A robotic system is very
unlikely to be able to amass all the skills of a physiotherapist, but it will be very
good at conducting comparatively simple repetitive and manually intensive
therapies. In this context, the physiotherapist would be doing all the clinical
decisions and when suitable, considered and executed on the robot.
Robotic therapy is appealing because it can deliver complex therapies that
would be too difficult for therapists to do, for instance provision of precise
repeatable force and haptic feedback coupled with interesting and motivating
visual feedback and/or the ability to augment movement errors to help correct a
movement pattern (Patton et al.,2006). According to Hesse (Hesse et al., 2006)
rehabilitation robots providing repetitive induced strategies are better positioned
for therapy delivery to severely impaired patients in need of external movement
assistance and support to overcome muscle weakness problems. Conversely, taskoriented therapy approaches are more suitable for mildly affected patients.
New strategies are being examined based on error augmentation (Patton et al.,
2006), gravity assistance (Sukal et al., 2005), bilateral training and cueing (Johnson
et al., 2006), exoskeletal training options (Nef and Riener, 2005), cooperative
home-rehabilitation paradigms (Loureiro et al, 2006) and functional whole-arm
(arm and hand) rehabilitation strategies (Loureiro et al., 2007; 2009) but there still
is a need to provide better training strategies anchored in motor learning and
neuroscience theories and affordable opportunities for intensive therapy.
Recent advances in computer power and graphics have prompted for a variety
of innovative highly-engaging gaming technologies coupled with realistic virtual
environments to reach the consumer market. Such technologies have the potential
of providing an interesting and effective way of delivering rehabilitation exercises
to people recovering from a brain injury or related cognitive deficit. Virtual
environments have the potential of delivering safe and customisable training
tailored to the patient’s disabilities with performance being monitored and
positivity encouraged (Rizzo and Kim, 2005). The relatively low-cost of such
gaming technologies, opens doors not only to its introduction in hospital acute
rehabilitation programmes but also to home rehabilitation paradigms. In this
context videogames could be used in the future where a group of people with
similar disabilities could be engaged in group therapy. The games could be
customised to allow the patient to succeed thus boosting morale and increase
Gaming and Social Interactions in the Rehabilitation of Brain Injuries: A Pilot Study with
the Nintendo Wii Console
3
participation. Clinicians could log on remotely and adjust the therapy regimes,
check progress and monitor safety. The next sections of this paper report on a pilot
study evaluating the potential and acceptance of the Nintendo Wii console as
additional tool to current rehabilitation practice.
Rehabilitation with the Nintendo Wii Console
At the end of 2006 Nintendo launched their home videogame console named
Nintendo Wii and from the outset it has proved so popular that it has outsold the
market competitors Microsoft and Sony (Lee, 2008).
Its low-cost and
revolutionary interactive game-play interface, the Wiimote (figure 1.1), has
attracted attention from the traditional hardcore gamers to older people,
encouraging group and family play. The Nintendo Wii has been so successful in
attracting new class of gamers that soon nursing homes in the USA started using
the console to keep elderly residents active with group activities (BBC News,
2008), followed by its use as part of treatment of burns victims in the south-East of
England (BBC News Health, 2008).
B Button
(trigger)
D-Pad
A Button
(a)
(b)
Figure 1.1. Nintendo Wiimote interface. (a) Wiimote controller details; (b) User holding
Wiimote. The IR camera is located on the darker area at the tip of the device.
Nintendo innovative Wiimote interface (figure 1.1) has been reversed
engineered by computer enthusiasts, and used to illustrate the remote interaction
capabilities of the Wiimote without the console in a variety of different
applications (Lee, 2008). The Nintendo Wiimote (figure 1.1-a) is a handheld
device packed with sensors such as buttons, a cursor, a 3-axis accelerometer and an
IR camera. Two actuators - a speaker and a vibration motor - provide audio and
tactile feedback. Data communication is made wirelessly via Bluetooth
connectivity and users interact with the game by holding the Wiimote controller on
their hand (figure 1.1-b) and pointing to the television set (figure 1.2) where often a
sensor bar – containing infrared LEDs - is placed. The IR camera incorporated
4
R.C.V. Loureiro, D. Valentine, B. Lamperd, C. Collin and W.S. Harwin
inside the Wiimote is used to transform the x-y coordinates from the dot pairs
produced by the infrared LEDs. The console software then uses this information
together with the accelerometer data to determine the x-y coordinates and the
rotation (yaw, pitch and roll) of the device. Distance to the television set is
estimated by calculating the separation of two opposite infrared LEDs on the
sensor bar relative to the camera view field (Lee, 2008).
Wii Therapy Usability Study
A small usability study was conducted at the Royal Berkshire NHS Foundation
Trust Hospital in Reading with nine acute subjects undergoing rehabilitation.
Subjects played a variety of games for the Nintendo Wii console and interacted
with the games using the Nintendo Wiimote. The Wii Sports and Wii Play game
packs were selected for this study as they contain a variety of fun games involving
varying degrees physical ability/mobility and cognitive function (figure 1.2) and
are suitable to subjects with different interests and ability. Games such as bowling,
tennis, shooting and air hockey promote unilateral play, whereas games such as
golf, boxing and baseball are intrinsically more suitable for bilateral movements.
Figure 1.2. Subject playing bowling using the Nintendo Wii console
Games were first demonstrated by one of the occupational therapists (OT)
delivering the rehabilitation intervention and played by the subjects either with
support from the OT or unaided. Each of the involved OTs completed a simple
questionnaire assessing the subjects’ clinical status prior to start the trial and on
trial completion reported gaming limitations observed with each subject during the
Wii sessions. At the end of the trial each subject was presented with a simple
usability questionnaire rating their opinion of the Wii rehabilitation sessions.
Gaming and Social Interactions in the Rehabilitation of Brain Injuries: A Pilot Study with
the Nintendo Wii Console
5
Table 1.1 shows a summary of the subjects who have participated in the study.
Subjects’ age ranged from 18 to 66 years (mean ± SD: 43.8 ± 17.1) and completed
between 3 to 12 sessions (mean ± SD: 8.4 ± 3.2) of Wii therapy. Each session
consisted of up to one hour of combined preparation/demonstration and game play.
The recruited subjects all had a neurological diagnosis which ranged from cerebral
vascular accidents, traumatic brain injuries to spinal cord disease and to other more
rare conditions such as Beri Beri with Korsakoff’s syndrome (table 1.1).
Impairments included weakness, reduced range of motion, reduced fine motor
skills, pain, oedema, sensory loss, and cognitive deficits including reduced
attention and concentration, slowed information processing and reduced memory.
Table 1.1. Subjects’ summary
Subject
Sessions
Age
No.
Completed
Diagnosis
Spine Injury + Critical
Illness Neuropathy
Cerebral Vascular
Accident
Cerebral Vascular
Accident
Impairment affecting subject
participation on the Wii sessions
Unable to weight bear through right
lower limb
1
66
10
2
55
12
3
30
3
4
31
10
5
49
6
6
18
10
7
57
4
8
28
10
Spinal Tumour
Reduced balance and left shoulder pain
9
60
11
Cervical Radiculopathy
Reduced active ROM, oedema in both
hands. Hand pain and reduced sensation
Reduced active ROM and balance
Ataxia and reduced dexterity
Reduced power, sensation and dexterity
Berri Berri +
in hands, severe pain with difficulties
Korsakoff's Syndrome with memory, attention, cognition and
reduced capacity to follow instructions
Chronic Inflammatory
Reduced balance, sensation,
Demyelinating
coordination and fine motor skills
Polyneuropathy
Reduced coordination, balance and
Traumatic Brain Injury
power in both upper limbs (L>R)
Cerebral Vascular
Reduced balance and coordination
Accident
Results and discussion
The results obtained from the user opinion questionnaire shows that 44% of the
subjects responded that they had already played with the Nintendo Wii before
starting the study (figure 1.3). All of the subjects reported that they enjoyed
playing on the Nintendo Wii, and 89% of the subjects thought that the Nintendo
Wii should be a regular part of their treatment sessions whilst in hospital, and
would like one at home to carry on with treatment after discharge (figure 1.3).
With regards to their ability to interact and hold the Wiimote while playing the
games, 33% of the subjects (subject 3, 4 and 9) required help either from the OT
6
R.C.V. Loureiro, D. Valentine, B. Lamperd, C. Collin and W.S. Harwin
when movement was too jerky to stabilise the arm or used both hands to perform
the movement. Interestingly, subject 4 used the left hand to operate the D-Pad (in
the bowling game) and the right hand to perform the swinging movement. Subject
9 on the other hand, required occasional help to press A+B buttons together.
Although all games were initially demonstrated by the OT, subjects chose the
game they wanted to play. Figure 1.4 shows that 78% of the participants preferred
playing bowling with only 11% preferring air hockey. One of the reasons for the
trend observed in figure 1.4 could be related to the fast pace required to play the
game (e.g. air hockey) and to the ability to play a game in a relaxed way with more
than one opponent (e.g. bowling).
33
Response
Need help to hold
wiimote
89
Want Wii at home
89
Wii part of treatment
100
Played before
44
0
Enjoyed playing
50
100
% Response
Figure 1.3. Participants’ response to the user opinion questionnaire
11
Air Hockey
44
Shooting
22
Game
Baseball
44
Boxing
44
Golf
Tennis
67
Bowling
78
0
20
40
60
80
100
% Played
Figure 1.4. Percentage of games played by participants
Gaming and Social Interactions in the Rehabilitation of Brain Injuries: A Pilot Study with
the Nintendo Wii Console
7
The self-evaluation responses (table 1.2) obtained with each subject after
completing the Wii therapy sessions shows responses in line with the rehabilitation
goals the OTs hoped to achieve during the Wii therapy sessions (table 1.4, second
column). The results show a positive impact on improving subjects’ trunk and
upper limb strength, balance, coordination, and on increased session participation,
attention and concentration.
It is clear from this study that while the main responses are positive, certain
aspects of the Wii therapy require attention and improvement, such as the hardware
interface. The subjects’ gaming limitations observed by the OTs (table 1.4, fifth
column) is in line with the participants’ comments and suggestions on how to
improve the Wii interface for easy use (table 1.3). The typical response was that
some of the games were too fast or the Wiimote too sensitive to movement change.
Likewise interaction with the Wiimote buttons caused difficulties to some
participants (subject 4, 5, 6, 9) who could not press the buttons due to their
physical size or because of the reduced sensation on the participants’ fingers.
While OTs were positive about group activities – with participants engaging in
competitions while playing bowling and boxing – contributing to improved
function and cognitive deficits, games were at times too fast and high tone patients
could not use the Wiimote. Although the gaming scores are useful for patient
feedback, objective data on posture, range of motion, grip and dexterity would be
useful to aid therapists’ patient assessment.
Table 1.2. Summary of participants’ response to question “How do you think using the
Nintendo Wii has helped you?”
How do you think using the Nintendo Wii has helped you?
Subject
Comment
1
2
Strenghted my trunk; Involvement in game tasks takes mind off pain
Improved my concentration and upper limbs
3
4
5
No response
Helped getting me moving around and using my hands
It has greatly improved my upper arm strength and fine motor skills.
It has also helped me improve my sitting balance and is now being
used to work my standing balance
It help me channel my anger, improve my balance and my hand
control
Helped me with balance and concentration. Improved my dexterity,
ability to getting involved in exercise, and helpd to get fit
6
7
8
Great cardiovascular exercise (boxing) and enjoyable leisure activity
9
Helped me improve my grip and coordination
8
R.C.V. Loureiro, D. Valentine, B. Lamperd, C. Collin and W.S. Harwin
Table 1.3. Summary of participants’ comments and suggestions to question “How can we
improve the Wii interface to make it easier to use?”
How can we improve the Wii interface to make it easier to use?
No. Participant comment*
1
2
3
4
Reduce sensitivity or cursor movements
Ability to restrict arm to concentrate on specific movements such as
flexion/extension only
Have a wiimote with larger buttons
Easier game interaction: A+B buttons difficult to press together
with weak hand and reduced coordination
* Several comments were repeated by different participants. To avoid repetition the
main point is stated once
Table 1.4. Summary of Occupational Therapists’ assessment on participants’ involvement
in the study
Subject
1
2
3
4
5
6
7
8
9
Subject UL AROM level
Rehabilitation Goals hoping
achieve during Wii sessions
Left
Right
Increase UL strength, trunk control,
Full but
Full but
standing balance and exercise
weak
weak
tolerance
Improve balance, progress to
Full but
Full but
palying standing, increase
weaker than
weak
attention up to RW, improve
Left
muscle strength and hand-eye
Increase participation in session;
Full ataxic
Full ataxic
use of both UL, improve
movement movement
coordination or movement
patterns, sequencing and timing
Full but
Full but
Increase participation in session;
bilateral
bilateral
improve coordination of limbs
weakness
weakness
Improve upper limb ROM and
Reduced
Reduced
strength, improve sitting balance
shoulder
shoulder
and core stabillity
flexion
flexion
Full but
Full but
Bilateral hand movements and use
reduced
reduced
of left UL, Increase balance and
coordination coordination
coordination, anger management
and power and power
No
ability to stand during therapy
Full
functional
session and completing game
movement
Reduced
Working on increasing dynamic
Left
Full
sitting balance
shoulder
due to pain
Improved coordination and grip of
upper limbs
Subject gaming limitations
observed
None. Sat for majority of
sessions
Confidence and tolerance
Sensitivity of cursor + infrared
bar; control or direction when
moving the controller
Complexity of controls,
sensitivity of controls to ratation
Sensitivity of cursor on screen.
Issue with which buttons on
wiimote could be pressed
Reaching speed too quick, did
not alow subject time to connect
with ball. Coordination of
swinging and releasing button
No response
None
Griping hand control due to
Limited due Limited due oedema. Pressing A+B together.
to oedema to oedema Feeling where button B was due
to decreased sensation
Gaming and Social Interactions in the Rehabilitation of Brain Injuries: A Pilot Study with
the Nintendo Wii Console
9
Conclusion
From our experience in conducting this pilot study, we have found that the
participants’ comments provide an essential insight into usability issues of the
Nintendo Wii console and the Wiimote as a low-cost alternative to physical and
cognitive exercise in the hospital and potentially at the home. We believe that the
Nintendo Wii and gaming in general have much to offer to brain injury and stroke
rehabilitation. In particular, well-designed games can be highly motivating and
engaging – if only addictive – while promoting limb movement and social
interactions with friends, family and other people recovering from similar
impairment.
It is evident from the results reported in this paper that this type of technology
should be considered as an integral part of the rehabilitation process. However, the
potential and promise of such highly engaging gaming technologies would need to
have the interface controls and game interaction customised to map the patients’
physical limitations to game play.
Acknowledgements
The authors would like to thank all the subjects who have participated on the
study and to the occupational therapists at the Neurorehabilitation Unit of the
Royal Berkshire NHS Hospital in Reading who have conducted the Wii therapy
sessions.
References
BBC News (2008). “Wii boost for care homes residents”, 30 January 2008,
http://news.bbc.co.uk/1/hi/wales/south_west/7218103.stm
BBC News Health (2008). “Games therapy for burns victims”, 25 February 2008,
http://news.bbc.co.uk/1/low/health/7262552.stm
Broeks J.G., Lankhorst G.J., Rumping K., and Prevo A.J. (1999). The long-term outcome of
arm function after stroke: results of a follow up study, Disabil & Rehabil, 21(8): 357-364.
Harwin W.S., Patton J.L., and Edgerton V.R. (2006). Challenges and opportunities for
robot-mediated neurorehabilitation. Proceedings of the IEEE, 94(9): 1717-1726.
Hesse S., Schmidt H. and Werner C. (2006). Machines to support motor rehabilitation after
stroke: 10 years of experience in Berlin. Journal of Rehabilitation Research and
Development, 43(5): 671-678.
Hesse S., Schulte-Tigges G., Konrad M., Bardeleben A., and Werner C. (2003). Robotassisted arm trainer for the passive and active practice of bilateral forearm and wrist
movements in hemiparetic subjects. Arch Phys Med Rehabil, 84(6): 915–20.
Johnson M.J., Wisneski H.J., Anderson J., Nathan D., and Smith R. (2006). Development of
ADLER: The activities of the daily living exercise robot. IEEE-EMBS Biomedical
Robotics (BioRob 2006), February 2006, Pisa, Italy, pp. 881-886.
10
R.C.V. Loureiro, D. Valentine, B. Lamperd, C. Collin and W.S. Harwin
Johnson, M.J., Van der Loos, H.F.M, Burgar, C.G., and Leifer, L.J. (1999), Driver’s SEAT:
simulation environment for arm therapy, ICORR’99: International Conference on
Rehabilitation Robotics. Stanford, CA, USA, pp227-234.
Krebs H.I., Hogan N., Volpe B.T., Aisen M.L., Edelstein L., and Diels C. (1999). Robotaided neuro-rehabilitation in stroke: three-year follow-up, ICORR’99: International
Conference on Rehabilitation Robotics, Stanford, CA, USA, pp. 34-41.
Lee J.C. (2008). Hacking the Nintendo Wii Remote. IEEE Pervasive Computing, Vol. 7, 3,
pp. 39-45.
Loureiro R., Amirabdollahian F., Topping M., Driessen B., and Harwin W. (2003). Upper
Limb Mediated Stroke Therapy – GENTLE/s Approach. In Special Issue on
Rehabilitation Robotics, Journal of Autonomous Robots, Springer, 15 (1), pp. 35-51
(ISSN 0929-5593).
Loureiro, R.C.V., and Harwin, W. S. (2007). Reach & Grasp Therapy: Design and Control
of a 9-DOF Robotic Neuro-rehabilitation System. In Proceedings of IEEE 10th
International Conference on Rehabilitation Robotics (ICORR 2007), Noordwijk, the
Netherlands, June 12-14, pp. 757-763.
Loureiro, R.C.V., Johnson, M., and Harwin, W.S. (2006). Collaborative tele-rehabilitation: a
strategy for increasing engagement. In Proceedings of the first IEEE International
Conference on Biomedical Robotics and Biomechatronics. February 20-22, Pisa, Italy.
Loureiro, R.C.V., Lamperd B., Collin C. and Harwin, W.S. (2009). Reach & Grasp Therapy:
Effects of the Gentle/G System Assessing Sub-acute Stroke Whole-arm Rehabilitation. In
Proceedings of IEEE 11th International Conference on Rehabilitation Robotics (ICORR
2009), Kyoto, Japan, June 23-26, pp. 755-760.
Lum P.S., Van der Loos M., Shor P., and Burgar C.G. (1999). A robotic system for upper
limb exercises to promote recovery of motor function following stroke. ICORR’99:
International Conference on Rehabilitation Robotics, Stanford, CA, USA, pp. 235-239.
Nef T., and Riener R. (2005). ARMin - Design of a novel arm rehabilitation robot. 9th IEEE
Conference on Rehabilitation Robotics, Chicago, USA, pp. 57-60.
Patton J.L., Stoykov M.E., Kovic M., and Mussa-Ivaldi F.A. (2006). Evaluation of robotic
training forces that either enhance or reduce error in chronic hemiparetic stroke
survivors," Exp Brain Res, 168: 368-383.
Reinkensmeyer D.J., Kahn L.E., Averbuch M., McKenna-Cole A., Schmit B.D., and Rymer
W.Z. (2000). Understanding and treating arm movement impairment after chronic brain
injury: progress with the ARM guide. Journal of Rehabilitation Research and
Development, 37: 653-62.
Rizzo A., and Kim G.J. (2005). “A SWOT analysis of the field of virtual reality
rehabilitation and therapy”, Presence: Teleoperators and Virtual Environments. Vol. 14,
pp. 119-146.
Shadmehr R. and Mussa-Ivaldi F.A. (1994). “Adaptive representation of dynamics during
learning of a motor task”, Journal of Neuroscience, Vol. 14, pp. 3208-3224.
Sukal T.M., Ellis M.D., and Dewald J.P.A. (2005). Dynamic characterization of upper limb
discoordination following hemiparetic stroke. 9th IEEE International Conference on
Rehabilitation Robotics, Chicago, USA, 519-521.
Gaming and Social Interactions in the Rehabilitation of Brain Injuries: A Pilot Study with
the Nintendo Wii Console
11
Title of paper:
Gaming and Social Interactions in the Rehabilitation of Brain Injuries: A Pilot
Study with the Nintendo Wii Console
Names of authors:
R.C.V. Loureiro1, D. Valentine2, B. Lamperd2, C. Collin2 and W.S. Harwin1
Contact Author: R.C.V. Loureiro
Email: R.C.V.Loureiro@reading.ac.uk
Address:
1
The University of Reading, School of Systems Engineering, Whiteknights
Campus, Reading RG6 6AY (phone: +441183788219; fax: +441183788220)
2
Royal Berkshire NHS Foundation Trust, London Road, Reading, RG1 5AN, UK
Submission type: formal paper
File name : Loureiro_et_al_CWUAAT2010_REVIEWED.doc
Version of Word used: 97-2003