Development of a
robotic system with
an augmented reality
interface for
rehabilitation of brain
injured individuals
James L. Patton , Ph.D. (RIC)
Robert Kenyon , Ph.D. (EVL at UIC)
Chris Scharver, MS.
Greg Dawe, MS
Ferdinando Mussa-Ivaldi, Ph.D.
Mary Ellen Phillips, MS OT
Kathy Stubblefield, OT
Rita Bode, Ph.D.
Patton Project Summary for Mars RERC Advisory Board Meeting, September 13-14, 2004
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Northwestern
University
`
MARS•RERC
Electronic
Visualization
Lab at UIC
19 cm
5.5 cm
Reachable
Workspace
Boundary
14 cm
y
x
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Northwestern
University
Electronic
Visualization
Lab at UIC
A. Unperturbed baseline
C. Final training
B. Early training
#2(60)
D. After-effects
E. Final washout
J. L. Patton, M. E. Phillips-Stoykov, M. Stojakovich,
W. Rymer, and F. A. Mussa-Ivaldi,
(Stroke"Performance
Subject sa38)
Improves With Force-Fields That Enhance Error In
Chronic Hemiparetic Stroke Survivors,"
conditionally accepted, EBR, 2004.
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Northwestern
University
Electronic
Visualization
Lab at UIC
Results on a stroke subject
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Northwestern
University
Electronic
Visualization
Lab at UIC
Distorting Vision: Stroke
Rossetti, Y., G. Rode, et al. (1998). "Prism adaptation to a rightward
optical deviation rehabilitates left hemispatial neglect." Nature, 395(6698): 166-9
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Northwestern
University
Electronic
Visualization
Lab at UIC
Adaptive Training
Recent Techniques that are Promising
• Error augmentation
• Custom-designed force fields
• Obstacle avoidance
• Visual distortions
• Sensory crossover
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Northwestern
University
Electronic
Visualization
Lab at UIC
Reachable
Workspace
Boundary
y
x
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Northwestern
University
Electronic
Visualization
Lab at UIC
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Northwestern
University
Electronic
Visualization
Lab at UIC
PARIS video
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Northwestern
University
Electronic
Visualization
Lab at UIC
C. Final training
Results:
healthy subject
A. Unperturbed baseline
D. After-effects
B. Early training
E. Final w ashout
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Northwestern
University
(Subject pilot2.1)
Electronic
Visualization
Lab at UIC
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Northwestern
University
Electronic
Visualization
Lab at UIC
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Northwestern
University
Electronic
Visualization
Lab at UIC
WREX



Passive (elastic elements)
Anti-gravity
4 DOF
Acknowledgments
Y. Wei
W. Z. Rymer
M. E. Phillips
M. Stojakovich
R. Haner
C. Malecka
P. Shah
C. Raasch
M. Peshkin
L. Kahn
D. Sha
Sensory Motor
Performance
Program (SMPP)
AHA 0330411Z
NIH R24 HD39627
NIH 5 T32 HD07418
NIH 5 RO1 NS 35673
NIH F32HD08658
NIDRR RERC 0330411Z
Falk Trust
Rehabilitation Institute of
Chicago
Northwestern University
Medical and
Engineering Schools
Rehabilitation
Institute of
Chicago
Northwestern
University
Electronic
Visualization
Lab at UIC
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Northwestern
University
Electronic
Visualization
Lab at UIC
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Northwestern
University
Electronic
Visualization
Lab at UIC
Adaptation
Capacity
25
20
15
10
5
0
healthy
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
stroke
Northwestern
University
Electronic
Visualization
Lab at UIC
Fugyl Meyr
Fugyl Meyr grand total
60
50
R2 = 0.0512
40
30
20
10
0
-5
0
Sensory Motor
Performance
Program (SMPP)
5
10
Rehabilitation
Institute of
Chicago
15
20
Northwestern
adaptation University
index
25
30
Electronic
Visualization
Lab at UIC
Sensory crossover
Vision
rotation only
Force field
supplement
(mixed)
Visually
Rotated
Targets
Baseline
0.1 m
Initial
Exposure
Final
Training
Washout
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Northwestern
University
Wei,Electronic
& Patton,
in review
Visualization
Lab at UIC
Vision Force field
rotation supplement
(mixed)
only
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Northwestern
University
Electronic
Visualization
Lab at UIC
Training
forces
reduced
error
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Training
forces
magnified
error
Northwestern
University
Electronic
Visualization
Lab at UIC
0.3
0.2
0.3
0.1
Control
Sensory Motor
Performance
Program (SMPP)
*2
Rehabilitation
Institute of
Chicago
Offset
Northwestern
University
*3.1
Error Improvement (m)
Time Constant of learning
(# movements)
Error Augmentation speeds up
& increases learning in healthy individuals
0
Electronic
Visualization
Lab at UIC
A
Early
exposure
ALTERING
MOVEMENTS
IN THE
HEALTHY
AfterC
Effects
B
Late
Training
Late
D
Washout
Mussa-Ivaldi, F.A., Patton, J.
L., IEEE International
Conference on Robotics and
Automation, 2000, San
Francisco, CA
Sensory Motor
Performance
Program (SMPP)
Rehabilitation
Institute of
Chicago
Northwestern
University
Electronic
Visualization
Lab at UIC
Library of Rehabilitation Programs
Robotic program
Target population
Expected outcome
Measure(s)
The robot measures the distance.
Motion analysis of the arm measures
Passive Range of motion assessment:
Therapist moves the arm toward 16 targets the
periphery of the workspace while patient is
relaxed.
All patients.
Active Range of motion assessment: Subject
attempts to reach to targets at the extent of the
workspace
All patients.
Same
Strength assessment: Ability to isometrically
push is 8 different directions at 4 different
positions in the workspace.
All patients.
The robot measures the forces and
Passive assist treatment: Motions are
externally imposed while the patient can remain
relaxed. [Accomplished with existing servo
software in phase 1].
All patients.
Maintenance or improvement of range of
motion at joints.
Same as above, plus:
For stroke subjects, Proximal movements
in the Wolf motor function Test and the
Upper extremity portion of the Fugl Meyer
exam **
For spinal cord subjects, the QIF and FIM
**
Active-assist treatment: This involves an
attempt by the patient to move a joint or
limb to a target, with external assisting
forces applied only as needed.
Stroke sub-acute (weeks following
injury) and SCI patients when the
patient cannot complete a desired
movement through the full range of
motion independently.
Improved ability to perform several
functional activities*. Increased active range
of motion, demon-strated by the patient and
the amount of assistance to go through full
range of motion will be decreased.
Same as above
Null forces treatment: The patient is un
impeded while attempting to move a joint or
limb to a target
Patients that have some movement
capabilities.
Improved ability to perform several
functional activities*
Same as above, plus, increased strength
Active Resist treatment: Resist using a viscous
and/or spring-like resistance as long as
subjects do not deviate from the desired
trajectory.
Patients that have some movement
capabilities.
Same as above
Same as above, plus, increased strength.
Sensory Motor
Performance
Program (SMPP)
individual joint excursions.
directions of the forces
Rehabilitation
Institute of
Chicago
Northwestern
University
Electronic
Visualization
Lab at UIC