Classification and Clustering of
Brain Pathologies from Motion
Data of Patients
in a Virtual Reality Environment
Via Machine Learning
Uri Feintuch, Hadassah- Hebrew University Medical Center
Larry Manevitz, University of Haifa,
Natan Silnitsky, University of Haifa
Data from:
Assaf Dvorkin, Northwestern University
Neuro computation laboratory day,
December 2011
Outline
•
•
•
•
•
•
•
•
Rehabilitation of Patients of Brain
Pathologies
Virtual Reality (VR) in Rehab
Research Goals
Techniques Used
Experiments
Architecture and Training
Results
Future directions
Brain pathologies
•
CVA - CerebroVascular Accident
(Stroke)
– Hemispatial neglect
•
TBI - Traumatic Brain Injury
Rehabilitation
•
Diagnosis
– Differential Diagnosis (e.g., Neglect vs.
Hemianopsia)
•
Evaluation
– Severity of deficit
– Progress during intervention
Example - Neglect
Traditional tools
• Star Cancellation
and their shortcomings…
• HD applied for and received back his
driver’s license, having shown intact visual
fields at Perimetry and no signs of neglect.
• HD scored 143/146 on his BIT test.
• Since obtaining the license, however, he
was involved in 9 car accidents, all
concerning the left side of his car.
(Deouell, Sacher & Soroker, 2005)
VR in Rehab (1)
• Virtual Mall
The way the patient views himself within
the virtual environment
A camera which films the patient and a
monitor which displays her
VR in Rehab
•
•
•
•
Replaces traditional methods
Ecological validity
Safety
Absolute control of stimuli
VR in Rehab (2)
• Assaf Dvorkin, Rehabilitation Institute of
Chicago, Northwestern Uni.
The VRROOM haptics/graphic system
A target in the field of view
VR in Rehab - Challenges
•
•
•
•
Human behavior is very complicated
Vast amount of information
geometry or physics formula (?)
Simplistic analysis (e.g., RT, % Errors)
Proposed solution
• Apply Machine Learning tools. Such tools
may detect patterns of behavior performed
within the Virtual Environment.
• In this work we used Artificial Neural
Networks (NN) Classifiers ,SVM, SOM and
k-means.
Research Goals
• Identify and differentiate between
meaningful clinical conditions
– Scarce data
– Perhaps noisy
• Broad spectrum conditions like neglect
– Mild, severe
– Use unsupervised learning approach
Machine Learning Techniques
Used
•
Supervised Learning
– Backpropogation NN
– SVM
•
Unsupervised Learning
– Kohonen
– K-means
2D Experiment
• Population: 54 HA, 11 CVA (without
neglect), 9 TBI, 25 HC.
• Data Encoding: Vector of hand movement
(dx,dy,dt)
NN Architecture for 2D
Output Layer
Hidden Layer
…(Full connectivity)…
dx
dy
dt
Data point (t)
dx
dy
dt
Data point (t+1)
dx
dy
dt
Data point (t+2)
Input Layer
dx
dy
dt
Data point (t+3)
dx
dy
dt
Data point (t+4)
NN Architecture for 2D (TBI vs. CVA)
Output Layer
…(Full connectivity)…
…(Full connectivity)…
dx
dy
Data point (t)
dt
dx
dy
Data point (t+1)
dt
dx
dy
dt
Data point (t+2)
Input Layer
dx
dy
dt
Data point (t+3)
dx
dy
dt
Data point (t+4)
Training for 2D
•
•
•
•
Levenberg-Marquardt
resilient back-propagation
300 epochs
Cross-validation
2D Results – 2 class
BP NN
Success Rate
By Patient
BP NN
Success Rate
By Data Point
Healthy/CVA
85%
90% (11500/12670)
Healthy/TBI
95%
95% (11167/11806)
TBI/CVA
72%
59% (1414/2393)
Populations
3D Experiment
• Population: 9 H, 9 N, 10 S
• Data Encoding: Vector of movement
(x,y,z,t)
• Only trials where movement occurred at all
• Phases
– Long vector: Entire trial from appearance of
stimulus (includes pre-movement data)
– Movement: Vector only from commencement
of movement
– Initial/Final segment – beginning/end of
movement
NN Architecture for 3D
• 1400 elements for a long vector (1400-5-1)
• 1000 elements for a movement vector
(1000-5-1)
• 130 elements for initial/final vectors (1305-1)
3D Data Set
• Population of Healthy, Neglect, Stroke
• Movement Vectors (x,y,z,t) of different
lengths
• Also tested on “snippets” for cross
platform
• Resilient back-propagation
• 50 to 300 epochs
3D Results – 2 class
Vector size
Populations
BP NN Success Rate
By Patient and
By Data Point
Including Reaction Time
(470 time steps)
From Start of Motion
(330 time steps)
Healthy/CVA
78%
62% (4010/6458)
Healthy/Neglect
89%
79% (4557/5791)
Neglect/CVA
72%
63% (3584/5703)
Healthy/CVA
78%
68% (4378/6458)
Healthy/Neglect
89%
78% (4515/5791)
Neglect/CVA
72%
66% (3737/5703)
3D Results – 2 class
Vector size
Populations
BP NN Success Rate
By Patient and
By Data Point
Small initial segment
(43 time steps)
Small final segment
(43 time steps)
Healthy/CVA
39%
43% (2791/6458)
Healthy/Neglect
83%
70% (4058/5791)
Neglect/CVA
78%
61% (3470/5703)
Healthy/CVA
67%
53% (3435/6458)
Healthy/Neglect
89%
61% (3547/5791)
Neglect/CVA
61%
52% (2962/5703)
Clustering for 3D
• Kohonen Self Organizational Map (SOM)
• Reproduced with K-means
Clustering for 3D
Clustering for 3D
• 2 Neurons
• 7 Neurons
• 200 Neurons
3D Results – 0 class
• Movement Vector, Neglect, (7 clusters)
• Movement, Healthy/Neglect, (7)
3D Results – 0 class
• Movement Vector, Neglect/CVA ,
(7 clusters)
Clustering for 2D
• Kohonen Self Organizational Map (SOM)
• Reproduced with K-means
2D Results – 0 class
• Healthy/CVA, (7 clusters)
• -> …
3D expriment - 1 class
• Architecture
• Movement vectors – 1000-200-1000
• initial/final vectors - 130-26-130
3D expriment - 1 class
• Architecture
• Movement vectors – 1000-200-1000
• initial/final vectors - 130-26-130
3D expriment - 1 class
• Threshold
choice
1 class results for 3D
Vector size
From Start of Motion
(330 time steps)
Training Set
Population
Compression NN
Average Success
Healthy
93%
CVA
Neglect
Non-Mild Neglect
69%
50%
76%
Severe Neglect
100%
1 class results for 3D
• Neglect classifier
for "Left targets
trials only" - 62%
• Non-Mild Neglect
classifier for "Left targets
trials only" - 83%
Combined Platforms
• Merging small samples from different
platforms. But…
Combined Platforms – 2 class
• (x,y,z) -> (x,y,0)
• "snippets"
• Experiments different data amounts
Combined Platforms – 2 class
Vector size
30 time steps
Training Set
Origin
VRROOM and
GestureXtreme
90 time steps
VRROOM only
GestureXtreme
only
BP NN
Average Success
75%
90%
50%
50%
Summary of results
• 2D experiment – Differential Diagnosis:
– Healthy vs. Patients
– TBI vs. CVA
• 3D experiment – DD + Evaluation
– Neglect vs CVA
– Clusters by severity
– 1 class classifiers (Severe Neglect)
Future Work
•
Merging data across platforms
•
Automatic Prognosis and Individualized
Treatment Protocols
– Construct models of patients with their
movement restrictions
– Run potential rehab protocols on the model
– Prognosis: via best results on model
– Apply best protocol to the patient
Acknowledgment
•
•
•
•
•
•
•
•
•
Assaf Dvorkin
Jim Patton
Eugene Mednikov
Debbie Rand
Rachel Kizony
Neta Erez
Meir Shahar
Patrice L. Weiss
The Caesarea Rothschild Institute