An Advanced User Interface for Pattern
Recognition in Medical Imagery:
Interactive Learning, Contextual Zooming,
and Gesture Recognition
Joshua R. New
Knowledge Systems Laboratory
Jacksonville State University
Knowledge Systems Lab
JN 5/29/2016
Outline
• Introduction
• Techniques: Segmentation, Magnification,
Exploration
• Solutions:
– Interactive Learning
– Contextual Zooming
– Gesture Recognition
• Conclusions
Knowledge Systems Lab
JN 5/29/2016
Introduction
Medical imagery…
• Consists of millions of images produced
annually which doctors must gather and
analyze
• Entails several modalities for each patient,
such as MRI, CT, and PET
Refine techniques for facilitating
comprehension of this data
Knowledge Systems Lab
JN 5/29/2016
Outline
• Introduction
• Techniques: Segmentation,
Magnification, Exploration
• Solutions:
– Interactive Learning
– Contextual Zooming
– Gesture Recognition
• Conclusions
Knowledge Systems Lab
JN 5/29/2016
Techniques
• Common techniques for facilitating data
comprehension:
– Segmentation – Labeling of images
– Magnification – Precision viewing
– Exploration – Interacting intuitively with
complex, 3D data
Knowledge Systems Lab
JN 5/29/2016
Why Segmentation?
• Doctors and radiologists:
– Spend several hours daily analyzing
patient images (ie. MRI scans of the brain)
– Search for patterns in images that are
standard and well-known to doctors
• Why not have the doctor teach the
computer to find these patterns in the
images?
Knowledge Systems Lab
JN 5/29/2016
Why Magnification?
• Doctors and radiologists:
– Must be able to precisely view and select
regions/pixels of the image to train the
computer
– Can easily lose where they are looking in
the image when using magnification
• Why not use visualization techniques to
preserve context while allowing precise
selections?
Knowledge Systems Lab
JN 5/29/2016
Why Exploration?
• Doctors and radiologists:
– Need to intuitively interact with the system
to maximize task performance
– Need to perform this interaction while
being unencumbered
• Why not use vision-based recognition to
allow interaction with the data?
Knowledge Systems Lab
JN 5/29/2016
Outline
• Introduction
• Techniques: Segmentation, Magnification,
Exploration
• Solutions:
– Interactive Learning
– Contextual Zooming
– Gesture Recognition
• Conclusions
Knowledge Systems Lab
JN 5/29/2016
Problems & Solutions
• Problem #1: Segmentation
• Solution #1: Interactive Learning
• Problem #2: Magnification
• Solution #2: Contextual Zoom
• Problem #3: Exploration
• Solution #3: Gesture Recognition
Knowledge Systems Lab
JN 5/29/2016
Platform
• Med-LIFE:
– “L”earning of MRI image patterns
– “I”mage “F”usion of multiple MRI images
– “E”xploration of the fusion and learning
results in an intuitive 3D environment
• Images used from “The Whole Brain Atlas”
– http://www.med.harvard.edu/AANLIB/home.html
Knowledge Systems Lab
JN 5/29/2016
Outline
• Introduction
• Techniques: Segmentation, Magnification,
Exploration
• Solutions:
– Interactive Learning
– Contextual Zooming
– Gesture Recognition
• Conclusions
Knowledge Systems Lab
JN 5/29/2016
Simplified Fuzzy ARTMAP
• Simplified Fuzzy
ARTMAP (SFAM)
– An AI neural network
(NN) system
– Capable of online,
incremental learning
– Takes seconds for tasks
that take
backpropagation NNs
days or weeks to
perform
Knowledge Systems Lab
JN 5/29/2016
Vector-based Learning
• Two “vectors” are sent to this system for
learning:
– Input feature vector provides the data from
which SFAM can learn
– ‘Teacher’ signal indicates whether that
vector is an example or counterexample
Knowledge Systems Lab
JN 5/29/2016
Feature Vector
• Pixel values from images (16 for each slice)
Knowledge Systems Lab
JN 5/29/2016
Learning Visualization
• Vector-based graphic visualization of learning
Category 1 - 2 members
y
Array of Pixel
Values
0.30
0.45
Category 2 - 1 member
Category 4 - 3 members
x
Knowledge Systems Lab
JN 5/29/2016
Learning Associations
Full Results
T2Results
Detailed
Knowledge Systems Lab
JN 5/29/2016
Varying Vigilance
• Only one tunable parameter – vigilance
– Vigilance can be set from 0 to 1 and corresponds to the
generality by which things are classified
(ie. vig=0.3=>human, vig=0.6=>male, 0.9=>Joshua New)
0.675
0.75
0.825
Knowledge Systems Lab
JN 5/29/2016
Input Order Dependence
• SFAM is sensitive to the order of the inputs
Vector 1
Category 1 - 2 members
Vector 2
Category 2 - 1 member
y
Vector 3
Category 4 - 3 members
x
JN 5/29/2016
Knowledge Systems Lab
Heterogeneous Network
• Voting scheme of 5 Heterogeneous
SFAM networks to overcome vigilance
and input order dependence
– 3 networks: random input order, set vigilance
– 2 networks: 3rd network order, vigilance ± 10%
Knowledge Systems Lab
JN 5/29/2016
Network Segmentation Results
Knowledge Systems Lab
JN 5/29/2016
Segmentation Results
Threshold results
Trans-slice results
Overlay results
Knowledge Systems Lab
JN 5/29/2016
Segmentation Screenshot
Knowledge Systems Lab
JN 5/29/2016
Interactive Learning
System Demonstration
Knowledge Systems Lab
JN 5/29/2016
Segmentation Solution
• Doctors and radiologists:
– Spend several hours daily analyzing patient
images (ie. MRI scans of the brain)
– Search for patterns in images that are standard
and well-known to doctors
• Solution:
– Doctors and radiologists can teach the
computer to recognize abnormal brain tissue
– They can refine the learning systems results
interactively
Knowledge Systems Lab
JN 5/29/2016
Outline
• Introduction
• Techniques: Segmentation, Magnification,
Exploration
• Solutions:
– Interactive Learning
– Contextual Zooming
– Gesture Recognition
• Conclusions
Knowledge Systems Lab
JN 5/29/2016
Zooming Approaches
Inset Overlay
Chip Window
Knowledge Systems Lab
JN 5/29/2016
Research & Business
• Carpendale PhD Thesis
– Elastic Presentation Space – rubber sheet
images via mathematical constructs
• IDELIX (www.idelix.com)
– Pliable Display Technology – software
development kit (SDK) product
– Boeing: 20% increase in productivity
Knowledge Systems Lab
JN 5/29/2016
Zoom Visualization
Contextual Zoom
Wireframe View
Knowledge Systems Lab
JN 5/29/2016
Contextual Zoom
System Demonstration
Knowledge Systems Lab
JN 5/29/2016
System Comparison
Previous System Zoom Overlay
Contextual Zoom
Knowledge Systems Lab
JN 5/29/2016
Magnification Solution
• Doctors and radiologists:
– Must be able to precisely view and select
regions/pixels of the image to train the computer
– Can easily lose where they are looking in the
image when using magnification
• Solution
– They can precisely select targets/non-targets
– They can zoom for precision while maintaining
context of the entire image
– The interface facilitates task performance
through interactive display of segmentation
results
Knowledge Systems Lab
JN 5/29/2016
Outline
• Introduction
• Techniques: Segmentation, Magnification,
Exploration
• Solutions:
– Interactive Learning
– Contextual Zooming
– Gesture Recognition
• Conclusions
Knowledge Systems Lab
JN 5/29/2016
Motivation
• Gesturing is a natural form of
communication:
– Gesture naturally while talking
– Babies gesture before they can talk
• Interaction problems with the mouse:
– Have to locate cursor
– Hard for some to control (Parkinsons or
people on a train)
– Limited forms of input from the mouse
Knowledge Systems Lab
JN 5/29/2016
Motivation
• Problems with the Virtual Reality Glove
as a gesture recognition device:
– Reliability
– Always connected
– Encumbrance
Knowledge Systems Lab
JN 5/29/2016
System Diagram
User
Rendering
Hand
Movement
Update Object
User Interface
Display
Gesture
Recognition
System
Image
Capture
Image Input
Standard
Web Camera
Knowledge Systems Lab
JN 5/29/2016
System Performance
• System:
•
OpenCV and IPL libraries (from Intel)
• Input:
•
•
640x480 video image
Hand calibration measure
• Output:
•
•
•
•
Rough estimate of centroid
Refined estimate of centroid
Number of fingers being held up
Manipulation of 3D skull in QT interface in
response to gesturing
Knowledge Systems Lab
JN 5/29/2016
Calibration Measure
• Max hand size in x and y orientation
(number of pixels in 640x480 image)
Knowledge Systems Lab
JN 5/29/2016
Saturation Extraction
Saturation Channel Extraction
(HSL space):
Original Image
Hue
Lightness
Saturation
Knowledge Systems Lab
JN 5/29/2016
Gesture Recognition Pipeline
Knowledge Systems Lab
JN 5/29/2016
Gesture Recognition Pipeline
Knowledge Systems Lab
JN 5/29/2016
Gesture Recognition Pipeline
a) 0th moment of an image:
M 00   I ( x, y)
b) 1st moment for x and y of an
image, respectively:
M 10   x  I ( x, y)
M 01   y  I ( x, y)
c) 2nd moment for x and y of an
image, respectively:
2
M 20   x 2  I ( x, y)
M 02   y  I ( x, y)
d) Orientation of
image major axis:

arctan


M



2  1 1  xc yc 


 M 00



  M 2 0  x 2    M 0 2  y 2  
c  
c 
M
  M 00

  00
2
Knowledge Systems Lab
JN 5/29/2016
Gesture Recognition Pipeline
Radius  0.19 * ( HandSizeX  HandSizeY )
•
The finger-finding function sweeps
out a circle around the rCoM,
counting the number of white and
black pixels as it progresses
•
A finger is defined to be any 10+
white pixels separated by 17+ black
pixels (salt/pepper tolerance)
•
Total fingers is number of fingers
minus 1 for the hand itself
Knowledge Systems Lab
JN 5/29/2016
System Setup
System
Configuration
System
GUI Layout
Knowledge Systems Lab
JN 5/29/2016
Interaction Mapping
Gesture to Interaction Mapping
Number of Fingers:
2 – Roll Left
3 – Roll Right
4 – Zoom In
5 – Zoom Out
Knowledge Systems Lab
JN 5/29/2016
Gesture Recognition Demo
Knowledge Systems Lab
JN 5/29/2016
Exploration Solution
• Doctors and radiologists:
– Need to intuitively interact with the system to
maximize task performance
– Need to perform this interaction while being
unencumbered
• Solution
– Can use intuitive gesturing to interact with
complex, 3D data
– Can interact by simply moving their hand in
front of a camera, requiring no physical device
manipulation
Knowledge Systems Lab
JN 5/29/2016
Outline
• Introduction
• Techniques: Segmentation, Magnification,
Exploration
• Solutions:
– Interactive Learning
– Contextual Zooming
– Gesture Recognition
• Conclusions and Future Work
Knowledge Systems Lab
JN 5/29/2016
Interactive Learning
• Users can teach the computer to recognize
abnormal brain tissue
• They can refine the learning systems results
interactively
• They can save/load agents for background
diagnosis on a database of medical images
or to allow expert analysis in the absence of a
well-paid expert
Knowledge Systems Lab
JN 5/29/2016
Contextual Zoom
• They can zoom for precisely viewing and
selecting targets/non-targets while
maintaining context of the entire image
• The interface facilitates task performance
through interactive and customizable display
of segmentation results
• This system can be used with any 2D images
and even with 3D datasets with some minor
alterations
Knowledge Systems Lab
JN 5/29/2016
Gesture Recognition
• Can use intuitive gesturing to interact
with complex, 3D data
• Can interact by simply moving their
hand in front of a camera, requiring no
physical device manipulation
• Easily replicated and distributable
• Mapping gestures to interaction is an
independent stage
Knowledge Systems Lab
JN 5/29/2016
Gesture Recognition
• Dynamic Gesture Recognition
• Other interface applications include:
graspable interfaces, 3D avatar /
MoCap, multi-object manipulation in
virtual environments, and augmented
reality
Knowledge Systems Lab
JN 5/29/2016
Platform
• Med-LIFE integration effort
– Gesture Recognition has already been
integrated into Med-LIFE’s Exploration tab
– Contextual Zoom and Interactive learning
have been combined, but not yet
integrated into Med-LIFE’s learning tab
• Med-LIFE will function as a single
application for medical image analysis
Knowledge Systems Lab
JN 5/29/2016