Visualization of
Electronic Health Records
Ben Shneiderman
ben@cs.umd.edu
@benbendc
Founding Director (1983-2000), Human-Computer Interaction Lab
Professor, Department of Computer Science
Member, Institute for Advanced Computer Studies
University of Maryland
College Park, MD 20742
Visualization of
Electronic Health Records
@benbendc
University of Maryland
College Park, MD 20742
Interdisciplinary research community
- Computer Science & Info Studies
- Psych, Socio, Poli Sci & MITH
(www.cs.umd.edu/hcil)
Design Issues
•
•
•
•
•
Input devices & strategies
• Keyboards, pointing devices, voice
• Direct manipulation
• Menus, forms, commands
Output devices & formats
• Screens, windows, color, sound
• Text, tables, graphics
• Instructions, messages, help
Collaboration & Social Media
Help, tutorials, training
• Visualization
Search
www.awl.com/DTUI
Fifth Edition: 2010
HCI Pride: Serving 5B Users
Mobile, desktop, web, cloud
 Diverse users: novice/expert, young/old, literate/illiterate,
abled/disabled, cultural, ethnic & linguistic diversity, gender,
personality, skills, motivation, ...
 Diverse applications: E-commerce, law, health/wellness,
education, creative arts, community relationships, politics,
IT4ID, policy negotiation, mediation, peace studies, ...
 Diverse interfaces: Ubiquitous, pervasive, embedded, tangible,
invisible, multimodal, immersive/augmented/virtual, ambient,
social, affective, empathic, persuasive, ...
Information Visualization & Visual Analytics
•
Visual bands
• Human percle
• Trend, clus..
• Color, size,..
•
Three challe
• Meaningful vi
• Interaction: w
• Process mo
1999
Information Visualization & Visual Analytics
•
Visual bandwidth is enormous
• Human perceptual skills are remarkable
• Trend, cluster, gap, outlier...
• Color, size, shape, proximity...
•
Three challenges
• Meaningful visual displays of massive da
• Interaction: widgets & window coordinati
• Process models for discovery
1999
2004
Information Visualization & Visual Analytics
•
Visual bandwidth is enormous
• Human perceptual skills are remarkable
• Trend, cluster, gap, outlier...
• Color, size, shape, proximity...
•
Three challenges
• Meaningful visual displays of massive data
• Interaction: widgets & window coordination
• Process models for discovery
1999
2004
2010
Treemap: Gene Ontology
+ Space filling
+ Space limited
+ Color coding
+ Size coding
- Requires learning
(Shneiderman, ACM Trans. on Graphics, 1992 & 2003)
www.cs.umd.edu/hcil/treemap/
Treemap: Smartmoney MarketMap
www.smartmoney.com/marketmap
Market falls steeply Feb 27, 2007, with one exception
Market mixed, February 8, 2008
Energy & Technology up, Financial & Health Care down
Market rises, September 1, 2010, Gold contrarians
Treemap: WHC Emergency Room
(6304 patients in Jan2006)
Group by Admissions/MF, size by service time, color by age
Treemap: WHC Emergency Room
(6304 patients in Jan2006) (only those service time >12 hours)
Group by Admissions/MF, size by service time, color by age
Information Visualization: Mantra
•
•
•
•
•
•
•
•
•
•
Overview, zoom & filter, details-on-demand
Overview, zoom & filter, details-on-demand
Overview, zoom & filter, details-on-demand
Overview, zoom & filter, details-on-demand
Overview, zoom & filter, details-on-demand
Overview, zoom & filter, details-on-demand
Overview, zoom & filter, details-on-demand
Overview, zoom & filter, details-on-demand
Overview, zoom & filter, details-on-demand
Overview, zoom & filter, details-on-demand
SciViz .
•
•
•
1-D Linear
2-D Map
3-D World
Document Lens, SeeSoft, Info Mural
InfoViz
Information Visualization: Data Types
•
•
•
•
Multi-Var
Temporal
Tree
Network
Spotfire, Tableau, Qliktech, Visual Insight
infosthetics.com
flowingdata.com
visual.ly
GIS, ArcView, PageMaker, Medical imagery
CAD, Medical, Molecules, Architecture
LifeLines, TimeSearcher, Palantir, DataMontage
Cone/Cam/Hyperbolic, SpaceTree, Treemap
Pajek, UCINet, NodeXL, Gephi, Tom Sawyer
visualcomplexity.com
perceptualedge.com
visualizing.org
eagereyes.org
datakind.org
infovis.org
Obama Unveils “Big Data” Initiative (3/2012)
Big Data challenges:
• Developing scalable algorithms
for processing imperfect data
in distributed data stores
•
Creating effective humancomputer interaction tools
for facilitating rapidly
customizable visual reasoning
for diverse missions.
http://www.whitehouse.gov/sites/default/files/microsites/ostp/big_data_press_release_final_2.pdf `
EHRs: Temporal categorical data
• A type of time series
Numerical
Stock: Microsoft
04/26/2010 10:00
04/26/2010 10:15
04/26/2010 10:30
04/26/2010 10:45
04/26/2010 11:00
31.03
31.01
31.02
31.08
31.16
Event
Category
Event
Patient ID: 45851737
12/02/2008 14:26 Arrival
12/02/2008 14:36 Emergency
12/02/2008 22:44 ICU
12/05/2008 05:07 Floor
12/14/2008 06:19 Exit
Time
Arrival
Emergency
ICU
Floor
Exit
Patient Histories: Our Research
Tool
Event
Types
Records
Display
LifeLines
Points,
Intervals
One
Individual
LifeLines2
Points
Many
Individual,
Summary
Similan
Points
Many
Individual
LifeFlow
Points
Many
Individual,
Aggregate
EventFlow
Points,
Intervals
Many
Individual,
Aggregate
www.cs.umd.edu/hcil/toolname
Patient Histories: Our Research
Tool
Event
Types
Records
Display
LifeLines
Points,
Intervals
One
Individual
LifeLines2
Points
Many
Individual,
Summary
Similan
Points
Many
Individual
LifeFlow
Points
Many
Individual,
Aggregate
EventFlow
Points,
Intervals
Many
Individual,
Aggregate
www.cs.umd.edu/hcil/toolname
LifeLines: Patient Histories
www.cs.umd.edu/hcil/lifelines
Patient Histories: Our Research
Tool
Event
Types
Records
Display
LifeLines
Points,
Intervals
One
Individual
LifeLines2
Points
Many
Individual,
Summary
Similan
Points
Many
Individual
LifeFlow
Points
Many
Individual,
Aggregate
EventFlow
Points,
Intervals
Many
Individual,
Aggregate
www.cs.umd.edu/hcil/toolname
LifeLines2: Align-Rank-Filter & Summarize
www.cs.umd.edu/hcil/lifelines
LifeLines2: Align-Rank-Filter & Summarize
www.cs.umd.edu/hcil/lifelines2
Patient Histories: Our Research
Tool
Event
Types
Records
Display
LifeLines
Points,
Intervals
One
Individual
LifeLines2
Points
Many
Individual,
Summary
Similan
Points
Many
Individual
LifeFlow
Points
Many
Individual,
Aggregate
EventFlow
Points,
Intervals
Many
Individual,
Aggregate
www.cs.umd.edu/hcil/toolname
Similan: Search
www.cs.umd.edu/hcil/similan
Patient Histories: Our Research
Tool
Event
Types
Records
Display
LifeLines
Points,
Intervals
One
Individual
LifeLines2
Points
Many
Individual,
Summary
Similan
Points
Many
Individual
LifeFlow
Points
Many
Individual,
Aggregate
EventFlow
Points,
Intervals
Many
Individual,
Aggregate
www.cs.umd.edu/hcil/toolname
LifeFlow: Aggregation Strategy
Temporal
Categorical Data
(4 records)
LifeLines2 format
Tree of Event
Sequences
LifeFlow Aggregation
www.cs.umd.edu/hcil/lifeflow
LifeFlow: Interface with User Controls
Patient Histories: Our Research
Tool
Event
Types
Records
Display
LifeLines
Points,
Intervals
One
Individual
LifeLines2
Points
Many
Individual,
Summary
Similan
Points
Many
Individual
LifeFlow
Points
Many
Individual,
Aggregate
EventFlow
Points,
Intervals
Many
Individual,
Aggregate
www.cs.umd.edu/hcil/toolname
EventFlow: Original Dataset
LABA_ICSs Merged
SABAs Merged
Align by First LABA_ICS
Reduce Window Size
Original Dataset
Discovery Process: Systematic Yet Flexible
Preparation
• Own the problem & define the schedule
• Data cleaning & conditioning
• Handle missing & uncertain data
• Extract subsets & link to related information
Discovery Process: Systematic Yet Flexible
Preparation
• Own the problem & define the schedule
• Data cleaning & conditioning
• Handle missing & uncertain data
• Extract subsets & link to related information
Purposeful exploration – Hypothesis testing
• Range & distribution
• Relationships & correlations
• Clusters & gaps
• Outliers & anomalies
• Aggregation & summary
• Split & trellis
• Temporal comparisons & multiple views
• Statistics & forecasts
Discovery Process: Systematic Yet Flexible
Preparation
• Own the problem & define the schedule
• Data cleaning & conditioning
• Handle missing & uncertain data
• Extract subsets & link to related information
Purposeful exploration – Hypothesis testing
• Range & distribution
• Relationships & correlations
• Clusters & gaps
• Outliers & anomalies
• Aggregation & summary
• Split & trellis
• Temporal comparisons & multiple views
• Statistics & forecasts
Situated decision making - Social context
• Annotation & marking
• Collaboration & coordination
• Decisions & presentations
UN Millennium Development Goals
To be achieved by 2015
• Eradicate extreme poverty and hunger
• Achieve universal primary education
• Promote gender equality and empower women
• Reduce child mortality
• Improve maternal health
• Combat HIV/AIDS, malaria and other diseases
• Ensure environmental sustainability
• Develop a global partnership for development
30th Anniversary!!!
www.cs.umd.edu/hcil
@benbendc
Office of National Coordinator: SHARP
Strategic Health IT Advanced Research Projects
- Security of Health Information Technology
- Patient-Centered Cognitive Support
- Healthcare Application and Network Platform Architectures
- Secondary Use of EHR Data
Univ of Maryland HCIL tasks
- Missing Laboratory Reports
- Medication Reconciliation
- Wrong Patient Errors
www.cs.umd.edu/hcil/sharp
Medication Reconciliation: Current Form
Univ of Maryland HCIL tasks
- Missing Laboratory Reports
- Medication Reconciliation
- Alarms and Alerts Management
www.cs.umd.edu/hcil/sharp
www.youtube.com/watch?v=ZGf1EiuIIIM
Twinlist: Medication Reconciliation
“Best reconciliation app
I have ever seen”
Dr. Shawn Murphy, PartnersHealthcare & Harvard Medical
“Super-cool demo”
Dr. Jonathan Nebeker, Univ of Utah & VA
“Twinlist concept is brilliant”
Dr. Kevin Hughes, Harvard Medical School
Tiffany Chao, Catherine Plaisant, Ben Shneideman
Based on class project of : Leo Claudino, Sameh Khamis, Ran Liu, Ben London, Jay Pujara
Students of CMSC734 Information Visualization class
www.youtube.com/watch?v=YoSxlKl0pCo
Twinlist: Medications Grouped
Reducing Wrong Patient Errors:
Animated Transitions & Photos
Reducing Wrong Patient Errors:
Animated Transitions & Photos
Reducing Wrong Patient Errors:
Animated Transitions & Photos
Reducing Wrong Patient Errors:
Animated Transitions & Photos
Error Recognition Rate for each Group
63%
0.7
63%
0.6
43%
0.5
43%
36%
0.4
36%
0.3
0.2
7%
7%
0.1
0
Control
Control
Animation
Animation
Photo
Photo
Combined
Combined
The combination of animation & photo resulted in a significant increase
in error recognition rate relative to the control & animation groups 
Dramatic implications for commercial systems
(Taieb-Maimon, Plaisant & Shneiderman, 2012)
UI Techniques to Reduce Selection Errors
•
•
•
•
Highlight on Departure + Animation
Re-sort lists, Group by attributes
Show floor plan
Larger fonts + Space between rows
UI Techniques to Reduce Selection Errors
EventFlow Team: Oracle support
www.cs.umd.edu/hcil/eventflow
www.umdrightnow.umd.edu/news/umd-research-team-developing-powerful-data-visualization-tool-support-oracle