FINDING PATTERNS
IN TEMPORAL DATA
KRIST WONGSUPHASAWAT
TAOWEI DAVID WANG
CATHERINE PLAISANT
BEN SHNEIDERMAN
HUMAN-COMPUTER INTERACTION LAB
UNIVERSITY OF MARYLAND
FINDING PATTERNS
IN TEMPORAL DATA
KRIST WONGSUPHASAWAT
TAOWEI DAVID WANG
CATHERINE PLAISANT
BEN SHNEIDERMAN
HUMAN-COMPUTER INTERACTION LAB
UNIVERSITY OF MARYLAND
TEMPORAL CATEGORICAL
DATA
• A type of time series
Numerical
Stock: Microsoft
04/26/2010 10:00
04/26/2010 10:15
04/26/2010 10:30
04/26/2010 10:45
04/26/2010 11:00
31.03
31.01
31.02
31.08
31.16
Event
Category
Event
Patient ID: 45851737
12/02/2008 14:26
12/02/2008 14:36
12/02/2008 22:44
12/05/2008 05:07
12/14/2008 06:19
Arrival
Emergency
ICU
Floor
Exit
Time
Arrival
Emergency
ICU
Floor
Exit
TEMPORAL CATEGORICAL
DATA
Electronic Health Records: symptoms, treatment, lab test
Traffic incident logs: arrival/departure time of each unit
Student records: course, paper, proposal, defense, etc.
Others: web logs, usability study logs, etc.
10+ years work on temporal visualization
(mostly on Electronic Health Records)
LIFELINES
[Plaisant et al. 1998]
http://www.cs.umd.edu/hcil/lifelines
LifeLines – Single Patient
working with physicians at
WASHINGTON HOSPITAL CENTER
EXAMPLE DATA
• Patient transfers
ARRIVAL
EMERGENCY
ICU
INTERMEDIATE
Arrive the hospital
Emergency room
Intensive Care Unit
Intermediate Medical Care
FLOOR
EXIT-ALIVE
EXIT-DEAD
Normal room
Leave the hospital alive
Leave the hospital dead
TASKS
• Example: Finding “Bounce backs”
ICU
Floor
ICU
within 2 days
LIFELINES 2
RECORD
[Wang et al. 2008, 2009]
http://www.cs.umd.edu/hcil/lifelines2
Multiple
Records
ARF (Align-Rank-Filter)
Framework
Temporal Summary
LifeLines2 – Search and Visualize
ALIGNMENT
• Sentinel events as reference points
Time
June
Patient #45851737
July
August
Arrival
Emergency
ICU
Floor
Exit
Patient #43244997
Arrival
Emergency
ICU
Floor
Exit
ALIGNMENT (2)
• Time shifting
Time
0
Patient #45851737
1M
2M
Admit
Emergency
ICU
Floor
Exit
Patient #43244997
Admit
Emergency
ICU
Floor
Exit
SIMILAN
RECORD
[Wongsuphasawat & Shneiderman 2009]
http://www.cs.umd.edu/hcil/similan
Similan – Search by Similarity
Similan – Search by Similarity
FINDING “BOUNCE BACKS”
Before
After
• Much faster to specify new query
• Visualizing the results gives better
understanding
USER STUDIES: SEARCH
LifeLines2
Exact
MUST have A, B, C
Similan
Similarity-based
SHOULD have A, B, C
Query
Query
Record#2
Record#2
Record#1
Record#1
Record#3
Record#3
more
similar
USER STUDIES: SEARCH
LifeLines2
Exact
MUST have A, B, C
1
Similan
Similarity-based
SHOULD have A, B, C
Query
Query
Record#2
Record#2
Record#1
Record#1
Record#3
Record#3
more
similar
NEW STUFF
Needs for an overview -> LifeFlow!
TASKS
• Example: Finding “Bounce backs”
ICU
Floor
ICU
within 2 days
• Other questions
Arrival
?
?
ICU
?
LIFEFLOW
VISUALIZE
Display the
aggregation
RECORD
RECORD
RECORD
AGGREGATE
Merge multiple records into tree
AGGREGATE
• Aggregate by prefix
#1
#2
#3
#4
Example with 4 records
AGGREGATE
• Aggregate by prefix
#1
#2
#3
#4
VISUALIZE
• Inspired by the Icicle tree [Fekete 2004]
Number of files
VISUALIZE (2)
• Use horizontal axis to represent time
• Video
DEMO – LIFEFLOW
When the lines are combined into flow
FUTURE WORK
• Comparison
ICU
Floor
Intermediate
Jan-Mar 2008
ICU
Intermediate
April-June 2008
TAKE-AWAY MESSAGE
Information visualization is a powerful way
to explore temporal patterns.
You can work with us
on new case studies.
TEMPORAL CATEGORICAL
DATA
Electronic Health Records: symptoms, treatment, lab test
Traffic incident logs: arrival/departure time of each unit
Student records: course, paper, proposal, defense, etc.
Others: web logs, usability study logs, etc.
EXAMPLE – TRAFFIC
INCIDENTS
ACKNOWLEDGEMENT
DR. PHUONG HO, DR. MARK SMITH, DAVID ROSEMAN
WASHINGTON HOSPITAL CENTER
http://www.whcenter.org
NATIONAL INSTITUTES OF HEALTH (NIH) - GRANT
CA147489
http://www.nih.gov
MICHAEL PACK, MICHAEL VANDANIKER
CENTER FOR ADVANCED TRANPORTATION
TECHNOLOGY LAB
(CATT LAB)
http://www.cattlab.umd.edu
TAKE-AWAY MESSAGE
Information visualization is a powerful way
to explore temporal patterns.
You can work with us
on new case studies.
More demos this afternoon
{kristw, tw7, plaisant, ben}@cs.umd.edu
http://www.cs.umd.edu/hcil/temporalviz
Q&A
Questions?
{kristw, tw7, plaisant, ben}@cs.umd.edu
http://www.cs.umd.edu/hcil/temporalviz
THANK YOU
Thank you
BACKUP SLIDES
Junkyard...
LIFELINES2
• 8 case studies
– Bounce backs
– Step ups
– BIPAP
– Etc.
DR. P
LifeLines2’s Temporal Summary [Wang et al. 2009]
Continuum’s Histogram [Andre 2007]
Does not help exploring sequential patterns
Needs a new overview
USER STUDIES
• 8 Extensive case studies
• Compared LifeLines2 with Similan
– Learn advantages & disadvantages
• Drawing is preferred
• Clear cut off points is needed
• Working on improvements
– Flexible temporal search
SIMILAN
• Compared with LifeLines2 in an experiment
– Learn advantages & disadvantages
– Drawing is preferred
– No clear cut off points
• Working on improvements
– Flexible temporal search
LIFEFLOW
AGGREGATE
Merge multiple records into tree
VISUALIZE
Display the tree
APPROACHES
Exact Search
MUST have A, B, C
Similarity-based Search
SHOULD have A, B, C
Query
Query
Record#1
Record#2
Record#2
Record#1
Record#3
Record#3
more
similar
RESEARCH
QUESTION#1
MOTIVATION
PRELIM. +
PROPOSED WORK
RESEARCH
QUESTIONS
CONCLUSION
RESEARCH
QUESTION#2
PRELIM. +
PROPOSED WORK
EXPECTED CONTRIBUTIONS
1. Design of visual representations, user
interfaces and interaction techniques
2. Algorithms for flexible temporal search
3. Evaluation results
4. Open new directions for exploring temporal
categorical data
NEEDS FOR AN OVERVIEW
• We learn
NEEDS
Visualize overview
or show summary
Where should I start?
TEMPORAL VISUALIZATIONS
Background and related work
RELATED WORK
• Single record
Patient ID: 45851737
12/02/2008 14:26
12/02/2008 14:26
12/02/2008 22:44
12/05/2008 05:07
12/08/2008 10:02
12/14/2008 06:19
Arrival
Emergency
ICU
Floor
Floor
Exit
• E.g. LifeLines, MIDGAARD, etc.
Visualization
RELATED WORK (2)
• Multiple records
Patient
ID:
45851737
Patient
ID:
45851737
Patient
ID:
45851737
12/02/2008
14:26
Arrival
Patient
ID:
12/02/2008 14:26 45851737
Arrival
12/02/2008
14:26Emergency
Arrival
12/02/2008
14:26
12/02/2008
14:26
Arrival
12/02/2008 14:26 Emergency
12/02/2008
14:26ICUEmergency
12/02/2008
22:44
12/02/2008
14:26ICUEmergency
12/02/2008
22:44
12/02/2008
22:44
ICU
12/05/2008
05:07
Floor
12/02/2008
22:44
ICU
12/05/2008 05:07 Floor
12/05/2008
05:07Floor
Floor
12/08/2008
10:02
12/05/2008
05:07
Floor
12/08/2008 10:02 Floor
12/08/2008
10:02ExitFloor
12/14/2008
06:19
12/08/2008
10:02ExitFloor
12/14/2008
06:19
12/14/2008 06:19 Exit
12/14/2008 06:19 Exit
• E.g. LifeLines2, Continuum, ActiviTree, etc.
Visualization
Visualization
Visualization
Visualization
More space please....
INFORMATION VISUALIZATION
MANTRA
OVERVIEW FIRST,
ZOOM AND FILTER,
THEN DETAILS ON DEMAND
RELATED WORK (3)
• Multiple records
Patient
ID:
45851737
Patient
ID:
45851737
Patient
ID:
45851737
12/02/2008
14:26
Arrival
Patient
ID:
12/02/2008 14:26 45851737
Arrival
12/02/2008
14:26Emergency
Arrival
12/02/2008
14:26
12/02/2008
14:26
Arrival
12/02/2008 14:26 Emergency
12/02/2008
14:26ICUEmergency
12/02/2008
22:44
12/02/2008
14:26ICUEmergency
12/02/2008
22:44
12/02/2008
22:44
ICU
12/05/2008
05:07
Floor
12/02/2008
22:44
ICU
12/05/2008 05:07 Floor
12/05/2008
05:07Floor
Floor
12/08/2008
10:02
12/05/2008
05:07
Floor
12/08/2008 10:02 Floor
12/08/2008
10:02ExitFloor
12/14/2008
06:19
12/08/2008
10:02ExitFloor
12/14/2008
06:19
12/14/2008 06:19 Exit
12/14/2008 06:19 Exit
• E.g. LifeLines2, Continuum
Visualization
SEQUENTIAL PATTERNS
• Examples: “Bounce backs”
ICU
Floor
ICU
within 2 days
Patient #1
Patient #2
Patient #3
Patient #4
DESIGN AN OVERVIEW
• Sequential patterns
• Scalability vs. Loss of information