A Patient Care Digital Library
Personalized Search and Summarization
over Multimedia Information
Kathleen McKeown, Shih-Fu Chang, James
Cimino, Paul Clayton, Judith Klavans
Columbia University
1
Vision

Personalized access to distributed resources
information vetting
 information fusion
 information understanding


Provision of patient-specific information
at the point of patient care
 in terms that can be understood


Benefits: awareness of new procedures,
increased compliance
2
Features
Tailor search, presentation, and
summarization of online medical literature
 Exploit online patient record as user model
 Patients and health care providers
 Multimodal access and presentation over
multimedia resources

3
Scope of Research
“exciting,” “innovative,”
“broad,” “ambitious,” “complex”
Computer Science and Medical Informatics
 Links into libraries: CRIA
 Suite of existing component technologies
 Five year vision
 Active partners

4
Continuum

Current: existing individual components for
multiple applications


summarization of terrorist news, image search,
categorization of images, multimedia briefings,
PATCIS, patient mammography record extraction,
video summarization of news, query translation
Future: DLI2 enables
unification and transformation of components
 new level of capability
 integrated in a single application

5
Outline

Desmond Jordan


Why this research is needed
Kathy McKeown
An example—integration of components
 Scope, organization, management
 Role of partners and new developments


Jim Cimino


Clinical information infrastructure
Kathy McKeown

Budget
6
Rounds



Patient-centric
Current: Access
to clinical data
Missing:
Access to
literature that
fits patient
profile
7
Data follows the patient—why not
literature?
8
Ongoing Collaboration: Generation
of Multimedia Briefings
Voice: . . . . Ms. Jones is an 80 year old,
diabetic, hypertensive, female patient
of Dr. Smith undergoing CABG. . . .
9
User Study Group

Speech, text, and
graphics output
were useful
10
Steve Feiner:
Multimedia Video
If I recall correctly, I found the
image at the bottom in a we
page for patients with CHF. May
have something to do with fluid
buildup.

Physicians: residents learn by comparison

Patients:
Video helps them understand their diagnosis
 They like to investigate prognosis

11
Query
Steve Feiner:
ThisWhy
slide
differs from the previous one
should I prescribe amiodarone?
in that it has a real graphical history,
although not in the medical domain. It
may be better, since it means that all of
the material on the slide will be real.
WebSeek
BTW, this is a “comment”. You can
either delete it or make it invisible with
the “View” menu “comments”
command
History
Lexical
Navigation
Key Issues: User Interface

Query augmentation using the patient
record


Congestive Heart Failure (CHF), male, Base Heart
Rate (BHR) > 90, history of ischemia
Lucent: speech, IBM: query refinement,
interface for image/video search
14
Distributed Search
Antiarrhythmic
Drugs
NSF is planning to implement a new electronic
project reporting system through which proposers
will be able to produce the Results from Prior
Support section of
the proposal. This new system will be part of the
NSF FastLane system and will permit updating of
reports on each of the PI's NSF projects and will
avoid re-en the normal mechanism for project
reporting to NS
Documents
Amiodarone
Amiodarone
profile
profile
profile
Amiodarone
Treatme CHF
NSF is planning to
implement a new
electronic project reporting
system through which
proposers will be able to
produce the Results from
Prior Support section of
the proposal. This new
system will be part of the
NSF FastLane system and
will permit updating of
reports on each of the PI's
NSF projects and will
avoid re-entry
of information previously
provided by the PI, either
with the original proposal
submission or in updates
provided usingting process
and become the normal
mechanism for project
reporting to NS
Augmented query
b
Search
Amiodarone
CHF
male
BHR > 90
Echocardiogram
Echo 10/15/98
Echo 9/15/98
Patient
heart
sounds
Database of Videos
Echo 8/15/98
Echo 7/15/98
Amiodarone
CHF
male
BHR > 90
Journal Articles
Methods:
All patients had
advanced CHF with ...
Documents
Amiodarone
Amiodarone
Amiodarone
Amiodarone
NSF is planning to
implement a new
electronic project reporting
system through which
proposers will be able to
produce the Results from
Prior Support section of
the proposal. This new
system will be part of the
NSF FastLane system and
will permit updating of
reports on each of the PI's
NSF projects and will
avoid re-entry
of information previously
provided by the PI, either
with the original proposal
submission or in updates
provided usingting process
and become the normal
mechanism for project
reporting to NS
Categorization
&
Filtering
NSF is planning to implement a new
electronic project reporting system
through which proposers will be able
to produce the Results from Prior
Support section of
the proposal. This new system will
be part of the NSF FastLane system
and will permit updating of reports
on each of the PI's NSF projects and
will avoid re-entry
of information previously provided
by the PI, either with the original
proposal submission or in updates
provided usingting process and
become the normal mechanism for
project reporting to NS
Prognosis
Article
NSF is planning to implement a new
electronic project reporting system
through which proposers will be able
to produce the Results from Prior
Support section of
the proposal. This new system will
be part of the NSF FastLane system
and will permit updating of reports
on each of the PI's NSF projects and
will avoid re-entry
of information previously provided
by the PI, either with the original
proposal submission or in updates
provided usingting process and
become the normal mechanism for
project reporting to NS
Prognosis
Article
NSF is planning to implement a new
electronic project reporting system
through which proposers will be able
to produce the Results from Prior
Support section of
the proposal. This new system will
be part of the NSF FastLane system
and will permit updating of reports
on each of the PI's NSF projects and
will avoid re-entry
of information previously provided
by the PI, either with the original
proposal submission or in updates
provided usingting process and
become the normal mechanism for
project reporting to NS
Diagnosis
Article
.
.
.
Target Summaries
Prognosis articles:
amiodarone patient, BHR > 90
A significant reduction in sudden death was observed in
patients with a BHR > 90 beats/min treated with
amiodarone
 pacemaker patient, BHR  90
In patients with a BHR  90 beats/min, amiodarone-treated
patients had a mortality rate similar to those not treated
with amiodarone. One article found that pacing fails to
improve hemodynamic function, while another showed a
significantly better quality of life.

PERSIVAL Management Structure
Technology
Search
Gravano
User Interface
Johnson
Summarization
McKeown
Project Leader
McKeown
Clinical
Infrastructure
Cimino
Evaluation
Industry Partners
Klavans
Patel
Content and
Library Team
Klavans
PERSIVAL Management Structure
Technology
Search
Gravano
User Interface
Johnson
Summarization
McKeown
Project Leader
McKeown
K. McKeown
Clinical
Infrastructure
Cimino
Evaluation
Industry Partners
Klavans
Patel
Content and
Library Team
Klavans
PERSIVAL Management Structure
Technology
Search
Gravano
User Interface
Johnson
Summarization
McKeown
Project Leader
McKeown
J. Klavans
Clinical
Infrastructure
Cimino
Evaluation
Industry Partners
Klavans
Patel
Content and
Library Team
Klavans
PERSIVAL Management Structure
Technology
Search
L. Gravano
Gravano
User Interface
Johnson
Summarization
McKeown
S. Johnson
Project Leader
McKeown
Clinical
Infrastructure
J. Cimino
Cimino
Evaluation
Industry Partners
Klavans
Patel
Content and
Library Team
Klavans
S.-F. Chang
PERSIVAL Management Structure
Technology
Search
S. Johnson
Gravano
User Interface
Johnson
Summarization
S. Feiner
McKeown
Project Leader
McKeown
Clinical
Infrastructure
Cimino
Evaluation
Industry Partners
Klavans
Patel
Content and
Library Team
Klavans
S.-F. Chang
PERSIVAL Management Structure
Technology
Search
Gravano
User Interface
Johnson
K. McKeown
Summarization
McKeown
Project Leader
McKeown
Clinical
Infrastructure
Cimino
J. Klavans
Evaluation
Industry Partners
Klavans
Patel
Content and
Library Team
Klavans
V. Hatzivassiloglou
PERSIVAL Management Structure
Technology
Search
J. Cimino
Gravano
User Interface
Johnson
Summarization
McKeown
D. Jordan
Project Leader
McKeown
Clinical
Infrastructure
Cimino
C. Friedman
Evaluation
Industry Partners
Klavans
Patel
Content and
Library Team
Klavans
S. Johnson
PERSIVAL Management Structure
Technology
Search
Gravano
User Interface
Johnson
V. Patel
Summarization
McKeown
Project Leader
McKeown
Clinical
Infrastructure
Cimino
J. Cimino
Evaluation
Industry Partners
Klavans
Patel
Content and
Library Team
Klavans
D. Jordan
PERSIVAL Management Structure
Technology
Search
J. Klavans
Gravano
User Interface
Johnson
Summarization
S. Jacobson
McKeown
Project Leader
McKeown
Clinical
Infrastructure
P. Molholt
Cimino
Evaluation
Industry Partners
Klavans
Patel
Content and
Library Team
Klavans
E. La Rue
Role of Technology Partners

Lucent: Spoken Language Dialogue
Systems (speaker independent)

IBM: Query refinement, evaluation, text
summarization, interface for image/video
search

GE: Concept based retrieval
32
New Partners and Changes




Siemens Corporate Research: grant for video
summarization integrating text/image, plus researcher time
 $68,680/year
AT&T: grant to build our networking infrastructure
 $50,000
Welch Allyn: information and equipment, Medical
Director’s time, cash
 $35,000
GE Medical Systems: commitment to commercialization
of results
33
Steve Feiner:
IBM is more than just a software
company; ditto Lucent, GE, etc.
Why not just group companies
together?
Partners
Partner
General Electric
Team
IBM
Companies
Lucent
Siemens
Libraries & New York Public
Museums
Library
MOMA
Donating
Bell Atlantic
Companies Intel
AT&T
Contribution
Software
Researcher
Software
Researcher
Software
Researcher
Student Support
Grant
Researcher
Student Support
Consumer Health
Information Specialists
Grant, Image Processing
Technology
Authentication Services
Computer Equipment
Grant, Network
Infrastructure
Value
$200,000
$15,000/year
$50,000
$110,000/year
$25,000
$125,000/year
$201,286 over 5 years
$25,000/year
$34,000/year
$9,680/year
$10,000
$115,000
$209,154 over 5 years
$50,000
34
Partners
Partner
Healthwise
Content Lion Reef
Providers
OVID Technology
Univ. of CA San Francisco
Welch Allyn
Contribution
Value
Consumer Health &
$10,000/year
Software
Images & Animation of
$30,000
Heart
Scientific Biomedical Data $15,000/year
& Software
Medical Images
Information and
equipment
Medical Director’s time
Unrestricted funds
$20,000
$10,000
$5,000
35
Clinical Information Systems
Clinical repository of coded and text data
 Medical Entities Dictionary server
 Web-based access for clinicians (WebCIS)
 Web-based access for patient (PatCIS)
 Infrastructure handles security, application
tracking, and evaluation
 Architecture handles integration of
components

36
Input for PERSIVAL
Coded data
 Natural language processing
 Medical Entities Dictionary
 UMLS

37
Web/PatCIS and Digital Library
Infobutton
Information
Resource
WebCIS/
PatCIS
CIS
MED
42
Web/PatCIS and Digital Library
Infobutton
Information
Resource
Query
Interface
PERSIVAL
WebCIS/
PatCIS
CIS
MED
43
Users

Clinical users
 Columbia-Presbyterian
Medical Center
 New York
Hospital
 37 affiliates

Patients
 Heart
failure
 Home diabetes telemonitoring
 High risk for heart attack
 Cancer patients
44
Evaluation and User-Centered
Design

Usability engineering


Evaluation of system components: cognitive
and quantitative


in situ and lab video recordings
Subjective assessment of search plus recall,
precision
Survivability

end-to-end evaluation, reduction of manual
intervention, training, university support of
applications
45
Summary
Tailor search, presentation, and
summarization of online medical literature
 Exploit online patient record as user model
 Patients and health care providers
 Multimodal access and presentation over
multimedia resources

46