“Modules of Workforce Development: Pragmatic
Approaches to Knowledge Acquisition,
Application, and Information Dissemination”
Enterprise Access to Evidence-Based Literature:
A Dynamic Digital Library for Public Health
Enterprise Access to Evidence-Based Literature:
A Dynamic Digital Library for Public Health
Karen Dahlen, Project Consultant
Elaine Martin, Director
New England Region NN/LM Lamar Soutter Library
University of Massachusetts Medical School
This project is currently funded by the NLM via contract through the
NN/LM, New England Region
Enterprise Library Approach to Knowledge Acquisition,
Application & Information Dissemination
Measure Importance of Trusted Information Access
Ensure Direct Access from Intranet-Based Site
Access (Without Name/Password Required)
Support Core Competencies/Training
Use Survey Instruments Adapted to the Location
Relate Cost Efficiencies to Enterprise Licensing/Creative Partnerships
Document Institutional Change in a Dynamic Environment
Evaluate Over Time (Illustrate Efficiencies)
Shifting the Culture
to Evidence-Based Practice
•
•
•
•
•
Culture of evidence-based practice was affected when funding for public
health libraries was eliminated.
State Libraries (in many states) were mandated to support state
agencies, but had no collections to support public health.
Streamlined access to resources through the Digital Library Platform-along with trust in new relationships--prompted resurgence in use of
resources and has facilitated this cultural shift.
Training stimulated access to resources reviving evidence-based
interest and relationship to practice.
Project has the ability to leverage government resources (NLM, CDC,
NAL) to improve evidence-based practice.
Background (Why this Project)
Project Rationale
Partnerships
ENTERPRISE ACCESS TO EVIDENCE-BASED LITERATURE: A
DYNAMIC DIGITAL LIBRARY FOR PUBLIC HEALTH
Council on Linkages:
Academia & Public Health Practice
Public Health Competencies
Project Tied to Competencies
Competencies: Council on Linkages
•
Analytic/Assessment Skills
–
–
–
•
•
Policy Development/Program Planning Skills
–
•
References sources of public health data and information.
Information technology to collect, store, retrieve data.
Utilizes data to address scientific, political, ethical, and
social public health issues.
Analyzes information relevant to specific public health
policy issues
Public Health Sciences Skills
–
–
Conducts a comprehensive review of the scientific
evidence related to : a public health issue, concern, or
intervention.
Retrieves scientific evidence from a variety of text and
electronic sources.
•
Knowledge Informatics
– Access trusted resources directly from the
desktop.
– Describe relationships and functionality of eresources to core public health interests.
– Introduce specific types of resources: coding
and classification tools.
Informatics Skills
– Train on interoperable features that support
public health work, including connectivity to
trusted sites; advanced search filters, use of
topical alerts and information mgt software.
Obstacles to Getting Information
Gathering and Analyzing EB Information
Rapid Detection of Epidemics
Detecting an anthrax
epidemic one day earlier
would save $1-7 billion
CDC Presentation by
C. Safran citing:
http://www.cdc.gov/ncidod/eid/vol3no2/kaufman.htm
Kaufmann AF, EID, V3, N2
Library Without Signature Building
Goal of the Project
Who Participates
Objectives & Data Sets
Development of Digital Library
ENTERPRISE ACCESS TO EVIDENCE-BASED LITERATURE: A
DYNAMIC DIGITAL LIBRARY FOR PUBLIC HEALTH
PHIA: Innovative Business Model
Not Tied to Traditional Building
Madison, WI
Atlanta, GA
Goal of PHIA
Public Health Information Access
To determine what
resources are core,
useful, evidence-based to
advance public health
practice.
Full text
directly
available
through DL
Core to
public
health
Full text
available
through LP
Unique Aspects of Digital Library
• Resides on PHD Intranet (outside a signature building)
without need for id and password.
• Relies on collaboration, interoperability & funding.
• Move toward analysis; less time on access
• Skills & training support personalized, interactive
styles tied to continuum of learning and institutionalspecific priorities and projects.
• Collection of evidence occurs in many aspects of
Brownson wheel not just at the literature review stage.
• Fundamental change in the way resources are
distributed and utilized.
Public Health Departments
& Library Partners
State
Public Heath Department
Library Partners
AK
AR
CO
Alaska Division of Public Health (AKDH)
Univ of Alaska (Anchorage) Medical Library
Arkansas Department of Health (ADH)
Univ of Alabama Medical Sciences Library
Colorado Dept of Public Health & Environment Poudre Valley Health System Library
CT
HI
IN
KY
ME
MA
MA
MD
NH
RI
VT
WI
WV
WA
Connecticut Public Health Department
Hawaii Department of Health
Indiana State Department of Health
Kentucky Department for Health
Maine CDC
Boston Public Health Commission*
Massachusetts Public Health Department
Maryland Dept of Health & Mental Hygiene
New Hampshire DHHS
Rhode Island Department of Health (RIDH)
Vermont Department of Health (VDH)
Wisconsin Department of Health Services
West Virginia Bureau of Public Health
Washington State Department of Health
University of Conn Maynard Stowe Library
University of Hawaii Medical Library
Ruth Lily Medical Library, Indiana University
University of Kentucky Medical Library
Maine Medical Center Library
University of Massachusetts Soutter Library
Lemuel Shattuck Hospital Library
University of Maryland HS Library
New Hampshire DHHS Library
Rhode Island Hospital/Lifespan
UVM Dana Medical Library
Ebling Library of the Health Sciences
West Virginia University Libraries
University of Washington HS Library
PHIA Evaluation Process
Tied to Data Model
To advance access to evidence-based
resources in support of improved
public health practice.
What resources are high use?
What technical issues arise?
What resources are missing? How can training be
improved?
What project components lead to improved
efficiencies?
What cost efficiencies are involved with an enterprise
approach to digital library implementation, and
training.
•
Logic Model
–
•
Resources, activities, outputs, outcomes
(minutes, visits, trainings, people lists, IP
management, levels of troubleshooting,
RML reports.
Data Tools
–
–
–
–
–
Overarching questions
Journal SurveyMonkey
Pre-Post Training Surveys
Just-in-time Information Checks
Formal Evaluation (interviews & focus
group sessions
Data Sets & Sources
•
•
•
•
•
•
•
•
Resource Metrics Collected
Journal Survey Monkey Identifies
Perceived Need & Library Use.
Document IT technical issues.
Licensing Issues Related to
Journals and Databases.
Training Surveys Collected
Preliminary Meetings/Updates Held.
Trainings Held
Year End Evaluation
--Interviews & Focus Groups
View of PHIA Digital Libraries
Connecticut Digital Library
Arkansas Digital Library
Resources on Digital Library
Use of Resources
Article Delivery Use & Costs
ENTERPRISE ACCESS TO EVIDENCE-BASED LITERATURE: A
DYNAMIC DIGITAL LIBRARY FOR PUBLIC HEALTH
Databases & Major
Publishers of Resources
PubMed , Other NLM Databases
NAL, CDC & Other Gov’t Resources
CLSI (Clinical Laboratory Standards)
Global Health
Cochrane Library
– Systematic Reviews
Stat!Ref—E Books (35)
-- Coding & Classification
ASABE –Health Linked to Agriculture
Identifying Core, Useful, &
Evidence-Based Journals
Publisher
Total Use
Journal Title
Use
Mass Medical Society (1)
14,136
New England Journal of Medicine
14,136
AAP(1)
9,888
Pediatrics
9888
Oxford University Press (20)
23,369
Clinical Infectious Diseases
3147
Springer (18)
2, 485
Maternal & Child Health
474
ASM Journals (12)
3,518
Journal of Clinical Microbiology
1729
Annual Reviews (9)
697
Annual Review Public Health
394
Sage (11)
939
The Diabetes Educator
135
BMJ (9)
2932
British Medical Journal
1832
Mary Ann Liebert (11)
1798
Breast Feeding Medicine
355
University of Chicago Press (6)
955
Infection Control & Hospital Epidemiology
749
Lippincott/Ovid (6)
122
J Public Health Mgt Practice
55
NEJM Use 2010-2013
2010=2087
2011=11,194
2012=14,022
2013=14,136
7000
6000
5000
4864
4000
3000
2000
1000
0
1772
982
554
314
527
337
311
890
1183 1178
308
494
422
ASM: Journal of Clinical Microbiology
Total Use 2013=1729
298
287
233
175
135
94
90
89
82
78
71
42
32
18
5
Oxford University Press Use
Jan-Dec 2013
Total Use by Each PHD
N=13,827
PHD
Total
Alaska DPH
Arkansas DH
BPHC
CDPHE
CT DPH
HI SDH
ISDH
KYDPH
Maine CDC
MDHMH
MDPH
NH DHHS
RIDH
1245
1483
182
608
1234
638
700
1134
757
1427
1817
157
620
VDH
WI DHS
709
1116
High Use Journal Titles
Journal Title
Total Use
Clinical Infectious Disease
American Journal of Epidemiology
Schizophrenia Bulletin
The Journal of Infectious Diseases
3147
941
771
627
ICES: Journal of Marine Science
JNCI: Journal of the National Cancer
Institute
474
Journal of Antimicrobial Chemotherapy
297
International Journal of Epidemiology
Alcohol and Alcoholism
Journal of Analytical Toxicology
Health Promotion International
288
274
262
252
Rheumatology
Age and Ageing
246
240
434
STAT!Ref E-Books/Tools Provide Basic
Understanding of Topic
Filters & Tools
– Codes[5]
•
•
CPT with RVUs Data File, INGENIX®
(2011)[1]
ICD-9+CM - Volumes 1, 2 & 3 (2012)
– Titles By Discipline[344]
Searchable Alerts (by topic)
–
–
–
–
Functionality with NLM
Chapters link to PubMed
TOXNET linked to Search
Related Concepts/Meta…
Functionality with CDC
–
MMWR, Community Guide;
STAT!Ref E- Books
for Public Health
Titles Containing PH Content
E-Book
Use
Oxford Textbook of Public Health - 5th Ed. (2009)
482
407
Manual of Clinical Microbiology - 10th Ed. (2011)
403
AHFS Drug Information (2013)
305
ACP PIER, Journal Club & AHFS DI Essentials (2013)
Coding & Classification Books
E-Book
Use
ICD-9-CM - Volumes 1, 2 & 3 (2013)
881
CPT with RVUs Data File, INGENIX (2013)
474
ICD-10-PCS: Procedure Coding System (2014)
12
Red Book: 2012 Report of the Committee on Infectious
Diseases - 29th Ed. (2012)
241
Emerging Infections Series (2008 - 2010)
216
ICD-10-CM: Clinical Modification (2014)
15
209
Diagnostic and Statistical Manual of Mental Disorders
4th Ed. (DSM-IV-TR, 2000)
107
Textbook of Modern Toxicology, A - 4th Ed. (2010)
CLSI (Clinical & Laboratory Standards)
20 Documents Used 790 Times
April 21, 2014
Doc #
Document Name
#
M100 M100-S24: Performance Standards for
-S24 Antimicrobial Susceptibility Testing; TwentyFourth Informational Supplement
248
EP09- Measurement Procedure Comparison and
A3
Bias Estimation Using Patient Samples;
Approved Guideline—Third Ed.
102
EP12- User Protocol for Evaluation of Qualitative
A2
Test Performance
55
MM0 Nucleic Acid Sequencing Methods in
9-A2 Diagnostic Laboratory Medicine; Approved
Guideline—Second Edition
54
Calculations Related
to Enterprise Licensing
State PHD
Enterprise Licensing is
based on 10% of the Total
FTE level to capture
baseline information and
measure use/interest of
resources.
FTE
10% Rule
ADH
AKDPH
BPHC*
CDPHE
CT DPH
Hawaii DPH
2781/1390
525
1100/550
1200/600
780
3000/1500
ISDH
Kentucky BPH
MDHMH
Maine CDC
MDPH
NH DHHS
RIDH
VDH
WDPHS
VW DHS
Washington
780
400
8000/4000
395
3000/1500
279
400
300
400
780
150
78
40
400
39
150
30
40
35
40
78
1600/800
10,999
80
1546
Total PHIA FTE
139
52
55
60
78
Article Delivery Costs Via Library Partnerships
May 2011 – April 2012; Cost=$18,384
STATE
Connecticut
ILL
DD
Total
0
83
83
Maine
Massachusetts
(BPHC)
Massachusetts
(MDPH)
New
Hampshire
560
Rhode Island
364
924
8
3
11
1
1
2
0
0
0
47
36
83
Vermont
21
191
212
Colorado
323
156
479
Total
960
834
1794
May 2012 --- April 2013; Cost=$12,241
STATE
ILL
DD
Total
2
20
22
Maine
Massachusetts
(BPHC)
Massachusetts
(MDPH)
New
Hampshire
154
90
244
9
10
19
7
6
13
0
0
0
Rhode Island
82
79
161
Vermont
54
243
297
Colorado
112
40
152
Arkansas
32
73
105
Kentucky
8
3
11
Wisconsin
0
0
0
462
593
1055
Connecticut
Total
Training Topics Embrace
National and State Interests
Maryland DHMH Health Initiatives
•
•
•
•
•
•
Health Disparities(sexual minorities)
Preventive Services ACA* HIV Screening
Immigrant Health
Undocumented individuals under ACA
Drug Resistant Disease which impacts
Impact on TB, STI (plus others)
Health Care Reform and Interpersonal
Violence/Domestic Violence
Infectious Diseases; Hepatitis C Virus
HIV Infections --HIV and HCV Co-Infection
Number of Trainings/Diversity of Workforce
STATE
PHD
Trainings
Attendance
Maryland
Indiana
Hawaii
Nutrition Specialist (2)
State Epidemiologist
Education Coordinator
(2)
Research Statistician
Supervisor – Labs + (3)
Asthma Coordinators(2
Policy Analyst (2)
Injury Prevention
Epidemiologist
Legislative Liaison
RI
RIDH
2
30
VT
3
46
ME
VDH
MAINE
CDC*
MA
BPHC*
4
58
MA
MPHD*
3
32
Program Mgr (2)
Staff Attorney
Epidemiologist
NH
NH DHHS
2
24
Epidemiologist
Microbiologist ( 2)
Research Analyst (2)
CO
CDPHE*
5
100
CT
CT DPH*
5
101
Medicare
Specialist/Advisor (2)
Regional Program
Director
Program Coordinator,
SAPB
KY
AR
KDPH
ADH
2
2
46
Policy Advisor (2)
Director of Prog
Evaluation
Epidemiologist (3)
WI
WI DPH
2
33
Psychiatric VT
Director of Prog
Development
Genetic Counselor
AK
AK DPH
2
50
Executive Ass’t
Field Epi Director
Tobacco Prevent Mgr
HI
HI DPH
2
68
IN
ISDH
2
60
Deputy
Secretary
Access Services Mgr
Informatics Analyst
MD
MDHMH
2
38
Director VSA
IT Supervisor
Nutritionist
15
41
769
TOTAL
3
52
31
Enterprise Efficiencies
Unique Aspects of the Business Model
Evaluation Processes
ENTERPRISE ACCESS TO EVIDENCE-BASED LITERATURE: A
DYNAMIC DIGITAL LIBRARY FOR PUBLIC HEALTH
Cost Efficiencies:
Article Analysis
Direct Access to Articles
Contract Year
Total #
Amount $
May 2013 –April 2014
17+
285,409
May 2012-Apr 2013
15
May 2011-Apr 2012
May 2010-Apr 2011
Alternative Article Delivery
Contract Year
Total #
Amount $
May 2013-March 2014
1038
10,102
200,352
May 2012-Apr 2013
1055
12,241
11
156,280
May 2011-Apr 2012
1794
18,385
9
96,252
May 2010-Apr 2011
672
8,263
Enterprise Licensing Efficiencies: Single Titles
Contract Year May 2012-April 2013
Journal Title
# Uses
Cost
Per Use
Replacement
Cost
Cost of
License
Am J Tropical Med & Hygiene
410
$1.58
$4,510
$ 650
Am J Respiratory & Critical Care Med
189
$14.28
$2,079
$2700
Health Affairs
Infection Control & Hospital
Epidemiology
Public Health Reports
2038
1.81
$22,418
$3690
749
1.50
$8,239
$1125
658
4.69
$7,238
$3085
Intern’l J of Tuberculosis & Lung
Diseases
192
5.83
$2112
$1120
Annual Review Series
595
12.60
$6545
$7500
Pediatrics
7846
.68
$86,306
$5300
PHIA: Innovative Business Model
•
•
•
•
•
•
Enterprise licensing saves time, money, and effort.
Central management of IP addresses ensures quick turn-around.
Ability to understand change (System, personnel, environment).
Resources are identified via benchmarking,metrics of direct use through
Digital Library, and requests from PHDs.
Journals are identified via “Article Delivery” on Library Partner side.
Alternative delivery of resources strengthens state relationships with
immediate access (within 4-24 hours).
EVALUATION PROCESSES
Levels of Data Collection
•
•
•
•
•
•
•
•
•
•
Journal SurveyMonkey (baseline data for PHIA and PHD)
Enterprise Licensing (cost effectiveness over time)
Vendor statistical reports (validates “use of resources”)
Monthly reports from “library partners” (measures use and identify
resources to develop Digital Libraries).
Ongoing capture of suggested e-books, journals, and databases to
enhance collection
Pre-Post training links
– (data related to knowledge of resources)
Feedback from hands-on training
Interviews with leaders and workforce
Focus group sessions (after one year; subsequently)
Quarterly/Annual Reports (submitted to NLM)
Results
• Digital libraries will be in place in 17 PHDs by July.
• Full-text access to more than 150 e-journals, 5
databases, reports and more.
• Introductory sessions have introduced PHD
leadership to the project and explained obligations.
• More resources have been added to “digital library.”
• Structured training has been held in 15 PHDs.
• More than 600 people have been trained with
backup models currently in test stage.
NACCHO Award Brings
Recognition to Project
• March 2012 and 2013, the PHIA PHD Digital Library Project received
a “promising practice award” from NACCHO.
• Presentations have been held at national and local meetings, e.g.,
APHA in Washington, DC in 2011 and Boston in November 2013.
• Article was published in AJPH in January 2014. Project was cited in
August 2012 EID (CDPHE).
• Presentations have been made at national public health meetings,
national and regional library meetings.
Testimonial
“Great recognition for this very valuable
initiative. Kudos and thank you.”
Jewel Mullen, MD, MPH, MPA, Commissioner,
Connecticut Department of Public Health
Discussion & Outcome
How Do We?
• Expand PHIA to all 50 states.
• Enhance partnerships & relationships.
• Continuously train PH Workforce to improve
competencies (knowledge and skills) given
staff turnover.
• Build a sustainable model including cost
structure and project management.