A Strategic Measurement and Evaluation
Framework to Support Worker Health
COMMITTEE ON DHS OCCUPATIONAL HEALTH
AND OPERATIONAL MEDICINE INFRASTRUCTURE
June 10-11, 2013
Ron Z. Goetzel, Ph.D. , Emory University and Truven Health Analytics
Workplace Health Promotion/Health
Protection Programs: What Should be
Evaluated?
• Structure
• Process
• Outcomes
2
LOGIC MODEL:
WORKSITE PROGRAMS
HEALTH
PROMOTION/PROTECTION
• STRUCTURE
• PROCESS
Employees
Modified Worksite Health Promotion (Assessment of Health Risk with Follow-Up) Logic Model adopted by the CDC Community Guide Task Force
• OUTCOMES
EVALUATION MEASURES
Worksite Health
Promotion/Protection
Programs
Structure
Process
Outcomes
Leadership engagement
Policies and Procedures
Environmental Support
Fidelity
Dose delivered/
Dose received
(participation, engagement,
satisfaction)
Health
Financial
Productivity
4
4
Program Structure
Structure defines the program -- how does it work – the WHAT,
HOW & WHEN?
• Individual components, e.g., HRA, feedback reports, mailings,
internet services, high risk counseling, referral to community
resources, incentives
• Environmental components, e.g., organizational policies,
cafeteria/vending machine choices, time off for health
promoting activities, senior management support, access to
physical activity programs, walking paths, shower/change
facilities, healthy company culture
5
http://www.cdc.gov/niosh/docs/2010-140/
6
Environmental Assessment Tool
J Occup Environ Med. 2008
Feb;50(2):126-37.
7
Checklist of Health Promotion
Environments at Worksites
8
Leading By Example Assessment
Am J Health Promot. 2010 NovDec;25(2):138-46..
9
HERO SCORECARD
Sample Results
ABC
Inc.’s
Score
Nation
al
Averag
e
Maximu
m
Points
1. Strategic Planning
7
5
11
2. Leadership
Engagement
18
16
33
3. Program Level
Management
7
11
22
4. Programs
14
28
56
5. Engagement Methods
40
29
67
6. Measurement and
Evaluation
2
5
11
TOTAL
88
94
200
Category
http://www.the-hero.org/scorecard_folder/scorecard.htm
accessed 5/12/12
.
Based on ABC Inc.’s response and database average as of [May 1, 2009].
10
CDC WORKSITE HEALTH SCORECARD
http://www.cdc.gov/dhdsp/pubs/worksite_scorecard.htm
PROGRAM PROCESS
Program process evaluation defines how well the
program is carried out:
– Participation rates
– Satisfaction with the program/process/people
– Completion rates
12
PROGRAM PROCESS COMPONENTS
• GOAL: To summarize program implementation and to form hypotheses
about how implementation may affect program outcomes
• To monitor progress during a program implementation and to inform
potential adjustments to the program to improve program quality
– Program Fidelity (quality) - how the program was implemented
– Dose Delivered (completeness) – frequency and intensity of the
program
– Dose Received (satisfaction) - how participants react to the
intervention
– Program Reach (participation rate) –The proportion of eligible
(employees) that participated in the various components of the
programs?
13
EXAMPLE ASSESSMENT
14
Satisfaction
15
Performance Rating
16
Program Impacts
Table 25. Satisfaction Survey: Program Impact and Effects of Wellness at Work
n
Very Positive (5)
Positive (4)
No Effect (3)
Negative (2)
Very Negative (1)
Morale at work
374
62.6%
35.0%
2.4%
Job performance
371
54.7%
43.1%
2.2%
Job satisfaction
365
49.9%
47.9%
2.2%
Employer
satisfaction
365
49.9%
48.5%
1.6%
Health
379
80.7%
16.9%
2.4%
Lifestyle
370
74.6%
23.2%
2.2%
Fitness level
376
75.3%
22.3%
2.4%
n
Excellent (5)
Very Good (4)
Good (3)
Fair (2)
Poor (1)
384
46.6%
39.1%
14.3%
Program Effect on:
Overall WAW Rating
17
PROGRAM OUTCOMES
• Program outcomes are evaluated by determining whether program
objectives are achieved, at a given level of quality, and within a defined
time framework
– Health outcomes
• Behavior change
• Risk reduction
– Medical care outcomes
• Health care utilization
• Health care costs
– Productivity outcomes
• Absenteeism
• Disability
• Workers’ compensation/safety
• Presenteeism
18
RESEARCH DESIGN
• Pre-experimental
• Quasiexperimental
• True experimental
Validity of results increases
as you move down this list
All are tools that can help
understand the impact of
the program
19
NON-EXPERIMENTAL DESIGN
(PRE-EXPERIMENTAL)
Program start
20
GENERAL TREND OR PROGRAM EFFECT?
Program start
21
PROBLEMS WITH A PRE-EXPERIMENTAL DESIGN:
REGRESSION TO THE MEAN
•
The most simple analysis may produce the wrong answers
Same people
Before the
Intervention
Intervention Period
Savings?
22
REGRESSION TO THE MEAN
23
RESEARCH DESIGN: QUASIEXPERIMENTAL
Pretest posttest with comparison group
01
X
02
Experimental Group
--------------------------
01
02
Comparison Group
24
24
ANNUAL GROWTH IN NET PAYMENTS
Annual growth in costs, Highmark, Inc.
For matched-participants and non-participants over four years`
3500
3000
2500
2000
Start of Program
1500
1000
500
0
2001
2002
2003
Participants
25
Controls
2004
2005
RETURN ON INVESTMENT AND NET
PRESENT VALUE
Return on Investment (ROI) =
Savings
Program Cost
= $1 break-even
Net Present Value (NPV)
= Savings – Program Cost
= $0 break-even
26
Cost-Benefit (ROI) Analysis
HRA & Incentive
Online
Group
Nutrition Coaching
10,000 Steps
Fitness Center
Highmark Challenge
Maintain Don't Gain Newsletter
Wellness Program Costs
Cost per participant
Estimated Annual Savings from Model $176.47/person
Net Savings (Estimated Savings - Wellness Program Costs)
2002
2003
2004
2005
# Used Total # Used Total # Used Total # Used
Total
GD Total
1892 $243,731 1303 $143,111 1308 $140,785 1355 $142,605
201
$1,142
247
$1,372
248
$1,300
512
$2,575
34
$1,544
56
$3,077
56
$3,010
0
$0
2
$66
23
$740
51
$1,585
111
$3,420
244
$2,441
413
$3,851
223
$2,061
407 $25,603
495 $29,939
879
$50,958
112
$348
910
$2,766
85
$182
93
$192
$246,483
$176,343
$181,000
$204,577
$130.28
$135.34
$138.38
$150.98 $808,403
$333,881
$87,398
$333,881
$157,538
$333,881
$152,881
Total Savings Estimated 4 Years
Total Costs 4 Years
Return on Investment
Wellness Program Costs, Highmark, inflation-adjusted to 2005 dollars
27
$333,881 $1,335,524
$129,304 $527,121
$1,335,524
$808,403
$1.65
Assessing Causality
• Evaluators must explicitly state the intervention pathway and metrics
used to measure:
– The “cause” or actual intervention
– The “effect” – proximate and/or ultimate outcomes that result from
the intervention
• Hypotheses that outcomes are “caused” by the HP program must be
articulated and tested
HP
Program
Ultimate
Outcomes
Proximate
Outcomes
Effect
Cause
28
CRITICAL STEPS TO SUCCESS
Financial ROI
Reduced Utilization
Risk Reduction
Behavior Change
Improved Attitudes
Increased Knowledge
Participation
Awareness
29
HEALTH RISKS – BIOMETRIC
MEASURES -- ADJUSTED
Results adjusted for age, sex, region * p<0.05 ** p<0.01
HEALTH RISKS – HEALTH BEHAVIORS
-- ADJUSTED
Results adjusted for age, sex, region * p<0.05 ** p<0.01
ADJUSTED MEDICAL AND DRUG COSTS
VS. EXPECTED COSTS FROM
COMPARISON GROUP
Average Savings 2002-2008 = $565/employee/year
Estimated ROI: $1.88 - $3.92 to $1.00
Summary
• Evaluation of Health Promotion/Protection Programs
is doable, but tricky
• Know your audience – the level of sophistication in
conducting financial analyses varies significantly –
well done studies are complex and expensive
• It’s easy to come up with the “wrong” answer if the
proper research design is not used
• Ask for help – good evaluation studies require a team
of individuals with diverse backgrounds and skill sets
• Tell the truth, the whole truth, even if it means
saying the program didn’t work
33
