Evaluating Workplace
Health and Wellbeing
Interventions
Georgia T. Karuntzos, Ph.D
Jeremy Bray, Ph.D
Jesse M. Hinde
© 2012 | WORK, FAMILY & HEALTH NETWORK
RTI International
Steps for Conducting Effective Evaluations
Step 6:
Ensure use
and share
lessons learned
Step 5:
Justify
conclusions
Step 1:
Identify Need and
Engage
Stakeholders
Standards
Utility
Feasibility
Propriety
Accuracy
Step 2:
Describe the
Program and it’s
intended effects
(Logic Model)
Step 3:
Focus the
Evaluation
design
Step 4:
Gather credible
evidence
Centers for Disease Control
2
Types of Evaluations
Adapted from:
Norland, E. (2004, Sept). From education theory.. to conservation practice Presented at the Annual Meeting of the International
Association for Fish & Wildlife Agencies, Atlantic City, New Jersey.
Theoretical/Logic Model
■ A diagram of the theory of how a program is supposed to work—
a graphic depiction of relationships between activities and results
(conceptual/theoretical model)
■ A logical chain of connections showing what a program intends
to accomplish
■ Increases intentionality and purpose
■ Guides prioritization and allocation of resources
■ Helps to identify important variables to measure; use evaluation
resources wisely
■ Supports replication
WFHN Evaluation Design
■ Single protocol
■ Work redesign, supervisor training and self-monitoring
■ Multiple industries and worksites
■ Healthcare (blue collar)
■ Telecommunications (white collar)
■ Group randomized field experiment
■ Intervention and comparison group assignment at the
worksite or work group level
■ Adaptive randomization to balance covariates
Outcome Evaluation Design
■ Nested cohort design
■ Worksite partners are randomized within industry to
intervention or control group
■ Outcomes evaluated at multiple levels, employees,
workgroups and worksites
■ Allows for multiple levels of clustering (e.g.,
workgroups within worksites), and a variety of
outcomes (e.g., discrete, continuous, count)
Formative(Process) Evaluation Framework
Implementation
Outcomes
What?
How?
QIs
ESTs
Implementation
Strategies
Acceptability
Adoption
Appropriateness
Cost
Feasibility
Fidelity
Penetration
Sustainability
Service
Outcomes*
Efficiency
Safety
Effectiveness
EquityPatientcenteredness
Timeliness
*IOM Standards of Care
Implementation Research Methods
8
Proctor’s Model of Implementation Outcomes
Individual
Outcomes
Satisfaction
Function
Health status/
symptoms
Process Evaluation Data
■ Document Reviews – provide information to build an “a priori”
understanding of the program content, operations, context, and program
stakeholders
■ Review reports, instruments, protocols, promotional materials, patient
materials, resource lists, organizational documents (org charts, flow
charts, operation manuals), web sites.
■ Observational Studies – provide empirical evidence to assess program
fidelity, and generate service flow and timing data to inform outcome and
cost analysis
■ Key Informant Interviews – provide contextual information related to
program utility, contextual factors that influence program implementation,
service delivery, dispersion, and sustainability
■ Practioner and Consumer Surveys -- provide systematic data related
to service delivery experiences and program related perceptions
Qualitative Analysis Methods
■ Recursive abstraction (Document Summaries)
■ Iterative process that generates summaries, classifications, lists,
rates, or groupings
■ Deductive and Inductive Analysis (data coding)
■ Deductive “a priori” framework
■ Inductive “grounded theory” analysis
■ Results in taxonomies, themes, categories, orders
■ Comparative analysis
■ Documented vs observed processes, behaviors or outputs
■ Document changes over time (e.g., model migration)
■ Variation between processes or outputs
■ Mixed Methods (triangulation and convergence)
10
Process Evaluation Results

Comprehensive description of program
components

Performance indicators and proficiency
scores for program delivery

Common barriers and facilitators across
worksites

Descriptive taxonomy of program
settings

Construction of moderator variables for
use in outcome and economic analyses

Program delivery protocols that are
used at each worksite site
Outcome Evaluation Questions
■ Program Outputs
■ To what extent is the program actually performed as
measured by (e.g. the number of health risk appraisals
completed, percentage of employees participating in
workshops, number of follow-up services delivered)
■ Proximal and Distal Outcomes
■ What is the effectiveness of the program on outcomes of
interest? (changes in sleep quality, changes in biomeasures)?
■ What stakeholder and employee-level characteristics
moderate the willingness and ability of performance sites and
practitioners to adopt the program
WFHN Study Data
Outcome Evaluation Methods
■ Descriptive statistics generating classifications or groupings of
the participating worksites, provider characteristics, employee
characteristics
■ Multivariate regression examining the relationship between
groups
■ Statistical modeling measuring changes on outcomes over
time
Outcome Evaluation Results
■ Rates and frequencies of program
participation
■ Performance site descriptive characteristics
(including staff characteristics) that are
associated with outcomes of interest
■ Within group pre to post changes in health
and wellness outcomes
■ Between group comparative changes in
health and wellness outcomes at multiple
time points
Intervention Effect on Work Environment
FSWE Hypotheses
■
H1. There are changes in work environment for the
■ H1a. TOMO intervention group
■ H1b. TOMO control group
■ H1c. LEEF intervention group
■ H1d. LEEF control group
■
H2. There are baseline differences in environment between intervention and
control groups for the
■ H2a. TOMO industry
■ H2b. LEEF industry
■
H3. There are differences in the rate of change in climate between intervention
and control groups for the
■ H3a. TOMO industry
■ H3b. LEEF industry
FSWE Factor Analysis
Family Supportive
Work Environment
0
1.465
Family Supportive
Supervisor
Behaviors
1.142
Organizational
Climate
11.4
Schedule Control/
Decision Authority
10.7
11.5
Family Supportive
Work Environment
1.465
11.4
0.844
Time Adequacy
8.3
-9.9
Family Supportive
Supervisor
Behaviors
0.655
1.142
Organizational
Climate
8.3
0.655
Time Adequacy
10.7
0.844
Schedule Control/
Decision Authority
11.5
Growth Model Results by Industry
Hypothesis
Industry
Group
Random Effect
Estimate
Standard Error
p
H1a
TOMO
Intervention
Slope
0.042
0.009
<0.0001
H1b
TOMO
Control
Slope
0.014
0.008
0.081
H1c
LEEF
Intervention
Slope
0.003
0.007
0.657
H1d
LEEF
Control
Slope
0.001
0.006
0.905
H2a
TOMO
Intervention
Vs. Control
Intercept
0.033
0.038
0.393
H2b
LEEF
Intervention
Vs. Control
Intercept
0.013
0.027
0.619
H3a
TOMO
Intervention
Vs. Control
Slope
0.028
0.012
0.016
H3b
LEEF
Intervention
Vs. Control
Slope
0.002
0.009
0.804
Economic Evaluation Questions
■ What is the program cost to worksites
and to other stakeholders?
■ What is the cost-effectiveness of the
program?
■ What characteristics moderate the
cost-effectiveness of the program?
Translational Study


An effective intervention is only useful
if it is communicated to and adopted
by workplaces
Complementary methods:
 Use participant feedback to inform
post-study messaging
 Analyze process data to study
employee perception for
developing portrayals of the
intervention
 Assess potential dissemination
channels
 Stimulate market demand for
effective intervention
Thank You
For more information on the Work, Family,
and Health Network Study
http://projects.iq.harvard.edu/wfhn/people