Economic tools to evaluate social science programs Bidisha Mandal
ECONOMIC TOOLS TO EVALUATE
SOCIAL SCIENCE PROGRAMS
BIDISHA MANDAL
SCHOOL OF ECONOMIC SCIENCES
WASHINGTON STATE UNIVERSITY
November 16, 2011 WSAC, 2011 Extension Directors Conference
Overview
Economic tools
Why evaluate?
What are economic tools, economic analyses?
When to evaluate?
How to evaluate?
Case studies from Health Extension
Spokane Public schools: Nutrition services intervention for middle school students
Strengthening Families Program: Substance abuse prevention for adolescents
Providence health care services: Transitional care model to reduce preventable hospital readmissions
Supplemental Nutrition Assistance Program Education: University of Idaho Extension
Magnitude of the Problem
Primary prevention
Health promotion
Secondary prevention
Screening, diagnosis, therapies
Tertiary prevention
Treatment to prevent or postpone complications
A Crucial Difference
Program effectiveness
Outcome oriented
Directly links the intervention with health outcomes of interest
Program evaluation
Ways to maximize the intended impact with available resources, or
Ways to obtain a particular impact with as little resources as possible
Importance of Evaluation
Resource constraints
To eliminate or reduce waste
Evidence of return on investment
Ideally…
Inform decisions
Help make choices about future allocations
Example: Health-care system
Getting value for money is a widely accepted and legitimate goal
Quality medical care in part translates into potentially expensive demands for new drugs and technology
Pressure to improve efficiency, make trade-offs, and develop incentive systems for patients and physicians while holding down healthcare costs
Address Two Questions
What works?
Multiple interventions could work
What works best?
Identify the one intervention that provides the greatest bang for the buck!
Example: Increased prevalence of diabetes among adults
Strategies and reasonable alternatives
Physical activity – Fitness programs (worksite, community, less
TV/computer time)
Diet – cooking programs, education program to change food consumption behavior
What is the objective
Increased physical activity? Short-term
Reduced risk of diabetes? Long-term
Economic Analysis
Evaluation is part of the program design and planning
There are always competing use of resources
Frame the study in order to consider opportunity costs for each of our choices
Identify, measure, value and compare the costs and consequences of alternative prevention strategies
Quantitative and analytic methods
Cost-benefit analysis
Cost-effectiveness analysis
Cost-utility analysis
Identify the Risks Factors
Define the target population for the intervention
Define the problem or question, and magnitude of impact
Define the information needs of the target population in reference to the program or intervention
These steps will
Influence the types of benefits and costs to be included
Help to determine which analytic method is most appropriate
Identify the Intervention(s)
Indicate clearly the preventive strategies under consideration, including baseline if any
Specify perspective of the program and analysis
Limit perspectives to those relevant to the study
Define relevant time frame in which program will be delivered
Determine how far into the future costs and effects that accrue from the intervention will be considered
Background on the Intervention
Can it work?
Efficacy
Degree to which intervention strategies can work under ideal conditions, with carefully selected participants, and optimal resources.
Example: Randomized controlled trials
Will it work?
Effectiveness
How well these strategies actually work in community settings
Demonstrates real-world effectiveness under practical resource constraints
Effectiveness is likely to be lower than efficacy
What are the benefits and costs of the intervention?
Units of measurement
How do benefits compare with costs?
What additional benefits could be obtained with additional resources?
Identify the Methods
Determine the analytic methods for decision-making
The choice will depend on the policy question, the outcomes of interest, and the availability of data
Determine whether analysis is to be marginal or incremental
Marginal analysis: Examines the effect of expanding or contracting an intervention
Incremental analysis: Compares the effects of alternative programs
Identify the Outcomes
Identify the relevant costs
Program costs
Productivity losses
Identify the relevant outcomes
Number and nature of health outcomes
Specify the discount rate or time preference for monetary and non-monetary costs
Identify sources of uncertainty and plan sensitivity analysis
The Role of Discount Rate
Individuals generally weight costs and benefits in the near future more heavily than in the distant future
This applies to the valuation of capital and investments and to health outcomes
Societal preference is for health benefits received today versus health benefits received in the future
Using an appropriate discount rate in an economic analysis allows
Adjusting the value of receiving benefits today versus in the future or of incurring costs today versus in the future
Makes benefits and costs comparable over time
Discount rate is selected based on the study perspective
Social or private or individual
The Role of Uncertainty
Precise estimates of costs and benefits/effects are often not available
Limited literature
Different population settings
Important to list all assumptions upon which estimation is dependent
Perform sensitivity analyses
How will result of evaluation change if the assumptions change?
Evaluation Tools
3 most commonly used techniques
Cost-benefit analysis (CBA)
Cost-effectiveness analysis (CEA)
Cost-utility analysis (CUA)
Each method
Allows comparison of different intervention strategies
Calculates resources consumed and outputs generated
Requires quite similar cost analysis
Assessment of outcomes, both benefits and harms (negative benefits)
Scope of analysis determines the appropriate analytic method
Cost-Benefit Analysis
All costs and benefits valued in dollars
Costs include
Cost of program
Cost to participants - out-of-pocket expenses, productivity losses, travel time, child care, intangible costs (pain, suffering)
Benefits include
All types of beneficial and harmful health outcomes, whether intended or not
Have to be valued in monetary terms
CBA is well suited to comparisons with interventions that include cross-sector considerations
Housing, education, transportation interventions
Cost-Effectiveness Analysis
Usually examines direct medical, non-medical, and productivity costs
Compares costs with outcomes in standard health units
Example: costs per case averted
Most suitable when comparing interventions that have similar health outcomes
Cost-Utility Analysis
Modified version of CEA
Compares direct medical and non-medical costs with health outcomes converted to a standard health unit, often a quality adjusted life year (QALY) combining both mortality and morbidity
Often used to compare health intervention which have different type of health outcomes
CASE STUDIES
Spokane Public Schools:
Objective
Spokane public schools’ lunch program
With Doug Wordell, Ruth Bindler, Kenn Daratha, Sue Butkus
Intervention program included reducing vending machine beverages, limiting ala carte offerings, and adding seasonal fruits and vegetables to student lunch menus
Compare pre-program and post-program behavior
Objective
I f there were associations between an altered school food environment and food choices of middle school students both in and outside of school
My involvement
Retroactive, after program was delivered and surveys were conducted
My role
Analyze survey data
Determine food behavior change
Spokane Public Schools:
Method & Results
Results
Healthful modifications in the school food environment associated with some positive food behaviors
The cost of conducting the intervention was approximately $24,000/year
Lost ala carte sales, loss in vending machine sales
More expenditure on produce
In this study, is it possible to show the benefits of improved food choices outweigh the costs?
NO - Related data was not collected for CBA
Spokane Public Schools:
What did I learn?
Difference between outcomes and impacts
In this study, there is no way to link improvement in behavior to improvement in health (short-term or longterm)
Even if they are positively related, we have no quantitative measure for the benefits
Economic analysis is not always possible
Unless evaluation is part of the program design
Strengthening Families Program:
Objective
Strengthening Families Program (SFP) for Parents and Youth
10-14 years
With Laura Hill, Robby Rosenman, Ron Mittelhammer
Voluntary, family-based intervention
Designed to discourage future substance abuse among adolescents and youth
Compare pre-program and post-program behavior
Randomized clinical trials (RCT) have shown SFP to be costeffective and that benefits outweigh costs
How does SFP’s impact in community dissemination compare to results from RCTs?
My involvement
Retroactive, after programs were delivered and surveys were conducted
Strengthening Families Program:
Method & Results
Community dissemination has many practical issues
Variation in program delivery across counties, states
Data not recorded systematically or consistently
CBA, CEA and CUA not possible
Do not have necessary data
But, have data to determine which factors in community dissemination of the program are different
Attrition – high incompletion rates, some people come to the sessions but do not respond to surveys
Self-selection – more motivated parents are more likely to attend SFP
Results
People who come to the sessions but do not respond to surveys have lower self-assessment scores
Strengthening Families Program:
What did I learn?
Validity of assumptions
Does sample match population?
Differential dropouts
Are facilitators similar?
Some sessions have additional orientation session
Language of delivery
Providence Hospital Transitional Care:
Objective
Transitional care model
With Cindy Corbett
Innovative model to improve and synchronize hospital discharge planning and deliver core transitional care intervention to patients at high risk for potentially preventable readmissions
Secondary objective: document barriers and facilitators of successful delivery in different hospital environments
My involvement
Contacted during the planning process
But, resource constrained
Unable to collect/record some necessary data for a complete CEA
Future studies will look at CUA
Providence Hospital Transitional Care:
Method & Results
Cost-effective analysis of transitional care model
Compared to patients not receiving transitional care
Have necessary costs
Cost of transitional care nurse, inpatient pharmacy, pharmacotherapy clinic, administrative costs, home health care costs, hospital care costs
Have some necessary effects (in $)
Hospital care avoided due to lower re-hospitalizations
Revenues from inpatient pharmacy, pharmacotherapy clinic not recorded
Result
Decrease in re-hospitalizations and ED visits
Total savings over 4 months = $55,752.34 (for ~ 100 patients)
Savings likely to increase over time since some of the cost items were fixed costs, and some revenues were unknown
Providence Hospital Transitional Care:
What did I learn?
Know your audience
Who is using the results?
What is the study perspective
Results of this study to be used by hospital administrators
CBA not appropriate
Supplemental Nutrition Assistance Program:
Objective
University of Idaho Extension, Nutrition and Food Safety
With Shelly Johnson, Joey Peutz and others
Follow Virginia Tech report for CBA (1996) to calculate the costs and benefits of
UI’s Supplemental Nutrition Assistance Program Education (delivered in Coeur d’ALene)
My role
A complete CBA with sensitivity analysis
My involvement:
Quite proactive
Involved in program design – control and intervention
Introduce new questions to pre- and post-program surveys to improve CBA
Supplemental Nutrition Assistance Program:
Method & Results
Costs and benefits
Collect up-to-date information on costs of health conditions/diseases averted due to improvement in nutritional intake and health behavior
Cost of program delivery (compared to control group)
Results
Control and intervention will take place in
February/March 2012
Analysis results expected middle of next year
Supplemental Nutrition Assistance Program:
What did I learn?
Practical issues
Participant enrollment
Differences in incentives to participants in control and intervention groups
Sample size for robust results
Are cost data in literature suitable for Idaho population?
Challenges
Time consuming process
Uncertain monetary values in CBA
Uncertain QALY values in CUA
Comparison of results under different situations
Validity of assumptions
Other questions? Contact me!
bmandal@wsu.edu
509-335-7553
Reference
Haddix, A.C., Teutsch, S.M., Corso, P.S. (2003). “Prevention Effectiveness: A
Guide to Decision Analysis and Economic Evaluation”. Second Edition,
Oxford University Press.
