Costs, Cost-Effectiveness
and Cost-Benefit Analysis
Brooks Bowden
Center for Benefit-Cost Studies of Education
Teachers College, Columbia University
Clive Belfield
Queens College, City University of New York
IES PI Meeting
December 11 2015
Outline
① Foundations of CA+CEA+CBA = “EA”
② Need for EA
③ Case studies
④ Future of EA
Context:
• EA research at CBCSE.org
• EA teaching: IES Methods Training Grant
(2014-17, #R305B140003)
Case Studies: EA
Socio-Emotional
Learning
interventions
Talent Search
Reading Partners
City Connects
Early literacy
3
Case Studies: Displacement
Socio-Emotional
Learning
interventions
• Outcome
specification
• Program fidelity
Talent Search
Reading Partners
• Treatment contrast
City Connects
• Service mediation
Early literacy
• Power of the test
4
COST, COST-EFFECTIVENESS,
AND COST-BENEFIT ANALYSIS
FOUNDATIONS
4
Basic Ideas
• Money is scarce; must consider if program is affordable
① Costs Analysis (CA): a basic description of how much
resource is needed for a program
•
Cannot just look at if a program works; must also look at
resources needed for it to work
② Cost-Effectiveness Analysis (CEA): the link between
resources needed for programs and their outcomes
• Must look at the economic consequences of a program
③ Cost-Benefit Analysis (CBA): test whether economic
returns justify the cost
5
Costs Analysis
• See if program can be afforded
• Enumerate all resources to implement a policy
• Incremental costs over “next best alternative” or “business-as-usual”
• This is an actual research project using ‘ingredients’ method:
o
o
o
o
Sampling
Quantities of ingredients
Prices of ingredients
Testing for difference in costs
• Average Cost per participant
• AC must be within budget
6
Cost Example
Mentoring
Program
Afterschool
Program
500
1,000
Personnel
$400,000
$480,000
After-school program for
1,000 students:
Facilities
$100,000
$120,000
6 counselors @ $80,000 per
Total Cost
$500,000
$600,000
$1,000
$600
High school mentoring
program for 500 students:
Total number of
students
4 teachers @ $100,000 per
4 classrooms @ $25,000 per
1 gym @ $120,000 per
Average Cost
7
Cost-Effectiveness Analysis
• Find cheapest way to do the thing we have decided to do
• Compares policy alternatives, based on ratio of their costs to
a quantifiable (but not monetized) effectiveness measure
• Compare alternative policies, not rule on single policy
• Compare similar policies: scale, target group, expected
effects
• CE ratio = $Cost/Unit of Effectiveness
• Lowest CE ratio is most cost-effective
9
CEA Example
HS Dropout rate = 20%
Mentoring
program
Afterschool
program
Total cost
$500,000
$300,000
Baseline
dropouts
100
100
5
2
$100,000
$150,000
School size = 500
Mentoring program:
o Costs $1,000 per student
o Reduces dropout rate by 5%
After-school program:
Yield of new
high school
graduates
o Costs $600 per student
o Reduces dropout rate by 2%
CE ratio
9
Cost-Benefit Analysis
• See if it is worth spending money on a program
• Compare policy alternatives, based on ratio of their costs to
quantifiable and monetized effectiveness measures
• Monetized effectiveness measures are “benefits” (e.g.
earnings gains, lower special education spending)
• BC ratio = $Benefits/Cost or B−C = Benefit-Costs
• Worth it if BC ratio > 1 or B−C > 0
11
CBA Example
Mentoring program
reduces dropout rate by
5%
Each new high school
graduate compared to a
dropout over lifetime:
o Earns $120,000 more
o Saves taxpayer $30,000 in
spending on crime
Mentoring
program
Total cost over baseline C
Yield of new high school
graduates
$500,000
5
Total earnings gain B
$600,000
Total crime saving B
$150,000
Net Benefit (B-C)
$250,000
Benefit-Cost Ratio (B/C)
12
1.50
COST, COST-EFFECTIVENESS,
AND COST-BENEFIT ANALYSIS
NEED FOR EA
12
Need for EA in Education
• Many interventions, reforms, programs, policies
• Intensively investigate “What Works”
• Very little guidance on
“..At What Cost?”
“..Is it Worth the Cost?”
13
Examples E
Effects
Class size reduction
✓✓
Reading Recovery
✓✓
High VA teachers
✓✓
Vouchers
✓
Web-learning
=
15
Examples E C
Effects
Costs
Class size reduction
✓✓
Expensive
Reading Recovery
✓✓
Very expensive
High VA teachers
✓✓
No idea
Vouchers
✓
Cheaper (?)
Web-learning
=
Cheaper (?)
16
Examples E C E?
Cost-Effective or Efficient?
Class size reduction
Perhaps
Reading Recovery
High VA teachers
Compared to what?
From whose perspective?
Vouchers
Is it affordable?
Web-learning
17
EA Methods
• They are methods:
o CA: ingredients method
o CEA: CA + evaluation impacts
o CBA: CA + shadow pricing methods
• There are methodological standards
• But not like impact evaluations because
hypothesis / statistical testing is not
determinative
18
EA Practice
• EA is becoming more common
• Slowly and from a low base
• Assumptions rarely harmonized
• Research questions rarely related to each other
• Limited hypothesis testing
19
Value of EA
Conventional answer:
• It will help decision-making
• If it does not help us decide between X and Y, it is
not needed
New answer:
• It will improve education research generally
19
SESSION II
APPLICATIONS &
RECOMMENDATIOS
21
Case Studies
Socio-Emotional
Learning
interventions
• Outcome
specification
• Program fidelity
Talent Search
Reading Partners
• Treatment contrast
City Connects
• Service mediation
Early literacy
• Power of the test
22
OUTCOME SPECIFICATION
CASE STUDY: SOCIO-EMOTIONAL
LEARNING
21
SEL Interventions
• Two full reviews of interventions that enhance
SE skills (Durlak et al., 2011; Sklad et al.,
2012)
• Focus on SE skills or test scores? (Can’t do
both)
• SE skills are broad-based, not task-specific
• Need to look at broad set of outcomes
• Need to weigh those outcomes
• This implies CBA or CBA framework
24
The Economic Value of Social
and Emotional Learning
25
Outcomes of SEL interventions?
Across the literature outcomes were:
• Inconsistent
• More or less
• Student, teacher, classroom, school level
• Durable or ephemeral
• Fast-acting or delayed
24
25
Conclusions
• Better “benefit maps”
• Focus on school-level or class-level effects of
SEL
• Examine how fast-acting and durable
interventions are
• Generally, pre-specify outcomes (Social
Impact Bonds)
28
Clincher
• We propose a cost-effectiveness analysis
• Question always asked is:
How are we measuring effectiveness?
• We don’t know – we are economists
29
PROGRAM FIDELITY
CASE STUDY: TALENT SEARCH
38
Talent Search
• Helps low-income and first-generation college
students complete high school and gain access to
college
• Many services: academic achievement, access to
financial aid, test taking and study skills
assistance, academic advising, tutoring, career
development, campus visits
31
Program Flexibility
Site-specific variation:
• Sites can serve students in G6-12 or cohorts from
6G through 12G; at few or many schools
• Sites are at schools and at colleges
• Programs vary with respect to services
Cost analysis needed to:
① Describe the program
② Estimate in-kind resources
③ Identify most cost-effective sites
32
EA of Talent Search
• Cost Analysis: Collected cost data using
ingredients method from 6 TX sites and 3 FL
sites
• CEA: Linked cost data to effects from
Constantine et al. (2006)
• CBA: Shadow priced estimates of earnings
gains from high school completion and college
attendance
33
34
Cost per Student per Site
(PV at age 18 2010$)
$5,190
$2,730
$640
35
Cost-effectiveness Ratios:
Cost per HS Completer
TX5
Infinity
TX6
$148,090
TX3
TX1
TX4
TX2
FL2
FL3
FL1
$10,830
$-
$30,000
$60,000
$90,000
36
$120,000
$150,000
Read 180
• For struggling readers G1-12 to improve decoding,
fluency, and comprehension skills
• Prescriptive program:
o 90-minutes daily of whole-group, small-group and individualized
instruction + videos + feedback
o Class size 15
• Collected cost data using ingredients method from
three sites (Levin et al., 2007)
37
Read 180 Cost per Student
Site 1
Site 2
Site 3
Students served
6,701
1,080
2,400
Personnel
(teachers)
$320
$950
$70
$50
$400
$60
Equipment/materials
(computers, licenses)
$250
$150
$140
Other (prof. dev., sub
teachers, other)
$-
$10
$10
$610
$1,510
$280
Personnel
(administrators,
technicians, coordinators)
Average Cost
46
JOBSTART
• Education/training program for low-income
youth, including remedial education, GED
preparation, and job placement assistance
• Youth can remain in the program for up to two
years
• MIS data from Cave et al. (2003)
39
JOBSTART Cost per Youth per Site
(PV at 18, 2014$)
NY City
$20,800
LA Jobs Corps
$16,190
Chicago
$14,760
San Jose
EGOS Denver
Corpus Christi
$6,650
$4,710
$4,470
40
Mean = $14,900
SD = $13,400
N =5050
Job Corps: Line Item Costs
Cost per
Mean
SD
Max
Counseling
$4,620
4360
$50,020
Admin
$3,640
3410
$53,540
Voc. training
$2,930
2760
$33,540
Allowance
$2,200
1940
$11,110
Child care
$1,810
1670
$17,450
Outreach
$1,630
1440
$9,860
Education
$1,600
1460
$15,210
Med/dental
$1,270
1180
$13,190
Food
$1,050
1020
$12,280
Conclusions
Cost Analysis yields information on:
• What the program is
• Actual program implementation
• Fidelity
• Many evaluations include (idiosyncratic?)
investigation of program fidelity
• Base fidelity on resource use indicators
43
TREATMENT CONTRAST
CASE STUDY: READING PARTNERS
28
Reading Partners
• Supplemental pull-out reading program:
o
o
o
o
o
o
One-on-one tutoring (2x45 minutes per week)
Dedicated school space and materials
Structured and individualized curriculum
Data-driven instruction
Rigorous ongoing training
Instructional supervision and support
• For struggling students in K-5
• Split responsibility:
o Run with volunteers supported by AmeriCorps Site Coordinators
o Site Coordinators supervised weekly by Program Managers
45
Mobilizing Volunteer Tutors
www.mdrc.org/sites/default/files/ReadingPartners_2015_FR.pdf
46
Evaluating RP
• Multi-site randomized design:
o 1,250 students randomly assigned at 19 school sites
o Grades 2-5, eligible for Reading Partners
o Evaluated during school year 2012-13
o Baseline equivalence of treatment and control
• Cost analysis was part of evaluation:
o What resources were needed to implement RP as per the
evaluation?
o What proportion of RP costs were borne by the school?
o How did these resource requirements compare with the
resources required to implement other supplemental
reading services in these schools?
47
RP Context
•
Cost ≠ Price
•
School selects RP as one option for supplemental reading services
•
Children in need must be served – impact relative to other supplemental
services (not no intervention)
•
Cost Study Sample: 6 of 19 study sites
o Chosen based on strong implementation, geographic diversity, and ability to collect reliable data
•
Data collected via implementation observations, teacher surveys, and
interviews with RP staff, teachers, reading support staff, and school personnel
•
Total social cost calculated by summing the costs of all the resources utilized
to implement program (Levin & McEwan, 2001)
o Included both in-kind and financial costs
o Used national prices
o 2013 $USD
•
Same approach used to assess the cost of the other supplemental reading
programs offered at these six schools
48
RP Costs
• Total social cost calculated by summing costs of
all resources utilized to implement program
(Levin & McEwan, 2001)
o Included both in-kind and financial costs
o Used national prices
o 2013 $USD
• Same approach used to assess the cost of the
other supplemental reading programs offered at
these six schools
49
RP Cost Analysis
Cost per student
RP staff
AmeriCorps members
School staff
Volunteer time + transport
Facilities
Materials/equipment
$690
$930
$90
$1,520
$300
$80
Total Ingredients
$3,610
50
RP Cost per Student
Cost per
student
Cost to
school
Cost to
others
RP staff
$690
$320
$370
AmeriCorps members
$930
School staff
$90
Volunteer time + transport
$930
$90
$1,520
Facilities
$300
Materials/equipment
$1,520
$300
$80
Total Ingredients
$3,610
51
$80
$710
$2,900
20%
80%
Cost of Reading Partners per Program Group Student
Cost
Cost per
Student
Ingredients
Reading Partners staff
AmeriCorps members
School staff
Volunteer time and transportation
Facilities
Materials and equipment
Total ingredients
690
930
90
1,520
300
80
3,610
Distribution of Cost per Student
Cost to
Cost to
Cost to
Cost to
School Volunteers Reading Partners AmeriCorps
690
930
90
1,520
300
390
1,520
320
Fee for service ($)
Portion of net cost per student (%)
52
0
-320
AmeriCorps grant ($)
Net cost per student (Total ingredients +
fee for service + AmeriCorps grant) ($)
80
1,700
-270
270
710
1,520
1,110
270
20
42
31
7
Cost of Other Supplemental
Reading Services
Total Cost Per Student
Reading Partners
Other Supplemental
Services
Site A
$3,450
$1,840
Site B
$3,420
$2,230
Site C
$3,570
$2,680
Site D
$5,190
$1,050
Site E
$4,210
$1,980
Site F
$2,740
$4,890
Pooled
$3,610
$1,780
53
RP Cost versus BAU Cost
School funded
Other resources
$2,900
$80
$1,700
$710
RP
BAU
54
Conclusions
• Schools receive a resource rich intervention
[RP] for a fraction of the cost
• Cost-effective option for under-resourced
schools
• Important to consider treatment contrast and
BAU in costs as well as effects.
55
SERVICE MEDIATION
CASE STUDY: CITY CONNECTS
56
Service Mediation Interventions
These SMIs have attributes of:
• Diagnosis/assessment
• Information
• Referral
• Assignment to services
• Tracking
• Service management
57
Service Mediation Interventions
Examples:
• Diagnosis/assessment: Interventions to change
college remediation
• Information: Letter grades and accountability systems
• Referral: Texts to motivate class attendance
• Assignment: Grade retention
• Tracking effect: FAFSA support to direct college
choice
• Tracking effect: Head Start to motivate future parental
investment
• Service management: City Connects
58
Implications (1)
• Impact of intervention is the sum of diagnosis + service
receipt
o New placement test impact depends on the test + remedial
services received based on the test result
o Grade retention policy impact depends on whether student is
retained + services received by retained v. non-retained
students
• Not possible to disentangle the impact attributable to
the intervention from the impact attributable to service
receipt; service change must be measured
59
Implications (2)
• Interventions might save resources in the long run
o New placement tests have reduced enrollment in remediation
(Scott-Clayton et al., 2014)
• Interventions with no positive effect might be
justified purely in resource terms
• Illustrate using economic framework
60
City Connects (CCNX)
• CCNX diagnoses student need, refers students to
most appropriate support services, manages support
services, and provides some information/services to
students directly
• Walsh et al. (2014b, Table 7) identify significant gains
in academic achievement from CCNX
• CCNX yields average gain of 0.39 standard deviations
in English Language Arts and math by 5th Grade
61
BCA of City Connects
62
Sample, Data, and Methods
• Sample:
o 2 representative K-8 schools in Boston
o 5 community partners per site, based on service intensity
• Data from interviews:
o Site coordinators, teachers, principals, City Connects
central office staff, and community partners
• Costs estimated with ingredients method:
o Opportunity cost not accounting costs/ expenditures
o Separate analyses of quantities/prices and by funder
63
Personnel
Central Program Staff
School Site Coordinators
School Staff
Parental Involvement
Materials
Facilities
Other
Total
Students
Average Cost
Costs for K-5 years per site
(PV at K, 2013$)
City Connects
Mediated Services
$4,868,900
$174,580
$900,040
$93,140
$4,990
$5,040
$42,070
$16,410
$186,910
$774,900
$1,204,560
$5,872,800
780
780
$1,540
$7,530
64
Average Cost per Student
Core +
No Extra
Service
Core +
Full Services
Core +
Estimated
Service Change
$1,540
$1,540
$1,540
$7,530
$3,030
$1,540
$9,070
$4,570
Effects (E)
0.39
0.39
0.39
C/E ratio
$3,950
$23,260
$11,720
City Connects program
Mediated services
Full Cost (C)
65
Service Mediation
• City Connects is effective – large gains in academic
achievement
• If only direct costs are counted, it is very costeffective
• But if all service costs are counted, it is unclear
• Need to perform CBA to fully evaluate City Connects
• Require different impact evaluation (outcomes,
durations)
66
POWER OF TEST
CASE STUDY: EARLY LITERACY
50
Power of Test
• Before identifying impacts, specify power of
test
• But economists are not focused on statistical
significance
• And distribution of ratio of two variables is not
same as distribution of each variable
• So use “power of economic test”
68
Early Literacy
• Estimated costs for all effective early literacy
interventions
• Costs based on ingredients of effective
implementations of interventions
• Related costs to effectiveness from
independent studies
69
Improving Early Literacy
70
Cost per Effect Size Gain
K-PALS
$40
Stepping Stones
$570
Sound Partners
$600
Reading Recovery
$2,090
Fast ForWord
$1,480
Corrective Reading
$6,360
Wilson Reading Systems
$13,390
71
Power of Economic Test
• How much more effective would Wilson have to be to
be as cost-effective as K-PALS?
• How imprecise would estimates of these effects have
to be for K-PALS to lose status as most cost-effective?
• Minimum detectable effect (MDE): the smallest true
effect that has a “good chance” of being found to be
statistically significant (Bloom)
• MDCE: the smallest true CE-ratio that has a “good
chance” of being found to be statistically significant?
72
CA: General Conclusions
• Gives better understanding of the program
• Programs typically cost more than guesstimated or
proposed or in budget documents
• Substantial variation in costs for a given program
• Most programs are not high cost given what is
expected to happen
73
CEA: General Conclusions
• No expectation more costly programs are more
effective
• Very low cost programs are unlikely to be effective
• Cost-effectiveness varies across programs, across
sites, and over time
• Targeted interventions are likely to be more costeffective
74
CBA: General Conclusions
• Targeted interventions are likely to have higher BC
ratios
• Effective programs tend to have very high benefits
• Almost always undercount benefits
• Test scores not as valuable as other skills
75
Friday afternoon! Thank you!
Brooks Bowden and Clive Belfield
www.cbcse.org
76