1
SHEEO Policy Conference
August 9, 2012
Sandy Baum, David Wright, Lee
Holcombe, Kim Hunter-Reed
@HCMStrat
1
Project background
 Started in 2011 by HCM Strategists, with support
from the Bill & Melinda Gates Foundation
 Brought together researchers with different
perspectives on higher education measurement
 Asked research participants to think like
policymakers and leaders
 Asked policymakers and leaders to consider and
respond to research
2
Key questions
How should we account for different student populations
when evaluating higher education outcomes?
 What methods are available or in use for policymakers
and leaders to account for variations in inputs?
 What are the best ways to do it and what are the
practical limitations?
 Can research findings be translated into policy?
3
4
Project components
 Seven white papers
 Synopsis paper
 Literature review
Coming soon . . .
 Policy briefs
 Toolkit for states
 Outreach--sharing findings, including more
voices, continuing to solicit input
4
Some general lessons and conclusions
 Input adjustments make a big difference
 Policymakers should make an effort to adjust for
inputs when institutional comparisons have
stakes attached
 Imperfect input adjustments are better than none
 Need to be clear about outcomes
5
Readers will find specific guidance on
 Important variables to include
 Examples of adjustment models and statistical
techniques
 Use of survey for input adjustments
 College learning assessments
 Strategies for working with messy administrative
data
 Strengths and weaknesses of U.S. News input
adjustments
 Many potential unintended consequences
6
Where to find out more . . .
Context for Success website:
www.hcmstrategists.com/contextforsuccess
Questions? Comments?
7
Advice for Moving Forward
Know the purpose, audience and questions to be
answered

Consumers: Judging quality and value

Institutions: Improving

Accountability: Success and cost-effectiveness
8
Advice for Moving Forward
Comparing outcomes: Focus on similar institutions



Differences in credentials
Goals of students
Role of transfer
Selecting variables

Preparation

Demographics

Attitudes, goals, behaviors
9
Context for Success White Papers
Charles Clotfelter. "'Context for Success' Synopsis
Paper."
Thomas Bailey and Di Xu. "Literature Review: Input
Adjusted Graduation Rates and College
Accountability."
Peter Riley Bahr. "Classifying Community Colleges
Based on Students' Patterns of Usage."
Thomas Bailey. "Developing Input Adjusted Measures
of Community College Performance."
Jesse M. Cunha and Darwin W. Miller. "Measuring
Value-Added in Higher Education."
10
Context for Success White Papers (cont.)
Robert Kelchen and Douglas N. Harris. "Can 'Value
Added' Methods Improve the Measurement of College
Performance?"
Stephen M. Porter. "Using Student Learning as a Measure
of Quality in Higher Education."
John Pryor and Sylvia Hurtado. "Using CIRP Student
Level Data to Study Input Adjusted Degree
Attainment."
David Wright, Matthew Murray, Bill Fox, Celeste
Carruthers, Grant Thrall. "College Participation,
Persistence, Graduation, and Labor Market Outcomes."
11
An Input-Adjusted Framework for
Assessing the Effectiveness of Tennessee’s
Higher Education Institutions
Context for Success project
Celeste Carruthers,a William Fox,a
Matthew Murray,a Grant Thrall,b
David L Wrightc
a University
of Tennessee, Knoxville; b University of Florida;
c Tennessee Higher Education Commission
Background
• Policy relevance:
– Complete College TN Act ties HEI funding to
student outcomes.
• Assumption:
– Funding incentives must take into account student
characteristics that are likely to be important
determinants of postsecondary outcomes.
Context for Success paper
SHEEO, 08-09-2012
Research questions
• Purposes:
1) We statistically test whether some TN
institutions were more effective than others at
advancing the outcomes of 2002 entering firsttime freshmen.
Context for Success paper
SHEEO, 08-09-2012
Research questions
• Purposes:
2) We go on to test whether accounting for student
inputs (ACT aptitude, age, low-income home
neighborhood, LifeMode marketing profile)
changes the ranking of institutions.
— Is model fit improved by supplementing or
substituting our standard stock of administrative
data with proprietary data on consumer tapestry
profiles (called LifeModes) or with Census
information?
Context for Success paper
SHEEO, 08-09-2012
Method
• Cohort: 2002 first-time freshmen in TN public
universities and community colleges
• Design:
– Multivariate OLS regression
– Used fixed effects to isolate institutional
effectiveness
Context for Success paper
SHEEO, 08-09-2012
Method
• Compared models with four different sets of
predictor variables:
1) naïve model
2) standard state agency administrative data
3) Census neighborhood data
4) proprietary LifeModes profiles
• Outcomes considered: student progression,
transfer, graduation, and near-term labor market
success
Context for Success paper
SHEEO, 08-09-2012
Selected findings
• Timely completion:
– Females, higher entrance exam scores, and adult
status were associated with more timely degree
completion.
– Low-income status delayed completion.
– Additional semesters in college reduced earnings
later on.
Context for Success paper
SHEEO, 08-09-2012
Selected findings
• Earnings:
– Females and higher entrance exam score were
associated with higher earnings in the years
immediately following college.
– A significant black-white earnings gap favored
white students.
Context for Success paper
SHEEO, 08-09-2012
Six-year graduation rates (bachelor’s) across 4-year
schools, relative to one omitted institution, unadjusted
(blue) or adjusted for student and home characteristics
0.35
0.30
Unadjusted
0.25
Adjusted for ACT,
demographics
0.20
0.15
Adjusted for ACT,
demographics, Census
0.10
0.05
0.00
A
B
C
D
E
F
G
-0.05
-0.10
H
Adjusted for ACT,
demographics, Census,
and LifeModes
4-Year Institution
Context for Success paper
SHEEO, 08-09-2012
Four-year graduation rates (associate’s) across 2-year
schools, relative to one omitted institution, unadjusted
(blue) or adjusted for student and home characteristics
0.1
0.08
Unadjusted
0.06
0.04
Adjusted for ACT,
demographics
0.02
0
-0.02
A
B
C
D
E
F
G
H
I
J
K
-0.04
L
Adjusted for ACT,
demographics, Census
Adjusted for ACT,
demographics, Census,
and LifeModes
-0.06
-0.08
2-Year Institution
-0.1
Context for Success paper
SHEEO, 08-09-2012
Observations and open questions
• Institutions do matter: even after controlling
for all observed student inputs, some
institutions are significantly more effective
than others.
– Also see Cunha and Miller in this series.
• Student inputs also matter: institutions’
apparent effectiveness changes when we
account for student aptitude and background.
• The gains from more data are not always high.
Context for Success paper
SHEEO, 08-09-2012
Observations and open questions
• Why are some institutions more effective than
others? Institutional effects are a black box.
• Student goals and objectives may not align
with traditionally tracked student outcomes.
– Especially in community colleges!
• Assessing and comparing student growth
across fields and campuses is challenging but
not insurmountable.
Context for Success paper
SHEEO, 08-09-2012
Conclusions
• Outcomes and institutional best practices should
be included in performance-based funding. (p.6)
• Incentivizing colleges for student progression
and completion relative to expectations is an
improvement over unadjusted rewards…
• … but, be aware that outcomes will vary
according to which students inputs are factored
into expectations for colleges and universities.
(p.32)
Context for Success paper
SHEEO, 08-09-2012
Input Adjusted Output Measures
Presented By: Lee Holcombe, Director, Higher Education
Policy Institute
Analysis: Jesse Cunha, of the Naval Postgraduate School
A Project of the Texas Higher Education Coordinating Board
and Trey Miller of the RAND Corporation
Goals
 Develop and implement a methodology that
generates quantitative metrics of institutional
performance.
 Come as close as possible to causal estimates
of an institution’s impact on a particular
outcome.
 Demonstrate benefits of collecting additional
data.
 Develop a set of concrete and actionable
policy recommendations for practitioners and
policymakers.
Higher Education Policy Institute
26
Outcomes
 Persistence into 2nd year of college:
 Attain 30 credit hours at any public or private
four-year college in Texas by the end of the 6th
academic year after graduating high school
 Graduation:
 Receive a Bachelor’s degree at any public or
private four-year college in Texas by the end of
the 6th academic year after graduating high
school
 Earnings:
 Sum of 4 quarterly earnings in the 8th calendar
year after high school graduation.
Higher Education Policy Institute
27
Data
 Higher education
 Demographic variables, application and
acceptance data, credits attempted, degree
completion
 K12 (Secondary)
 Demographic variables, test scores, courses taken
 Unemployment Insurance earnings
 From College Board:
 SAT scores and survey info
Higher Education Policy Institute
28
Sample
 Graduates of Texas public high schools in 5 cohorts
(1998-2002)
 Include those Texas public high school graduates who
ever enrolled in a Texas public university (up to
2010).
 217,723 enrollees, 169,239 of whom had earnings
greater than $2,000
Higher Education Policy Institute
29
Findings
 Large differences in unconditional outcomes across
colleges diminish as each set of controls is added to
the model.
 Adding high school indicators and courses changes
results much more than adding race and gender.
 Different results across outcomes.
 Measures are highly variable over time.
Higher Education Policy Institute
30
Policy Recommendations of Report
 Assessment of institutional quality should involve a
broad set of outcomes representing multiple
dimensions of institutional performance.
 Researchers should work with state policymakers to
develop models that reflect state priorities.
 Do not assume that results are causal.
 Create broad classifications of institutional
performance as opposed to explicit rankings.
Higher Education Policy Institute
31
Policy Response
 Elusive quest for causality.
 Excludes many non-traditional students.
 Consider improvements within-institution
 Dig inside the higher education black box.
 Provide higher levels of information and support.
 Use of more nimble data systems.
 Consider learning outcomes.
Higher Education Policy Institute
32
Where to find out more . . .
COMING SOON
Context for Success website:
www.hcmstrategists.com/contextforsuccess
Questions? Comments?
33