Observations on Developing a
Postsecondary Institution
Rating System (PIRS)
Dr. Braden J. Hosch
Asst. Vice President for Institutional Research, Planning & Effectiveness
Stony Brook University
January 22, 2014
CONTEXT (1)
Current role:
Asst. VP for Institutional Research, Planning &
Effectiveness at Stony Brook University (NY)
IPEDS National Trainer, Advisory Board Member
Former role:
Director of Policy & Research,
Connecticut Board of Regents for Higher Ed
2
CONTEXT (2)
• Connecticut changed its approach to higher education
accountability in 2011
• Broad stakeholder involvement
• 5 goals with 23 indicators, disaggregated
• Used data publicly available through IPEDS
• Institutional performance measured over time
compared to similar institutions
• Accountability more than consumer information
• Work in progress
See Higher Education Coordinating Council. (2012). Public Policy Framework for Higher
Education. http://www.ct.edu/files/opr/A_CTFramework.pdf
3
CHALLENGES
Comparison Groups
Graduation Rates
Learning Outcomes
Affordability
Employment Outcomes
4
COMPARISON GROUPS
• Exclusively mathematical approach to identify institutions
for comparison groups was elusive
• A balance of data, statistics, and judgment was needed
• Data from IPEDS based on about 10 institutional characteristics
generated lists of 20-50 institutions in most cases
• Outliers remained a problem, likely due to data quality / unique
state practices
• Campuses asked to add or remove institutions from the
comparison group based on judgment to arrive at 10-15
comparison institutions
5
GRADUATION RATES
• Controls for student inputs are essential to the
interpretation of graduation rates; individual factors
matter much more than institutional factors
• Data from HERI illustrate this relationship, independent of
institutional effects
6
INSTITUTION-BASED 6-YEAR GRAD RATES BY
SAT COMPOSITE SCORE
Source: IPEDS Data Center
Public Institutions
Private Not-for-Profit Institutions
7
Graduation Rate (percent)
INDIVIDUAL-BASED GRADUATION RATES
BY HIGH SCHOOL GPA
100
90
80
70
60
50
40
30
20
10
0
79.3
70.6
59.8
48.7
36.6
21.2
6.3
C or less
27.7
9.8
58.3
47.8
35.9
25.2
15.5
C+
BB
B+
AHS GPA as Reported on the CIRP Freshman Survey
4-Years
A+, A
6-Years
Source: Higher Education Research Institute (UCLA)
8
Graduation Rate (percent)
STONY BROOK UNIV. GRADUATION RATES
BY HIGH SCHOOL GPA
100
90
80
70
60
50
40
30
20
10
0
79.1
58.9
33.9
Below 87
4-Years
62.6
40.2
87-89.9
HS GPA
67.3
46.1
90-92.9
71.9
60.6
50.3
93-95.9
96+
6-Years
Source: SBU Office of Institutional Research, Planning & Effectiveness
9
Graduation Rate (percent)
STONY BROOK UNIV. GRADUATION RATES
BY HIGH SCHOOL GPA – AFRICAN AMERICAN ONLY
100
90
80
70
60
50
40
30
20
10
0
75.2
74.5
66.3
80.4
71.9
56.9
51.7
52.3
49.1
87-89.9
HS GPA
90-92.9
93-95.9
39.7
Below 87
4-Years
96+
6-Years
Source: SBU Office of Institutional Research, Planning & Effectiveness
10
COMPLETIONS PER 100 FTE ENROLLMENT
Connecticut adopted a completions per FTE metric
intended to supplant graduation rate
• Statistical adjustments were necessary
– Lag enrollment by 2 years
– Weight certificates by 1/3
– Degree-seeking UG enrollment only
• Data adjustments were necessary
• Results not comparable across sectors
• Cannot be interpreted without reference groups
11
LEARNING OUTCOMES
Deemed very important for quality assurance but
measurement deferred pending further research
Research had demonstrated
•
•
Testing is sensitive to recruitment practices and testing conditions
Student motivation affected institutional results
State pursuing external validation of learning using
“authentic assessment” through a nine state
multi-state collaborative
12
AFFORDABILITY AND NET PRICE
• Net price involves reasonable components, but
wide variation exists in how institutions determine
costs associated with room & board and other
expenses for students living off-campus.
• Regulatory or legislative action needed to define
calculation method
13
EXAMPLE NET PRICE ANOMALIES
$25,000
$20,000
Expenses Living Off-Campus, Not with Family:
4 Institutions within 1.6 miles
room and board
other expenses
$8,528
$15,000
$10,000
$5,000
$7,410
$2,146
$6,840
$5,823
Assoc
(public under 4-yr)
Assoc
(public rural)
0
0.4
$13,117
$3,450
$4,900
$0
Private, for-profit Private, not-for-profit
non-degree granting
4-yr Bacc
1.5
1.6
Distance (miles)
Source: IPEDS Data Center, components are for 2011-12
14
EMPLOYMENT OUTCOMES
Available employment data have severe limitations
• Alumni survey response rates for public institutions typically range
20-30%, with likely high levels of non-respondent bias
• State unemployment insurance (UI) data do not cross state lines or
identify full-time/part-time employment or partial/full quarter earnings
• IRS data are problematic for joint returns
• Age and prior employment confound central tendencies – more
research is needed
15
EXAMPLE WAGE DATA ANOMALIES
Avg. Weekly Wages & Change in Weekly Wages for
Undergraduate Completers
$1,200
$1,076
$1,000
$800
$600
$400
$540
$473
$254
$451
$507
$472
$270
$200
$0
CT Community Colleges
Charter Oak State College
(Public online college)
Weekly Wages Upon Entering Employment
CT State Universities
(Public Regional)
Univ. of Connecticut
(Public Flagship)
Change in Wages from Start of Program
Data gathered from unemployment insurance records through CT Dept. of Labor
Source: CT Employment & Training Commission Legislative Report Card (2013)
16
RECOMMENDATIONS
• Consider institutional input for comparison
groups
• Adopt individualized graduation rate calculator
• Define cost of attendance methodology for offcampus arrangements
• Defer inclusion of and support research on:
•
•
Learning outcomes
Employment outcomes
17
REFERENCES
Connecticut Dept. of Labor and Connecticut Dept. of Higher Education (2010). Building Connecticut’s
Workforce: Report on 2007-08 Graduates. Retrieved January 13, 2014 from
http://www.ctdhe.org/info/pdfs/2010/HigherEdReport-2008grads.pdf
Connecticut Higher Education Coordinating Council. (2012). Public Policy Framework for Higher
Education. Retrieved January 10, 2014 from http://www.ct.edu/files/opr/A_CTFramework.pdf.
DeAngelo, L., Franke, R., Hurtado, S., Pryor, J. H., & Tran, S. (2011). Completing college: Assessing
graduation rates at four-year institutions. Los Angeles: Higher Education Research Institute, UCLA.
Hosch, B. (2008). Institutional and student characteristics that predict retention and graduation rates.
Annual Conference of the North East Assoc. for Institutional Research, Providence, RI.
Hosch, B. (2012). Time on test, student motivation, and performance on the Collegiate Learning
Assessment: Implications for institutional accountability. Journal for Assessment and Institutional
Effectiveness 2(1): 55-76.
National Governors Association. (2010). Complete to Compete: From Information to Action: Revamping
Higher Education Accountability Systems
18