Moving from Aggregate to Subgroup
Analysis to Assess Equity and Outcomes:
Unpacking the Meaning of Transformative
Change
Matt S. Giani, Debra D. Bragg, & Heather L. Fox
Office of Community College Research and Leadership
University of Illinois at Urbana-Champaign
TCI Research Symposium
November 11, 2015
Equity in TCI
“Transformative change refers to implementing, sustaining,
and scaling change that produces unprecedented results
without sacrificing the historic commitment of community
colleges to access, opportunity, and equitable outcomes.”
Assessing Equity in TAACCCT
Q: How do we know if TAACCCT reforms impacted
institutional equity?
DOL does not require that evaluations assess whether
institutions became more equitable as a result of TAACCCT
Research Questions
1) What were the postsecondary outcomes of H2P
students, and how did outcomes vary according to
students’ demographic and socioeconomic background?
2) What impact did H2P make on the outcomes of
healthcare students?
3) To what extent did H2P reduce gaps in outcomes
stemming from student demographic and
socioeconomic characteristics?
H2P Consortium
H2P’s Vision
“Not only produce a highly skilled healthcare workforce but
also galvanize a national movement to dramatically
redesign and enhance health professional education and
training though national curricular reform, industry
engagement, innovative practices and programs, and
intensive usage of data and accountability systems to
ensure student success and program excellence” (H2P
Proposal, 2009, pp. 1-2).
H2P Co-Grantees
• Anoka-Ramsey Community College - Coon Rapids, MN
• Ashland Community & Technical College -Ashland, KY
• Cincinnati State Technical & Community College - Cincinnati, OH, (H2P Lead)
• El Centro College - Dallas, TX
• Jefferson Community & Technical College - Louisville, KY
• Malcolm X College - Chicago, IL
• Owens Community College - Toledo, OH
• Pine Technical & Community College - Pine City, MN
• Texarkana College - Texarkana, TX
H2P’s Strategies
1)
2)
3)
4)
5)
6)
Online assessment and enhanced career guidance
Contextualized developmental education
Competency-based core curriculum
Industry-recognized stackable credentials
Enhanced retention support
Training programs for incumbent health professions
workers
7) Enhance data and accountability systems
8) Galvanize a national movement
1. Online
Assessment &
Career Guidance
(No. students,
No. credits)
Participant
2. Contextualized
Dev Ed (No.
students enrolled,
No. students
increase to college
level, No. students
earning academic
credit)
6. Incumbent
training (No.
enrolled, Min.
semester credit)
7. Data (No. students w/
education and employment
records)
3. Core Curriculum
(No. students
enrolled, No.
students complete)
•
•
4. Retention Services
(No. enroll fall to
spring, No. enroll fall
to fall, % completed of
attempted)
•
5. Stackable
credentials (No.
certificates per 100
FTE, No. certificates)
•
•
•
Outcomes
Employment
Employment
retention
Average
earnings
Credit
attainment
Certificate
attainment
(< 1 and >1)
Degree
attainment
8. National movement (No.
using labor market info, No.
enrolled in core curriculum)
H2P Participants – 6,569* Total
372
533
451
Anoka-Ramsey (ARCC)
Ashland (ACTC)
305
Cincinnati State (CSTCC)
556
El Centro (ECC)
1836
460
Jefferson (JCTC)
Malcolm X (MXC)
436
Owens (OCC)
1620
Pine (PTCC)
Texarkana (TC)
*Includes Type 1 and Type 2 Participants. Total accurate as of Fall 2014
Methods
Research Questions
1) What were the postsecondary outcomes of H2P
students, and how did outcomes vary according to
students’ demographic and socioeconomic background?
2) What impact did H2P make on the outcomes of
healthcare students?
3) To what extent did H2P reduce gaps in outcomes
stemming from student demographic and
socioeconomic characteristics?
Analytic Techniques
1) What were the postsecondary outcomes of H2P
students, and how did outcomes vary according to
students’ demographic and socioeconomic background?
• Analyzed mean retention and attainment rates by
demographic group
• Used logistic regression to estimate the impact of
demographic characteristics on retention and attainment
Analytic Techniques
2) What impact did H2P make on the outcomes of
healthcare students?
• Drew on retrospective sample of healthcare students
who enrolled in H2P colleges in fall 2009
• Restricted sample to students in LVN and ADN programs
• Estimated impact of H2P using propensity score matching
(PSM)
Analytic Techniques
3) To what extent did H2P reduce gaps in outcomes
stemming from student demographic and
socioeconomic characteristics?
• Selected particular student subgroups (students of color,
Pell eligible, non-employed)
• Fit separate PSM models to each subgroup
• Analyses reveal how impact of H2P varied by student
subgroup
Data Categories
•
•
•
•
•
Demographic
Course Data
Credential Data
UI Wage Data
Strategy Data (credit for prior learning, retention
services, etc.)
Outcome Variables
1)
2)
3)
4)
Any credential
Any credential or still enrolled by fall 2014
Long term certificate or associate’s degree
Associate’s degree
Control/Matching Variables
•
•
•
•
Demographic
• (race/ethnicity, age, gender, Pell eligibility, whether
first-time college student)
Course data
• (dev ed participation, pre-cohort credits)
UI Wage Data
• (whether employed quarter before enrollment, prior
earnings)
College
Results
Credential and Retention Rates, by College
100%
91%
90%
77%
80%
68%
70%
62%
60%
50%
41%
27%
43%
40%
23%
32%
22%
20%
24%
46%
18%
71%
68% 66%
64%
54%
40%
30%
67%
83%
83%
60%
49%
46%
19%
11%
10%
8%
12%
14%
PTCC
TC
0%
All H2P
ARCC
ACTC
Credential
CSTCC
ECC
JCTC
Retained, No Credential
MXC
OCC
Retained or Credential
Credential Rates, by Credential Type
58.7%
16.9%
10.2%
11.3%
Long-Term
Certificate
Associate’s
Degree
2.9%
No Credential
Very Short
Short-Term
Certificate
No Credential, by Race/Ethnicity
84.0%
72.8%
61.6%
68.8%
61.4%
57.9%
49.0%
40.9%
AI/AN
Asian
Black
Latino Multi-race
NH/PI
White
Unknown
Long Certificate/Associate’s Attainment, by
Race/Ethnicity
35.0%
30.0%
25.0%
18.2%
15.1%
20.0%
18.6%
15.0%
8.4%
7.5%
7.0%
8.5%
8.9%
Asian
Black
Latino
10.0%
5.0%
13.6%
0.0%
AI/AN
Long Certificate
12.8%
12.5%
Multi-race
5.3%
0.0%
NH/PI
Associate’s Degree
White
1.0%
3.9%
Unknown
Credential Attainment, by Pell Eligibility
70.0%
60.0%
57.5%
60.4%
50.0%
40.0%
30.0%
15.8% 16.8%
20.0%
13.0%
10.0%
2.6%
8.7%
11.2% 11.0%
3.2%
0.0%
No Credential
Very Short
Certificate
Short Certificate
Eligible
Not eligible
Long Certificate
Associate’s Degree
Credential Attainment, by Pre-H2P
Employment
70.0%
60.0%
57.9% 59.2%
50.0%
40.0%
30.0%
18.6%
20.0%
15.6%
10.1% 10.3%
10.0%
3.7%
9.7%
12.5%
2.4%
0.0%
No Credential
Very Short
Certificate
Short Certificate
Employed
Not Employed
Long Certificate
Associate’s Degree
Logistic Regression
Credential or
Retained
Odds
Sig.
Ratio
Race/Ethnicity1 (White)
American Indian /
.964
Alaskan Native
Asian
1.364
Any Credential
Long Certificate or
Associate's
Odds
Sig.
Ratio
Odds
Ratio
Sig.
.946
1.069
.900
.794
.123
.978
.908
Associate's
Odds
Ratio
Sig.
.639
.834
.751
.878
.508
.921
.709
Black
.602***
.000
.576***
.000
.615***
.000
.642**
.002
Latino
1.029
.814
.666**
.003
.621***
.001
.410***
.000
Multi-race
Native Hawaiian /
Pacific Islander
Unknown
.533
.102
.417*
.035
.446
.150
.000
.998
1.374
.597
1.189
.751
.183
.104
.255
.194
.836
.182
.407***
.000
.161***
.000
.043***
.000
Impact of H2P on Attainment (PSM)
20.0%
18.0%***
18.0%
16.0%
14.0%
12.0%
10.0%
8.0%
7.0%
6.0%
4.0%
2.0%
0.4%
0.0%
All Impact Sample
LVN/LPN
RN
Impact of H2P on Attainment –
Students of Color
30.0%
25.4%***
25.0%
20.0%
15.0%
10.0%
6.5%
5.0%
0.0%
Non-White
White
Impact of H2P on Attainment –
Pell-Eligibility
8.0%
7.0%
6.0%
5.0%
4.0%
3.0%
2.0%
1.0%
0.0%
-1.0%
-2.0%
-3.0%
6.8%
-1.7%
Eligible
Non-Eligible
Impact of H2P on Attainment –
Pre-H2P Employment
8.0%
6.9%
6.0%
4.0%
2.0%
0.0%
-2.0%
-4.0%
-2.8%
Non-Employed
Employed
Discussion
Limitations
•
•
•
•
•
No data on transfer students
Analyses focused on longer-term programs
PSM can’t control for unmeasured differences
Subgroup analyses limited by sample sizes
Estimates show how big of an effect, but not why the
effect occurred (requires mixed-methods evaluation)
Conclusions
•
•
•
•
•
H2P students more likely to receive credential compared
to Retro healthcare students
H2P students of color had significantly lower attainment
rates than white students
*H2P students of color had significantly higher
attainment rates than Retro students of color
Results suggest H2P improved equity of outcomes for
students of color
Ongoing research is unpacking this effect
• Questions?
• How can we help?
Contact Info:
Matt S. Giani – mgiani@illinois.edu
Debra D. Bragg – dbragg@illinois.edu
Heather L. Fox – hlfox2@Illinois.edu
Office of Community College Research and Leadership (OCCRL) –
http://occrl.Illinois.edu
© 2015 Board of Trustees, University of Illinois