Accountability Reporting for
California Community
Colleges
Patrick Perry
Vice Chancellor of Technology,
Research, & Info. Systems
CCC Chancellors Office
1
Data Preamble
 “Information
is the currency of democracy.”
-Thomas Jefferson

“Get your facts first, then you can distort
them as you please.” -Mark Twain

“In the twenty-first century, whoever controls
the screen controls consciousness,
information and thought.” -Timothy Leary
2
The CCC System
 109
campuses, 72 districts, all locally
governed
 2.6 million students (annual unduplicated)
 1.1 million FTES (annual)
 35% white; half over age 25; 70% part-time
 No admissions requirements
 $20/unit; 40% get fees waived
 Highest participation rate of any CC system
in US; 25% of all CC students are CCC
3
Participation (and Fees)
4
CCC Chancellor’s Office
 Weak
authority; powers vested locally
 Unitary MIS data collection (1992)
Student, faculty, course, section, session,
grade level detail
 Data collected end of term, 3x/yr
 Used for IPEDS, apportionment,
accountability, research, online data mart

5
History of CCC Accountability
 Simple
reporting, fact books until 1998
 1998: State provides $300m ongoing in
exchange for accountability reporting

“Partnership for Excellence” was born
CCC developed report in isolation
 CCC allowed to determine “adequate progress”
 “Contingent funding” never triggered


Used 5 metrics to measure system and collegelevel performance
6
PFE Metrics
 Annual
volume of transfers to CSU/UC
 Annual volume of awards/certificates
 Rate of successful course completions
 Annual volume of Voc. Ed. Course
completions
 Annual volume of basic skills
improvements (lower to higher level)

4 of 5 are volume metrics, only 1 rate
7
The State Said:
 Your
metrics allow for no adequate college
comparisons
 Your method of determining “adequate
progress” is suspicious
 You only look good because you are growing
 Partnership over (2001), but keep reporting,
(until 2004)

we have to spend your money buying energy
from Enron
8
What Happened Next
 Gov.
Gray Davis: recalled for spending
money buying energy from Enron
 Replaced
by
“The Governator”
9
The Governator
 Likes
Community Colleges
Comes from a country that has European
“academic bifurcation” (Austria)university vs trade paths
 Attended Santa Monica Community
College

 Took
ESL, PE, bookkeeping,
micro/macroeconomics
 Transferred to U. Wisconsin-Superior
10
And Arnold Said:
 We
shall haves deez accountabeelity
seeztem for de community collegez.
A
bill was passed to create the
framework, and eventually the
framework was enacted.
 Named: Accountability Reporting for
Community Colleges (ARCC).
11
Arnold Said:
 There
shall be no pay for performance,
but there will be the ability to compare
performance.
12
We Said:
 Some
metrics will be system only;
others will be at college-level
 College metrics will be rates (to
mitigate size for comparison)
 No rankings—we will compare colleges
against their “peers”
 No $$$=ARCC is a “dashboard”
accountability report.
13
Arnold Said:
 Colleges
need to address their
performance annually to the State.
14
We Said:
 Colleges
are more responsive to their
local district Board; annual requirement
to take local ARCC results to local
Board and submit minutes to State
 Colleges must submit 500 word
response, which becomes a part of the
final report.
15
Arnold Said:
 The
report shall be done in
collaboration with the State, not in
isolation.
16
We Said:
 The
Dep’t of Finance, Leg Analyst, and
Secretary of Education shall be a part of
the technical advisory committee (along
with CCC researchers and
stakeholders).
 We
will either succeed or fail together.
 This was a really smart move.
17
ARCC
 The

Model:
Measures 4 areas with 13 metrics:
 Student
Progress & AchievementDegree/Certificate/Transfer
 Student Progress & AchievementVocational/Occupational/Workforce Dev.
 Pre-collegiate improvement/basic skills/ESL
 Participation

“Process” is not measured
18
Student Prog. & Achievement:
Degree/Cert/Xfer
 College:
Student Progress & Achievement Rate(s)
(SPAR)
 “30 units” Rate for SPAR cohort
 1st year to 2nd year persistence rate

 System:
Annual volume of transfers
 Transfer Rate for 6-year cohort of FTF’s
 Annual % of BA/BS grads at CSU/UC
who attended a CCC

19
Student Prog. & Achievement:
Voc/Occ/Wkforce Dev
 College:

Successful Course Completion rate:
vocational courses
 System:
Annual volume of degrees/certificates by
program
 Increase in total personal income as a result
of receiving degree/certificate

20
Precollegiate
Improvement/Basic Skills/ESL
 College:
Successful Course Completion rate: basic
skills courses
 ESL Improvement Rate
 Basic Skills Improvement Rate

 System:

Annual volume of basic skills
improvements
21
Participation
 College:

None yet…but coming.
 System:

Statewide Participation Rate (by
demographic)
22
Major Advancements of
ARCC
 Creating
a viable alternative to the GRS
Rate for grad/transfer rate.
 Finding transfers to private/out of state
institutions.
 Doing a wage study.
 Geo-mapping district boundaries.
 Creating peer groups.
23
Defining Grad/Transfer Rate
 Student
Progress & Achievement Rate
(SPAR Rate)

IPEDS-GRS for 2-yr colleges stinks:
 No
part-timers
 How do you define degree-seeking?
 Tracking period too short
 Outcomes counting methodology terrible


AA/AS/Cert counted before transfer
Transfer to 2-yr college is counted
24
SPAR Rate
 Defining

the cohort:
Scrub “first-time” by checking against past
records (CCC, UC, CSU, NSC)
25
SPAR Rate
 Define
“degree-seeking” behaviorally
for CC populations

Not by self-stated intent; this is a poor
indicator
 Behavior:
did student ever attempt
transfer/deg-applicable level math OR
English (at any point in academic
history)

Students don’t take this for “fun”
26
Defining Degree-Seeking
Behaviorally
 Separates
out remedial students not yet
at collegiate aptitude

Measure remedial progression to this
threshold elsewhere
 Creates
common measurement “bar” of
student aptitude between colleges

Same students measured=viable
comparison
27
SPAR Rate-Unit Threshold
 CCC
provides a lot of CSU/UC
remediation
Lots of students take transfer math/Eng
and leave/take in summer
 Should not count these as success or “our”
student

 Set
minimum unit completed threshold
(12) for cohort entrance

Any 12 units in 6 years anywhere in system
28
SPAR Denominator:
 First-Time
(scrubbed)
 Degree-seeking (at any point in 6 years,
attempt transfer/degree applicable
math or English)
 12 units (in 6 years)
 This
represents about 40% of students
in our system
29
SPAR Numerator
 Outcomes



Earned an AA/AS/certificate; OR
Transfer: to a 4-yr institution; OR
Become “transfer-prepared”;OR



Completed 60 xferable units
Became “transfer-directed”:


the State wants:
Completed both xfer level math AND English
No double-counting, but any outcome counts
SPAR Rate=51%
30
Tracking Transfers
 SSN-level
matches with CSU, UC
 Nat’l Student Clearinghouse for private,
proprietary, for-profit, out of state
Match 2x/yr, send all records since 1992
 Update internal “xfer bucket”

 Works
great for cohort tracking
 Needed method for “annual volume”
31
Tracking Transfers
 Annual

CSU/UC: they provide these figures based
on their criteria
 We

Volume of Transfers
didn’t want to redefine this
Private/Out of State: NSC “cross-section”
cut method
 Validated
source

against CSU/UC xfers from NSC
Added another 30% to annual volumes
32
97-98 98-99 99-00 00-01 01-02 02-03 03-04 04-05 05-06 06-07
FTF
→
→
→
→
→
→
→
→
T
FTF
→
→
→
→
→
→
→
R
FTF
→
→
→
→
→
→
A
FTF
→
→
→
→
→
N
FTF
→
→
→
→
S
FTF
→
→
→
F
FTF
→
→
E
FTF
→
R
FTF
06-07
MIN
12
UNITS
33
Sector
01-02
02-03
03-04
04-05
05-06
06-07
CSU
50,473 50,746 48,321 53,695 52,642 54,391
UC
12,291 12,780 12,580 13,211 13,462 13,874
ISP
17,070 15,541 18,100 18,365 17,840 18,752
OOS
10,762 10,540 11,150 11,709 11,726 11,825
Total
90,596 89,607 90,151 96,980 95,670 98,842
34
Transfer: Sector of Choice
% to
Instate % to Out
Private of State
% to UC
% to
CSU
White
17.9%
60.7%
11.0%
10.4%
AfrAm
11.5%
51.2%
18.1%
19.2%
Hisp/Lat
15.1%
67.7%
12.1%
5.1%
Asian
37.0%
49.9%
9.2%
3.9%
35
Demography of Transfer
Demog FTF
All
XFER(06-07) Stdents Stdents CSU
XFERUC
XFERISP
XFEROOS
AfrAm
9%
8%
5%
3%
11%
13%
Asian
11%
12%
12%
26%
8%
7%
Hisp/
Latino
35%
29%
23%
16%
23%
13%
White
29%
35%
37%
40%
44%
55%
36
The Rise of The Phoenix
96-97
2,166
97-98
2,829
98-99
3,374
99-00
4,194
00-01
5,055
01-02
5,586
02-03
6,515
03-04
8,222
04-05
8,585
05-06
8,134
06-07
9,216
37
Who Transfers to Phoenix?
Ethnicity
UC
Asian
29.3%
14.2%
4.6%
2.4%
5.2%
16.8%
Hispanic/Latino
13.6%
23.8%
28.6%
White
39.1%
43.6%
37.5%
African American
CSU
Phoenix
38
Wage Study
 What
was the economic value of the
degrees (AA/AS/certificate) we were
conferring?
 Required

data match with EDD
Had to pass a bill changing EDD code to
allow match
39
Wage Study
 Take
year
all degree recipients in a given
Subtract out those still enrolled in a CCC
 Subtract out those who transferred to a 4yr institution

 Match
degree
wage data 5 years before/after
40
Wage Study
 Separate
out two groups:
Those with wages of basically zero before
degree
 Those with >$0 pre wage

 The
result: The Smoking Gun of Success
41
Income Increase from Attaining CCC Degree or Certificate
60,000
50,000
Income (in dollars)
40,000
30,000
CCC Income Data: Received any
award during 2000-2001 and not
enrolled in next 2 years and not
transferred to 4-yr and on EDD wage
file
20,000
10,000
Source: CA Dept of Finance Table D20
(Median H-hold), Table D6 (Per Capita);
EDD Base Wage File
0
CA Per Capita Income
CCC Median Income (no zeroes)
1993
22,635
17,408
1994
23,203
19,197
1995
24,161
21,004
1996
25,312
22,995
1997
26,490
25,696
1998
28,374
27,468
1999
29,828
29,109
2000
32,464
32,456
2001
32,877
42,891
2002
32,807
47,331
2003
33,389
48,718
Years
42
Mapping Districts
 CC
Districts in CA are legally defined,
have own elections, pass own bonds
 We did not have a district mapping for
all 72 districts

So we couldn’t do district participation
rates
43
Mapping Project
 Get
a cheap copy of ESRI Suite
 Collect all legal district boundary
documents
 Find cheap labor—no budget for this
44
45
Peer Grouping
 “Peers”



historically have been locally defined:
My neighbor college
Other colleges with similar demography
Other colleges with similar size
46
Peer Grouping
 Taking

peering to another level:
Peer on exogenous factors that predict the
accountability metric’s outcome
 Thus
leaving the “endogenous” activity as the
remaining variance

Cluster to create groups
 We
picked 6 clusters, with a min of 3 in a
cluster

Each metric produces different factors,
peers, clusters
47
Peer Grouping: Example
 Peering
the SPAR Rate:
109 rates as outcomes
 Find data for all 109 that might predict
outcomes/explain variance
 Perform regression and other magical SPSS
things
 See how high you can get your R2

48
Finding Data
 What
might affect a grad/transfer rate
on an institutional level?
Student academic preparedness levels
 Socioeconomic status of students
 First-gen status of students
 Distance to nearest transfer institution
 Student age/avg unit load

49
Finding Data
 We
had to create proxy indices for
much of these (142 tried)
GIS system: geocode student
zipcode/ZCTA
 Census: lots of data to be crossed by
zip/ZCTA
 Create college “service areas” based on
weighted zip/ZCTA values

 Different
than district legal boundaries
50
51
Finding Data
 The

Killer Predictor
“Bachelor Plus Index”, or what % of service
area population of college has a bachelor’s
degree or higher
 “Bachelor
Plus Index” a proxy for:
First gen
 Academic preparedness
 Socioeconomic status
 Distance to nearest transfer institution

52
Peering SPAR Rate
 Exogenous
Rate:
factors that predict SPAR
Bachelor Plus Index
 % older students
 % students in basic skills

 R2

= .67
What’s left is implied institutional variance
 Demo
53
Peering: What’s Bad
 Its
complex and somewhat confusing and
labor intensive.
 Colleges traditional notion of “peer” is
shaken
 Multiple peers for multiple metrics; can
change every year
 You could do well vs. State average,
increasing over time, but last in your peer
group
54
Peering: What’s Good
 Its
complex and somewhat confusing
 You will likely look good in some areas,
OK in others, and low in others
 Its not very likely anyone will be high
or low in all 6 metrics
 It eliminated rankings.
55
The ARCC Report
 Is
almost 800 pages.
 Comes out every March.
 Takes 4 PY’s to complete (about 6
months/yr)
 Is generally regarded highly in CA
academic and Legislative circles.
 DOF and LAO and Sec. of Ed love it.
 Local Trustees/Boards love it.
56
The ARCC Collaboration
 Has
brought the system more money:
$33 mil in basic skills
 Increased noncredit reimbursement rates
by $300/FTE

 Has
brought about trust between
system and State stakeholders.
 Has educated both sides tremendously.
57
No More “Girlie-Man” Accountability!
58