Reporting Program‐Level Student Success Measures AIR 2014 Alison Joseph & David Onder •
•
•
•
•
•
•
•
Problem ‐ We are asked for program‐level success data (retention and graduation) •
Complicated (particularly for undergrads) •
Different programs serve different purposes •
High stakes – program prioritization (AA driven) –
Reports lumped all non‐retained together (whether they graduated or stopped out) Background –
Historically reported on freshmen cohort, University‐level only –
Only one segment of our population (no graduate students, no transfers, no part‐time) What We Wanted –
Solid & simple approach (easy to explain and defend) –
Fair (Useful for all types of programs) –
Meaningful for decision‐making (high‐ and low‐level) –
Not overly complicated display –
Illuminates: overall performance, historic trends, when students are lost –
Something that can be generated yearly w/o too much effort 5 Outcomes –
Five outcomes for each student that declares a given major: Retained in program, Graduated in program, Retained in different program, Graduated in different program, Not retained (stop‐out/drop‐out) ‐ Exclusive and exhaustive General Approach (based on cohorts) –
A student is placed in a program cohort the 1st time they declare a given program –
Each student is flagged as one of the 5 possible outcomes for each ½ yr interval (each regular semester) –
At each interval we report where the members of the cohort fall –
Each student will only appear in one cohort for a program (usually) Why this works –
We can report data on any interval, if asked –
If a student stops‐out, then returns, they are picked back up –
Bridges the gap between retention reports and graduation reports Technical Approach –
Used SAS to generate data set; Excel for reporting (Expanded on techniques used for our Fact Book automation) –
There is one report built per level •
Drop down lists shows all the programs at that level •
Formulas reference program code, and populate report based on that code (sumifs, averageifs, etc.) Horizontal stacked bar 0%
20%
40%
60%
80%
2003‐2004
2005‐2006
2007‐2008
2009‐2010
Program Success
•
•
•
•
•
Non‐program Success
Not Retained
100%
What are people asking?
• How is the program performing? • How are students performing overall at institution? • How many are dropping out? We graph 5 Flags too –
Horizontally ‐ See people transitioning to completers and drop‐outs over time, follow specific cohorts –
Vertically ‐ Compare performance of different cohorts at the same intervals Why this report is REALLY awesome –
Use a special approach with a named range to find a list of all unique programs, and populate drop‐down box with this list (http://www.datawright.com.au/excel_resources/excel_dynamic_ranges.htm) –
VBA to cycle through, do calcs, print PDF out to a directory (by dept and college) and move on to next report Expanding the Idea –
CIP code (groups up old and new program codes), Dept (similar programs that students transfer between), College Ideas for Next Steps – True Success Rate (VSA, incorporating of Clearinghouse Data) –
Consider rolling averages or other approaches to smooth turbulent data on small groups –
Compare retention data against unit‐level goals Contact Information –
Alison Joseph, Applications Analyst (ajoseph@wcu.edu) –
David Onder, Director of Assessment (dmonder@wcu.edu) – Office of Institutional Planning and Effectiveness ‐ oipe.wcu.edu , (828) 227‐7239 20%
62%
59%
58%
55%
58%
60%
61%
59%
CIP Ret
20%
69%
75%
73%
54%
30%
50%
65%
53%
CIP Ret
40%
2%
2%
3%
4%
3%
4%
4%
4%
15%
14%
15%
15%
13%
8%
9%
14%
60%
@1 year 60%
8%
13%
13%
31%
22%
17%
9%
0%
CIP Grad WCU Ret
40%
0%
0%
0%
0%
9%
0%
0%
0%
CIP Grad WCU Ret
WCU Grad
1%
1%
1%
1%
1%
0%
0%
0%
WCU Grad
0%
0%
0%
0%
4%
0%
0%
0%
80%
Not Retained
20%
25%
23%
23%
25%
27%
25%
22%
80%
Not Retained
23%
13%
13%
15%
35%
33%
26%
47%
1
100%
100%
100%
100%
100%
100%
100%
100%
100%
Total
100%
100%
100%
100%
100%
100%
100%
100%
100%
Total
0%
39%
37%
37%
34%
37%
38%
37%
CIP Ret
0%
54%
38%
40%
38%
17%
33%
39%
CIP Ret
@2 years 20%
13%
11%
12%
16%
17%
20%
18%
60%
WCU Grad
0%
0%
0%
0%
4%
0%
0%
40%
18%
17%
18%
17%
10%
12%
13%
60%
WCU Grad
2%
1%
2%
4%
2%
1%
1%
@2 years 40%
23%
13%
40%
31%
9%
28%
30%
CIP Grad WCU Ret
20%
0%
13%
0%
8%
9%
11%
0%
CIP Grad WCU Ret
80%
Not Retained
28%
34%
31%
29%
34%
29%
31%
80%
Not Retained
23%
38%
20%
23%
61%
28%
30%
100%
99%
101%
100%
100%
100%
100%
100%
Total
100%
100%
100%
100%
100%
100%
100%
100%
Total
2
0%
4%
5%
6%
4%
5%
CIP Ret
0%
0%
0%
0%
15%
4%
CIP Ret
@4 years 20%
44%
38%
39%
42%
45%
40%
5%
8%
6%
5%
5%
60%
80%
WCU Not Grad Retained
14%
31%
12%
37%
12%
36%
9%
39%
9%
37%
80%
WCU Not Grad Retained
0%
23%
0%
50%
13%
47%
8%
38%
13%
61%
60%
@4 years 40%
31%
13%
7%
8%
9%
CIP Grad WCU Ret
20%
46%
38%
33%
31%
13%
CIP Grad WCU Ret
910000000000
Information Management
College of Information Studies
Bachelors
Undergraduate
910000000000
Information Management ‐ BS
* 12‐Digit CIP codes are used in the North Carolina state system. The first 6 digits are used for the discipline, the next three for the degree code, and the final three for the specialty code. Not Retained
WCU Graduation
WCU Retention
Program Graduation
0%
12.07
12.67
14.05
13.08
13.06
12.47
14.36
14.76
New Cohort
Program Retention
Legend
2005‐2006
2006‐2007
2007‐2008
2008‐2009
2009‐2010
2010‐2011
2011‐2012
2012‐2013
Year
University Average
Not Retained
WCU Graduation
WCU Retention
Program Graduation
0%
13
8
15
13
23
18
23
15
New Cohort
Program Retention
Legend
2005‐2006
2006‐2007
2007‐2008
2008‐2009
2009‐2010
2010‐2011
2011‐2012
2012‐2013
Year
New Cohort
0.5
@1 year Retention and Graduation ‐ by 12‐digit CIP Code
CIP Code: 910000000000
Career Level:
Degree Level:
Program CIP (12‐digit):
Program Department:
Program College:
Program CIP (12‐digit):
Program Name
4
100%
99%
101%
100%
100%
100%
Total
100%
100%
100%
100%
100%
100%
Total
0%
0%
1%
0%
CIP Ret
0%
0%
0%
0%
CIP Ret
@6 years 20%
48%
43%
45%
40%
0%
1%
1%
60%
80%
WCU Not Grad Retained
19%
32%
18%
38%
16%
38%
80%
WCU Not Grad Retained
15%
38%
13%
50%
20%
40%
60%
@6 years 40%
0%
0%
7%
CIP Grad WCU Ret
20%
46%
38%
33%
CIP Grad WCU Ret
6
100%
99%
101%
100%
Total
100%
100%
100%
100%
Total