Reporting Program-Level Retention
and Graduation
Alison Joseph & David Onder
SAIR 2012
•
•
•
•
9608 students
Master’s Comprehensive
Mountain location
Residential and Distance
The Problem
• We are asked for program-level data
– Lack of program-level retention and graduation rates
– Complicated (particularly for undergrads)
– Different programs serve different purposes
– High stakes – program prioritization (AA driven)
• Reports lumped all non-retained together (whether
they graduated or stopped out)
Background
• Historically reported on freshmen cohort
– University-level only
– Only one segment of our population
• No info on grad students
• No info on transfers
• No info on those who start part-time
What We Wanted
• Solid & simple approach (easy to explain and defend)
• Fair
– Useful for all types of programs
• Meaningful for decision-making (high- and low-level)
• Not overly complicated display
• Illuminates
– Overall performance
– Historic trends
– When are students lost
• Something that can be generated yearly w/o too much
effort
5 Outcomes
• Five possible outcomes for each student that declares
a given major
– Retained in program
– Graduated in program
– Retained in different program
– Graduated in different program
– Not retained (stop-out/drop-out)
• Exclusive and exhaustive
General Approach
• Based on cohorts:
– A student is placed in a program cohort the 1st time they
declare a given program
– Each student in the cohort is flagged as one of the 5 possible
outcomes for each ½ year interval (each regular semester)
– At each interval we report where the members of the cohort
fall
– Each student will only appear in one cohort for a program
(usually)
Why this works
• Students are not double-counted (usually)
• We can report data on any interval, if asked
• If a student stops-out, then returns, they are picked
back up
• Bridges the gap between retention reports and
graduation reports
Technical Approach
• Used SAS to generate data set
• Data put in Excel sheet
• There is one report built per level
– Drop down lists shows all the programs at that level
– Formulas reference program code, and populate report based
on that code (sumifs, countifs, averageifs, etc.)
– Expanded on techniques used for our Fact Book automation
Multiple Iterations
Started with term-level data
New Cohort
@1 year
0.5
Year
1996
1997
Semester
Fall
Spring
1997
1998
Fall
Spring
New Cohort
Prg Ret
28
3
54%
67%
14
4
50%
75%
15
YearFall Semester
Spring
5
1996
Fall
1999
Fall
4
2000
1997Spring Spring 4
1998
1999
2000
2001
Fall
Spring
1997Fall Fall
2001
2002
1998Spring Spring
2002
2003
Fall
Spring
3
5
Prg Grad
New
18% Cohort
4%
WCU Ret WCU Grad
25%
33%
46%
33%
@10%year 0%
7%
25%
43%
50%
47%
40%
40%
0%
14%
0%
0%
0%
0%
0%
36%
25%
100%
100%
80%
60%
7%
0%
0%
20%
0%
0%
13%
20%
75%
75%
0%
0%
0%
0%
25%
25%
67%
40%
0%
0%
0%
0%
33%
60%
100%
100%
50%
100%
100%
75%
New Cohort
28
0%
0% 3
0%
0%
9
5
56%
60%
22%
0%
14
11%
0%
4
10
6
60%
67%
10%
0%
10%
0%
73%
75%
9%
0%
2004
2005
12
3
67%
67%
0%
0%
17%
0%
17
8
71%
50%
6%
13%
0%
0%
1999Fall Fall
Spring
2000
Spring
Prg Grad
100%
100%
11
4
2005
2006
Prg Ret
25%
33%
1998Fall Fall
2003
2004
1999Spring Spring
Fall
Spring
Total
0%
0%
0.5
15
0%
0%
5
4
4
Prg Ret
100%
100%
54%
100%
100%
67%
Prg Grad
18%
25%
25%
25% 0%25%
0%
20%
0%
20%
14%56%
22%
60%
0% 0%
0%
0%
11%
40%
0%
0%
20%
33%
100%
100%
20%
33%
40%
33%
0%
0%
18%
25%
80%
100%
100%
60%
36%
75%
7%45%
0% 0%
0%
0%
17%
33%
100%
100%
25%
67%
0%
13%
24%
25%
75%
100%
100%
75%
0%
0%
0%
Not
Retained
Total
Prg Ret
Prg Grad
29%
33%
100%
100%
21%
33%
61%
33%
0%
0%
0%
0%
50%
25%
100%
100%
14%
0%
50%
0%
20%
13%
40%
0%
40%
0%
WCU Ret WCU Grad
0%
0%
0%
0%
@3 years
2
Not
Retained
0%
0%
@2 years
1
WCU Ret WCU Grad
4%
0%
0%0%
50%
50%
100%
100%
0%
100%
0%100%
0%
0%
0%
0%
100%
60%
100%
100%
11%
0%
0%0%
0%0%
11%
40%
0%100%
0%100%
0%
0%
10%
0%
Not 75%
Retained
13%
53%
25%
0%
50%
25%
33% 25%
33%
20%
0%
20%
Total
Prg Ret
Prg Grad
0%
0%
18%
33%
100%
100%
14%
0%
64%
33%
0%
0%
0%
0%
36%
25%
0%
0%
50%
75%
0%
0%
0%
20%
33%
40%
1
Total
100%
0%
0%
0%
0%
100%
0%
0%
0%
0%
36% 56%
22%
0%
25% 0%
100%
0%
11%
0%
0%
100%
10%
17%
10%
0%
30%
33%
100%
100%
0%0%
0%
0%
20%0%
18%
25%
0%100%
0%100%
42%
0%
0%
0%
8%
0%
25%
33%
100%
100%
0%
33%
35%
25%
0%41%
13%
0%
0%
25%
0%0%
13%
0%
24%
25%
0%100%
100%
0%
12%
13%
50%
50%
13% 73%
9%
25%
20% 25%
@4 years
3
Not
Retained
WCU Ret WCU Grad
10%
0%
50%
50%
67%
60%
11%
100%
100%
100%
Prg Ret
100%
100%
25%
100%
100%
33%
100%
100%
7%
100%
100%
25%
Prg Grad
13%
0%
0%
0%
0%
0%
0%
0%
20%
33%
100%
100%
18%
50%
47%
100%
100%
40%
8%
0%
42%
33%
100%
100%
100% 0%
25%
100%
29%
25%
100%
100%
6%
0%
100%
0%
0%
0%
0%
100%
50%
33%
0%
0%
25% 53%
25%
25%
6%
13%
25%
25%
10%
0%
0%
25%
0%
0%
60%
40%
46%
50%
50%
33%
33%
40%
43%
78%
0%
50%
50%
67%
40%
73%
25%
0%
50%
67%
25%
59%
38%
25%
0%
Total
Prg Ret
Prg Grad
21%
67%
100%
100%
7%
0%
79%
33%
0%
0%
0%
0%
50%
25%
100%
100%
0%
0%
57%
0%
0%
0%
20%
27%
40%
100%
100%
0%
@2
0% years
0%
0%
0%
WCU Ret WCU Grad
0%
0%
0%
0%
@6 years
4
Not
Retained
WCU Ret WCU Grad
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
11%
0%
0%
50%
50%
67%
60%
11%
100%
0%
100%
100%
0%
100%
100%
75%
Not
Retained
7%
67%
0%
0%
0%
0%
0%
0%
100%
100%
0%
0%
20%
0%
17%
0%
0%
10%
0%
30%
33%
100%
100%
9%
0%
0%
0%
0%
0%
18%
50%
0%
100%
100%
20%
0%
0%
0%
0%
8%
0%
42%
33%
100%
100%
0%
0%
0%
0%
0%
6%
38%
0%
29%
25%
100%
100%
0%
0%
0%
0%
40%
29%
50%
50%
33%
33%
40%
50%
78%
0%
25%
60%
67%
13%
73%
25%
40%
50%
67%
50%
65%
38%
50%
6
Not
Retained
Total
0%
0%
14%
67%
100%
100%
0%
0%
43%
25%
100%
100%
0%
20%
27%
40%
100%
100%
50%
50%
100%
100%
67%
60%
100%
100%
11%
80%
100%
100%
30%
17%
100%
100%
18%
75%
100%
100%
42%
33%
100%
100%
29%
25%
100%
100%
WCU Ret WCU Grad
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
2
Total
100%
0%
0%
100%
0%
0%
100%
11%
0%
100%
10%
0%
100%
9%
0%
100%
8%
0%
100%
6%
38%
100%
2006
2007
Fall
Spring
7
9
57%
33%
0%
22%
0%
22%
0%
0%
43%
22%
100%
100%
0%
0%
29%
56%
29%
22%
0%
0%
43%
22%
100%
100%
0%
0%
29%
56%
14%
0%
14%
11%
43%
33%
100%
100%
0%
0%
29%
56%
0%
0%
14%
11%
57%
33%
100%
100%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
2007
2008
Fall
Spring
13
3
92%
33%
0%
0%
0%
0%
0%
0%
8%
67%
100%
100%
38%
0%
46%
33%
0%
0%
0%
0%
15%
67%
100%
100%
15%
0%
69%
33%
0%
33%
0%
0%
15%
33%
100%
100%
8%
0%
77%
33%
0%
0%
0%
0%
15%
67%
100%
100%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
2008
2009
Fall
Spring
18
7
67%
57%
6%
0%
6%
0%
0%
0%
22%
43%
100%
100%
22%
57%
50%
14%
6%
0%
0%
0%
22%
29%
100%
100%
17%
29%
61%
43%
6%
0%
0%
0%
17%
29%
100%
100%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
2009
2010
Fall
Spring
15
3
80%
67%
0%
0%
7%
0%
0%
0%
13%
33%
100%
100%
27%
67%
53%
0%
0%
0%
7%
0%
13%
33%
100%
100%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
2010
2011
Fall
Spring
17
6
76%
50%
0%
0%
0%
0%
0%
0%
24%
50%
100%
100%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
2011
Fall
9
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
Academic Year
Next combined data into academic years
New Cohort
0.5
Year
2003-2004
2004-2005
2005-2006
2006-2007
2007-2008
2008-2009
2009-2010
2010-2011
New Cohort
15
15
25
16
16
25
18
23
@1 year
Prg Ret Prg Grad WCU Ret WCU Grad
73%
67%
64%
44%
81%
64%
78%
70%
7%
0%
8%
13%
0%
4%
0%
0%
0%
13%
0%
13%
0%
4%
6%
0%
@2 years
1
0%
0%
4%
0%
0%
0%
0%
0%
Not
Retained
Total
20%
20%
24%
31%
19%
28%
17%
30%
100%
100%
100%
100%
100%
100%
100%
100%
2
Prg Ret Prg Grad WCU Ret WCU Grad
47%
33%
32%
0%
31%
32%
33%
33%
33%
32%
44%
44%
40%
44%
0%
0%
8%
25%
0%
4%
0%
0%
7%
4%
0%
0%
0%
6%
Not
Retained
Total
20%
27%
24%
31%
25%
24%
17%
100%
100%
100%
100%
100%
100%
100%
Summary
All students lumped into 3 groups
@1 year
Year
2003-2004
2004-2005
2005-2006
2006-2007
2007-2008
2008-2009
2009-2010
2010-2011
New Cohort
15
15
25
16
16
25
18
23
@2 years
Program Success
Non-program Success
Not Retained
Program Success
Non-program Success
Not Retained
80%
67%
72%
56%
81%
68%
78%
70%
0%
13%
4%
13%
0%
4%
6%
0%
20%
20%
24%
31%
19%
28%
17%
30%
80%
67%
64%
44%
75%
72%
78%
0%
7%
12%
25%
0%
4%
6%
20%
27%
24%
31%
25%
24%
17%
This is the most summarized data we can (read: are willing to) provide.
The bottom line = Bold 3-group number
But how does that compare?
This is average program data to use as a comparison
@1 year
Year
2003-2004
2004-2005
2005-2006
2006-2007
2007-2008
2008-2009
2009-2010
2010-2011
New Cohort
10.74
12.52
12.33
12.73
12.53
12.92
13.53
13.70
@2 years
Program Success
Non-program Success
Not Retained
Program Success
Non-program Success
Not Retained
72%
65%
51%
62%
73%
72%
74%
75%
4%
8%
20%
13%
3%
2%
1%
1%
24%
27%
29%
26%
24%
26%
25%
24%
65%
58%
46%
58%
68%
68%
63%
7%
11%
20%
12%
2%
3%
4%
28%
32%
34%
30%
30%
29%
34%
Visual Representation
50
45
40
35
30
25
20
15
10
5
0
Prg R
WCU R
NR
Pgm G
WCU G
NR
WCU G
Pgm G
WCU R
Pgm R
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
5.5
6
6.5
7
7.5
8
8.5
9
9.5
10
0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
50
45
40
35
30
25
20
15
10
5
0
Pgm R
Pgm G
All R
All G
0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5
Horizontal stacked bar
• Normally we would never do this, but …..
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2003-2004
2004-2005
2005-2006
2006-2007
2007-2008
2008-2009
2009-2010
2010-2011
Program Success
Non-program Success
Not Retained
What are people asking:
- How is the program performing?
- How are students performing overall at institution?
- How many are dropping out?
We graph 5 Flags too
@ 1 year
0%
50%
100% 0%
@ 2 year
@ 3 year
@ 4 year
@ 6 year
50%
50%
50%
50%
100%0%
100% 0%
100% 0%
100%
2003-2004
2004-2005
2005-2006
2006-2007
2007-2008
2008-2009
2009-2010
2010-2011
This works because:
- You can see people transitioning to completers and drop-outs over time
- You can follow specific cohorts
Why this report is REALLY awesome
• Use a special approach with a named range to find a
list of all unique programs, and populate drop-down
box with this list
(http://www.datawright.com.au/excel_resources/excel_
dynamic_ranges.htm)
• VBA to cycle through, do calculations, print PDF out
to a directory (by department and college) and move
on to next report
Ideas for Next Steps
• Combining programs
– Discontinued programs and their replacements
– Distance and Residential programs
• Department- and College-level retention and
graduation
• True Success Rate (VSA, incorporating of
Clearinghouse Data)
• Consider rolling averages or other approaches to
smooth turbulent data on small groups
Contact Information
David Onder, Director of Assessment
dmonder@wcu.edu
Alison Joseph, Applications Analyst
ajoseph@wcu.edu
Office of Institutional Planning and Effectiveness
oipe.wcu.edu, (828) 227-7239