Student Success Measures & Systems
Audit Committee – October 17, 2007
John Asmussen, CPA, CIA, CISA, MBA
Executive Director, Office of Internal Auditing
Beth Buse, CPA, CISA, CIA, GSEC
Deputy Director, Office of Internal Auditing
Marita Hickman, CPA, MBA
Regional Audit Coordinator
The Minnesota State Colleges and Universities System is an Equal Opportunity employer and educator.
Student Success Measures and Systems
• Part I: System-wide Student Success
Measures
– How reliable are the student success measures that
are reported to the Board of Trustees?
– How accurate is the underlying student record data or
other source data?
• Part II: Student-level Success
– Are the information technology support systems
implemented as intended?
» Degree Audit Reporting System (DARS)
» Course Applicability System (CAS)
Slide ‹#›
Part 1: Student Success Measures Background
• June 2003 – Board of Trustees approved
system-wide accountability framework
• September 2004 – initial version of
measures designed to assess student
success
– Retained
– Transfers
– Graduates
• Dashboard designed and published
Slide ‹#›
Example of Why MnSCU Student Success Measure is needed
Metropolitan State University (US DOE data)
Slide ‹#›
Example of Why MnSCU Student Success Measure is needed
Metropolitan State University (MnSCU v. US DOE data)
Metropolitan State Comparative Success Rates
Fall 2000 Cohort at 6 years
Source
% Students Included
Graduation Rate
Transfer-Out Rate
Retained Rate
Success Rate
Slide ‹#›
USDOE
3%
21%
14%
n/a
35%
MnSCU - MnSCU Full-time
All
22%
100%
54%
30%
15%
18%
4%
7%
73%
55%
Part 1: Student Success Measures –
System Dashboard example
Slide ‹#›
Part 1: Student Success Measures –
Overall Conclusions
• Data supporting the system-wide student success
measures is reliable, compiled consistently across
institutions and conforms to a uniform definition
– Preliminary Report presented in May 2007
– Certain Limitations
• More credibility of measures needs to be built
– Lack of awareness of measures
– Linking system-wide measures to institutional operations
– Colleges and universities
• Certain underlying measurement limitations and national
trends merit continued study to further enhance the validity
of measures.
Slide ‹#›
Student Success Reliability Test Results
• Underlying data is reliable, compiled
consistently and conforms to a uniform
definition.
• Two short-term limitations
– Race/Ethnicity of Students
– Transfer v. Regular Student Classifications
Slide ‹#›
% Students with Unknown
Race/Ethnicity
Data Limitation on Race/Ethnicity
35%
30%
25%
20%
15%
10%
5%
0%
2000
2001
2002
2003
Fall Term of Entering Cohort
Slide ‹#›
2004
2005
Ratio of Regular to Transfer
Undergraduates
Data Limitation on Transfers
5
4
3
2
1
0
2000
2001
2002
2003
2004
Fall Term of Entering Cohort
Slide ‹#›
2005
Short-term Considerations
• Disclose limitations
• Build awareness
• Work with institutions to facilitate use
• Base public information on “Official”
MnSCU measure
Slide ‹#›
Long-term Considerations
• Limitations
– Overall Institutional Success Rates
– Non-degree Seeking Students
– Students Earning Certificates
– Transfer-out Success Definition
– Excluding Spring Term Entrants
• Monitor national accountability
developments
Slide ‹#›
Success Rate
Overall Success Rates Affected by
Variability in Subcohorts
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18
Subcohort (See list in report)
Retained
Slide ‹#›
Transfer
Graduate
Overall Success Rate example
Slide ‹#›
Student Success Measure
Next Steps
• Dashboard update released in Nov 2007
• More training
• Explore solutions to legal constraints on
data sharing
• Approved ITS project for accountability
database
• Other continuous improvement efforts
Slide ‹#›
DARS and CAS - Introduction
• DARS – Degree Audit Reporting System
– Purchased software
– Automated Report for Students - ‘Degree
Audit’
• Academic Program requirements
• Individual courses taken, including transfer
• Tracks courses needed to complete
academic award
– Automated transfer process
• Course equivalencies
Slide ‹#›
DARS and CAS – Moorhead DARS
Brochure
Slide ‹#›
DARS and CAS – Introduction
• Why DARS?
– Complexity of Degree Requirements
• Examples
– Minnesota Transfer Curriculum – Inver Hills
– Accounting Degree - St. Cloud State
University
– Increasing Mobility of Students
– Number of Articulation Agreements
– We are in calendar year 2007
– Future Potential
Slide ‹#›
DARS and CAS – Introduction
• Benefits of DARS
– Accurate and consistent information for
advising
– Increased efficiency and consistency in
determining transfer course equivalencies
– Reduction in manual errors in transfer
evaluation and academic award verification
– Degree Verification time reductions
Slide ‹#›
DARS and CAS - Introduction
• CAS – Course Applicability System
– Purchased web-based software
– Multi-state participants
– Transfer evaluation tool for students and
advisors
– Allows students capability to ‘degree shop’
– Dependent on DARS implementation
• Academic program requirements
• Course equivalencies
– www.mncas.org
Slide ‹#›
DARS and CAS - Introduction
Slide ‹#›
DARS and CAS - Background
• Legislature Funded DARS in 1999
– FY 2000 – $1.2 million
– FY 2001 and thereafter - $500,000
• Office of the Chancellor allocated a total of
$1.2 million in FY01 and 03 to institutions
• DARS implementation expectation –
December 31, 2002
• CAS implementation expectation –
December 31, 2004
– Dependent on DARS implementation
• Several presentations to Board on systems
Slide ‹#›
System-wide Summary of DARS and CAS
Implementation – As of August 2007
Components of DARS/CAS Implementation
System-wide Profile (n=37)
Data Integrity
Undergraduate program encoding complete and up-to-date.
28
Transfers entered into DARS.
31
Exceptions entered into DARS.
24
Course equivalencies established for common transfer courses.
5
4
2
4
23
4
9
10
4
Functionality Available to Students
DARS reports available for monitoring academic progress toward degree.
CAS reports available to assess course transfer options.
26
17
7
11
4
9
Functionality Used by Institution
Used to evaluate transfer courses.
31
Used to determine whether graduation requirements are met.
26
Counselors and advisors use to work with students on academic progress.
28
Legend:
Slide ‹#›
- Substantially Implemented
– Partially Implemented
1
4
5
7
4
5
- Not Implemented
DARS and CAS - Results
• DARS and CAS Implementation
Roadblocks
– Lack of appropriate staff
– Complexity of DARS
– No champion or resistance to change
– Lack of faculty/advisor support
Slide ‹#›
DARS and CAS - Results
• Lack of Information Technology
Resources has resulted in delays on
DARS and CAS improvements
– DARS and CAS IT Maintenance
– Outstanding IT Projects
• Interactive Audits
• e-Transcript
• Course Equivalency Builder
• CAS Request Import Interface
Slide ‹#›
DARS and CAS Long-term Considerations
• University of Florida ‘Tracking’ system
– 7% improvement in graduation rate
• Potential Enhancements
–
–
–
–
–
–
–
Slide ‹#›
Course Sequencing
Graduate Planner
Targeted Advising
Course Seat Predictor
Interactive Audits (degree shopping)
Degree verification ISRS Edit
Common Course Numbering
DARS and CAS
Next Steps
• College and University
– Yellow and Red status – plan to be fully
implemented by December 31, 2007.
– If unable to make date, put a detailed plan together
for Linda Baer’s approval.
– Maintain DARS and CAS.
• Office of the Chancellor
– Assist and train colleges and universities on
implementing DARS and CAS
– Support IT infrastructure and approved projects
Slide ‹#›
Office of Internal Auditing…
… A catalyst for improvement
www.internalauditing.mnscu.edu
651-296-3471
Slide ‹#›