Audit Committee
May 16, 2007 Meeting
Student Success Measures Project
Update
The Minnesota State Colleges and Universities System is an Equal Opportunity employer and educator.
Objectives
•
How reliable are the student success measures
that are reported to the Board of Trustees?
•
How accurate is the underlying student record
data or other source data?
•
Are colleges and universities complying with
board policies related to student success, such
as admissions, transfer, and graduation?
Are the information technology support systems
operating as intended?
•
–
–
–
Slide ‹#›
ISRS
Degree Audit Reporting System (DARS)
Course Applicability System (CAS)
Background
• Student Success Measures
– Measures developed using a consultative
process by the Institutional Research
Directors.
– Definition of Success includes Students who
• Re-enroll
• Transfer
• Receive an academic award in an approved
MnSCU program
Slide ‹#›
Three Parts of the Project
• Part I: System-wide Student Success
Measures
– Preliminary results being presented today.
• Part II: Institutional Student Success
Measures
– Testing in progress
• Part III: Student-level Success
– Work to be completed in summer
Slide ‹#›
Part 1 – System-wide Student Success
Measures
• Planning completed
– Discussions with Research Unit, ITS
Division and DARS/CAS staff
• Tested system-wide statistical sample
• System-wide data analysis in progress
Slide ‹#›
Part 1 – System-wide Student Success
Measures
• Statistical sample
– 456 sample items
– 95% confidence level with 2% margin of
error
– 540,891 student records in population
(detail data that supports student
success measures)
Slide ‹#›
Part 1 – System-wide Student Success
Measures - Preliminary Conclusions
• Reliability Testing Exceptions being
Corrected by Research Unit
– Error in identifying some students with
degrees.
– Issue identified in how transfer students are
determined.
• Clearinghouse data based on FERPA
directory information.
• Correction in progress to use alternative
methods for identifying transfer students,
including using MnSCU ISRS data.
Slide ‹#›
Part 1 – System-wide Student Success
Measures - Preliminary Conclusions
Comparison of Success Rates on Full-time Undergraduate Students –
Adjusted for Exceptions
Status for Fall
1999 – 2005
Cohorts
Original Student
Success Measure
%
Exceptions added to
Success %
Graduated
27.27%
31.02%
Transferred
18.18%
19.25%
Retained
19.79%
19.25%
Overall
65.24%
69.52%
Note: The effect varies significantly on the transfer out exceptions
by college or university.
Slide ‹#›
Part 1 – System-wide Student Success
Measures - Preliminary Conclusions
• Dashboard Reliability - Limitations
– Admission status for transfer students not reliable
prior to Fall 2005.
• Solution = Combine undergraduate regular and
transfer
– Race/Ethnicity data incomplete in earlier cohorts.
• Solution = Be cautious about interpreting trends
based on counts. Trends in success should be
reasonable.
– Combining all admission statuses is difficult to
interpret.
• Solution = Default on full-time undergraduates as
indicator.
Slide ‹#›
Part 1 – System-wide Student Success
Measures – Assumption Questions
• Unresolved questions about Student
Success Measures Assumptions
– Should academic standing be a criteria for
defining success, particularly for transfer out
students?
– Should students who have not met
“success” criteria but have substantially
completed be considered successful?
Slide ‹#›
Part 1 – System-wide Student Success
Measures – Assumption Questions
• Possible definition of Good Academic
Standing:
– 2.0 cumulative Grade Point Average
– 66.67 percent cumulative completion rate
(earned credits/attempted credits)
Slide ‹#›
Part 1 – System-wide Student Success
Measures – Assumption Questions
Comparison of Success Rates on Full-time Undergraduate Students Adjusted for Academic standing for transfer out students
Slide ‹#›
Status for Fall
1999 – 2005
Cohorts
Original Student
Success
Measure %
Exceptions
added to
Success %
Poor Academic
Standing Deleted
from Success %
Graduated
27.27%
31.02%
31.02%
Transferred
18.18%
19.25%
8.02%
Retained
19.79%
19.25%
19.25%
Overall
65.24%
69.52%
58.29%
Part 1 – System-wide Student Success
Measures – Assumption Questions
• Possible Substantial Completion Criteria
for Students who have not met
“Success” criteria.
– Good Academic Standing
– For Undergraduate Students – must have
completed 60% of program (2 years – 36
earned credits and 4 years – 72 earned
credits)
– For Graduate Students - must have
completed 60% of program (20 earned
credits)
Slide ‹#›
Part 1 – System-wide Student Success
Measures – Assumption Questions
Comparison of Success Rates on Full-time Undergraduate Students –
Added Students with Substantial Completion
Original
Student
Success
Measure %
Exceptions
added to
Success %
Poor Academic
Standing
Deleted from
Success %
Substantial
Completion
Added to
Success %
Graduated
27.27%
31.02%
31.02%
31.02%
Transferred
18.18%
19.25%
8.02%
8.02%
Retained
19.79%
19.25%
19.25%
19.25%
Substantial Completion
-
-
-
4.28%
Overall
65.24%
69.52%
58.29%
62.57%
Status for Fall
1999 – 2005 Cohorts
Slide ‹#›
Part 1 – System-wide Student Success
Measures – Analysis of other Assumptions
• Other Assumptions being reviewed
– Only Fall Cohorts Tracked
•
•
•
•
•
75% of total entrants are Fall entrants
87% of university entrants are Fall entrants
96% of FT university entrants are Fall entrants
66% of college entrants are Fall entrants
82% of FT college entrants are Fall entrants
– Graduate Success
• Any degree awarded tracked as a success
– Certificates - 5% of graduates in sample
had less than 10 credits
• Only first degree tracked
Slide ‹#›
Preliminary Results – Other
Observations
• Excessive ISRS access to enter or
modify academic awards
– Over 1,200 employees have access to
enter academic awards in ISRS
• Data inconsistencies/complexities
– Consortium credits
– Academic Suspensions
Slide ‹#›
Next Steps
• Complete Testing at Colleges and
Universities on Student Success
Measures and DARS
• Complete Surveys at each College and
University
• Complete system-wide data analysis
• Analyze Results
• Final Report – September 2007
Slide ‹#›