“Improving the Success of Students at Risk of
Dropping Out: Changing Practices and Policies”
Dr. Andrew Parkin
Associate Executive Director
Canada Millennium Scholarship Foundation
Canada
18th Annual 2009 European Access Network Annual Conference
Changing the Culture of Campus
Towards an Inclusive Higher Education
June 22-24, 2009
York, UK
Outline
1.
Introduction to the issue of persistence
2.
Who leaves PSE and why?
3.
Strategies to improve persistence
a)
b)
4.
Foundations for Success
LE,NONET
Reflections on experimental research
Introduction to the Issue of Persistence
Introduction to the Issue of Persistence
• Persistence:
– the ability of students to continue their post-secondary studies
from one year to the next and ultimately complete their programs
• Access  Success
– Meaningful access requires that students brought into to PSE
must be successful in their studies
• Low levels of persistence pose a problem...
– For students: left without the credential they need
– For institutions: poor use of resources and poor performance
– For societies:
• lower educational attainment
• exacerbates social cleavages
An Increasingly Important Concern
As a recent OECD report puts it:
• “the growing portion of disadvantaged students enrolled
in tertiary education makes the ongoing issue of their
retention and programme completion an increasingly
important concern in tertiary education.”*
* Santiago, Paulo, Karine Tremblay, Ester Basri and Elena Arnal. 2008.
Tertiary Education for the Knowledge Society, Volume 2. Paris: OECD.
Persistence: How Much Do We Know?
• “We know very little about how many students drop out
of programs, or why” (Rae, 2005)
• Little research on the issue of persistence in PSE in
Canada until recently
– Previous data on persistence rates limited to institution-specific
studies
– Little program evaluation or evidence of what works
– New research now becoming available:
• Canada Millennium Foundation pilot projects and program
evaluations
The Need for Program Evaluation
“Presently…there is little evidence about the effects of institutional
support programmes on student outcomes.”
Paulo Santiago, Karine Tremblay, Ester Basri and Elena Arnal. 2008. Tertiary
Education for the Knowledge Society, Volume 2. Paris: OECD.
“It is ironic that institutions that spend so much time and money
insisting on evidence-based decisions, spend so little time on
research that evaluates higher education itself. Research on the
postsecondary sector is inadequate and poorly publicized.
This should change.”
Bob Rae, Ontario: A Leader in Learning. Report and Recommendations. Toronto:
Government of Ontario, Ministry of Training, Colleges and Universities (2005)
The Challenges of Program Evaluation
• Student support programs are notoriously
difficult to evaluate because:
– Programs are designed and implemented without the
requirements for a viable evaluation in mind
– Data that would facilitate evaluation are not available
– It is difficult to isolate the effects of the program from
the possible effects of external factors
– There is no counterfactual (no reasonable “control”
group against which to compare the outcomes of
program recipients)
Who Leaves PSE and Why?
Persistence Rates in Canada
Overall Persistence Rates in Post-Secondary Education in Canada
Graduated
Still in postsecondary education
Discontinued postsecondary education
(%)
(%)
(%)
Year 1
12.0
75.2
12.9
Year 2
36.9
45.8
17.3
Year3
57.0
25.1
17.9
Year 4
66.2
14.8
19.0
Year 5
73.1
8.8
18.0
Year 1
1.1
91.0
7.9
Year 2
3.6
86.7
9.6
Year3
11.2
78.8
9.9
Year 4
45.0
45.2
9.8
Year 5
69.4
20.4
10.2
College
University
Source: Finnie and Qiu, Table 6b.
Barriers to Access and Persistence
35%
33%
Never attended
31%
29%
30%
25%
Discontinued
27%
22%
19%
20%
14%
15%
13%
12%
9%
10%
8%
11%
10%
5%
%
Finances
Career
indecision
Malatest (2007), Class of 2003
Lack of
interest
Program not Employment
what expected
Academic Personal/family
challenges
Who Leaves PSE and Why?
Self-Reported Reasons for Discontinuing Post-Secondary Studies
Class of 2003 Study
YITS
Lack of interest/
lack of program fit/
lack of career direction
52%
32%
Financial reasons
(other than desire to work)
23%
11%
Academic reasons
14%
6%
Desire to work
11%
7%
Who Leaves PSE and Why?
Factors correlated with low persistence:
•
•
•
•
•
•
•
•
Poor academic performance (both secondary and PSE levels)
Low engagement
Inadequate financial aid package or high levels of debt
Uncertainty about career goals
Lower levels of parental education (in some studies)
Aboriginal ancestry
Gender (men are more likely to drop out than women)
Age & family status (older students and students with dependent
children are more likely to drop out)
Percent of PSE Students Aged 24-26 Who Had Discontinued Their
Original Stream of PSE *, By Grade Average in High School
40
38
35
College
Stream
34
30
30
27
University
Stream
25
22
20
18
15
14
11
10
7
5
* Note: while some
of these students
will have
discontinued their
studies, others will
have switched
streams.
0
59% and less
60% -69%
70% -79%
80% -89%
90% -100%
Source: Shaienks, Danielle, Tomasz Gluszynski and Justin Bayard. 2008. Postsecondary Education: Participation and Dropping
Out: Differences Across University, College and Other Types of Postsecondary Institutions. Ottawa: Statistics Canada.
Percent of Post-Secondary Students who Have
Discontinued Their Studies (By Age Group)
25
23
20
21
15
20
16
15
14
Non-Aboriginal
Students
14
Aboriginal
Students*
10
9
5
0
18-20
20-22
22-24
Source: YITS (Cohort B) – special calculation.
* Excludes First Nations youth living on reserve
24-26
Who Leaves PSE and Why?
• Those who persist are more likely to attempt more than one program
than are those who drop out
• A key difference between those who persist and those who leave is
the ability to make adjustments along the way
• Importance of “resilience” (the capacity of overcome obstacles,
adapt to change, or to survive and thrive despite adversity)
– Those who persist are able to make adjustments that help them to stay
enrolled
– Factors contributing to resilience in youth include supportive
relationships with adults and parental expectations
Strategies to Improve Persistence
Foundations for Success
LE,NONET
Research on Policy Responses
• Persistence should be addressed through a
comprehensive approach
– Need to address interconnected barriers to success
– Need to work at institutional level rather than isolated policies
implemented by various departments
• There are many programs designed to improve
persistence, but few of these are evaluated for impact
• Student support programs can be implemented in the
context of a research endeavour that allows the impact
of the program to be assessed effectively
– Confirms that programs deliver the intended benefits to students
– Confirms that resources are well-spent
Overcoming Barriers to Access and Success
(Millennium Pilot Projects)
Career development
activities
Financial
incentives
Community
Support
Academic
preparation
Improving access to PSE
Academic support
Persistence
Mentoring & cultural
support
Career development
Financial support
GRADUATION
Off-campus
community support
Tackling the Drop-Out Rates in
Community Colleges through Financial
Incentives and Case Management
Foundations for Success
Research questions:
–
Do case manager-mediated support services
increase the probability of completing a college
program?
–
Do financial incentives in combination with case
manager-mediated support services increase the
probability of completing a college program?
Project Participation: At-Risk Factors
1) Remedial English/Communications:
• Measure: English Placement assessment results (slight variation within
participating colleges)
• Intervention: students are either streamed to regular Communications
or developmental course and are encouraged to undertake tutoring and
related academic support
2) Mentoring:
• Measure: Fast-Track survey question on whether the student selfidentified as someone who would benefit from a mentor.
• Intervention: Assignment of a mentor.
3) Career indecision:
• Measure: Four Fast-Track career clarity questions, with a cumulative
score greater than 10 (i.e. neutral response to strongly agree/disagree).
• Intervention: two “Career Gear” workshops; Myers-Briggs; StrongCampbell; one group debriefing; one one-on-one with counsellor (over 2
semesters)
Innovative Features of the
Foundations for Success Model
• Post-admissions testing where responses result in
actual redirection to existing services
• Case management approach to advise identified at-risk
students (one-on-one advisement)
• “Case managers” follow students’ progress for twoyears. This involves encouragement, identification of
students’ needs and challenges, and redirection to
appropriate services.
Random Assignment
New students in 2-year programs are invited to complete the Fast Track
survey, language assessment and sign informed consent, making them
eligible for the study
Students deemed at risk on
at least one item
Students not deemed at risk
(no further involvement in project)
Random Assignment occurs
Program Group 1
SERVICES
Academic, mentorship
and career exploration
support
Program Group 2
SERVICES PLUS
Academic, mentorship
and career exploration
support and financial
incentives
Program Group 3
COMPARISON
Number of Participants per College
Seneca
College
Mohawk
College
Confederation
College
TOTAL
Services
404
393
242
1039
Plus
406
393
242
1041
Compensation
387
434
242
1062
PROJECT TOTAL
1197
1219
726
3142
Year 1 Results
Participation in FFS Activities
• Service group significantly more likely to participate in
any FFS activities compared to Control group
• Service Plus group significantly more likely than Service
group to participate in any FFS activities
Source: 2008 College Administration Data, n=2,008.
Participation – Sub-group Analysis
82%
73%
37%
36%
High School Grades of 65
or less
38%
59%
Skipped High School
Class Once a Month +
Semester 2
43%
Not Confident in
Succeeding
40%
54% 59%
55% 58%
Grade 12 Education or
Less
Semester 1
45%
70%
Income Under $25,000
ESL
43%
58%
First Generation
51%
Women
61%
64%
63%
Men
63%
29+ Age
70%
Overall
Percent of respondents
• Higher Semester 1 participation rates among ESL students, first
generation, older, low-income and low-confidence students.
• Higher Semester 2 participation rates among ESL students, older
students, women, low income and low-confidence students.
Semester 1 GPA
Percent of respondents
• Service Plus group GPA was higher (2.13) compared to the Control
group (2.01)
2.01
2.06
2.13
Control group
Service Group
Service Plus Group*
Source: 2008 College Administration Data, n=2,008, missing GPAs were imputed using a regression model.
Semester 2 GPA
Percent of respondents
• Service Plus group had a higher GPA in Semester 2 (1.96) compared
to the Control group (1.79).
1.79
1.82
1.96
Control group
Service Group
Service Plus Group**
Source: 2008 College Administration Data, n=2,008, missing GPAs were imputed using a regression model.
Student Retention – Beginning of Year 2
•
Percent of respondents
Service Plus group significantly more likely to still be in program
(67.2%) compared to Control group (62.6%).
• Adjusting for the students that did not participate in any FFS activities,
the adjusted effect of the Service Plus treatment is a 6.4% increase in
retention one year into students’ college program.
62.6%
65.8%
Control group
Service Group
Source: 2008 College Administration Data, n=1,711.
67.2%
Service Plus Group*
Sub-Group Analysis – Who Gets the Most Benefit
from the Interventions?
Percent of students
• Retention: Service Plus ESL students, low-income students, and
students with high school grades of 65 or lower show significantly
higher retention rates than similar students in the Control group.
74%
73%
60%
ESL students
59%
55%
Family Income Less
than $25,000
Control group
44%
High School Grades of
65 or Lower
Service Plus Group
Source: 2008 College Administration Data, n=1,711, Cohort 1 only.
“Success after enduring many hardships” (Salish)
Improving the persistence rate of
Aboriginal learners at the University of Victoria
LE,NONET
• Pilot project to test the effectiveness of initiatives to improve the
retention of Aboriginal students at the University of Victoria
• Project recognizes that improved outcomes necessitates change not
only in students but also in the university
– “the onus for adjustment” should not be placed solely on Aboriginal
students
• LE,NONET project involves the creation and implementation of a
series of programs and support structures:
–
–
–
–
–
Student Mentoring
Bursaries (regular and emergency)
Community Internship
Research Apprenticeship
Staff and Faculty Aboriginal Cultural Training (SFACT)
LE,NONET
Research question:
• will a series of interventions involving financial,
academic, peer and cultural support have a
demonstrable effect on Aboriginal students’ likelihood of
completing their program of studies?
• is any particular type of intervention more effective and
do the interventions have greater impact when delivered
in combination with one another?
The Research Framework
•
Research will document the process through which changes are
proposed, developed, implemented, and modified at the university,
faculty, department and program level over the course of the project
•
Since one of the objectives of the program is to change the university
culture, it is possible for the program to affect non-participants as well
as participants (cannot use random assignment)
– evaluate the direct effects of participation in the various programs offered by
the LE,NONET project
– evaluate the indirect effects that such programs might have on students
who decline to participate
•
Comparison of the persistence of Aboriginal students during the periods
before and after the program was introduced
•
The LE,NONET approach will allow students to determine the depth of
their involvement in the programs — making differing ‘levels of
exposure’ to the interventions a key variable of interest.
Participation
• Number of participants (2006 & 2007): 139
–
–
–
–
–
–
Bursaries = 87
Mentor = 17
Mentored = 35
Preparation seminar = 44
Community intership = 24
Research apprenticeship = 17
• Historical cohort: 997
• Non-participants: 728
Interim Findings
• The majority of participants (77%) reported that their
participation in LE,NONET program components
contributed to the development of their sense of self
as an Aboriginal person.
• A strong majority (87%) agreed that their participation
in the programs contributed to their sense of
connection to the on-campus Aboriginal community.
• Just over half (54%) said that their involvement with
LE,NONET also contributed to their sense of
belonging at the university.
Interim Findings: Return to School
• Overall, about half of the research participants said that
LE,NONET had contributed to their decision to return to
school the following year.
• For some students, the financial support they received
through the Project made it financially viable for them to
afford their tuition and living expenses.
• Other respondents said that returning to school was
made easier by the emotional support and sense of
community provided by LE,NONET.
Registration Status (Interim): LE,NONET
70%
63%
60%
Historical Cohort
48%
Participants
Non-Participants
21%
10%
10%
23%
20%
27%
23%
30%
31%
40%
48%
50%
0%
Continuing
Graduated
Withdrawn
Reflections on Experimental Research
Reflections on Experimental Research
1.
These projects show that student support programs can be
implemented in the context of experimental research initiatives
that will lead to an increase stock of evidence about what may or
may not work.
2.
Experimental research is possible but not always easy to put in
place. Obstacles include:
a)
b)
c)
d)
Money
Time
Confusion about the objectives of the research
Tensions between educators and researchers
•
•
e)
3.
Sticking to the script
What is the measure of success?
Recruitment of participants
All of these obstacles can be overcome, but only when
researchers develop effective partnerships with educators and
administrators.
Andrew Parkin
aparkin@bm-ms.org
Canada Millennium Scholarship Foundation
1000 Sherbrooke West, Suite 800
Montreal, QC H3A 3R2
1-877-786-3999
All our publications are available online:
www.millenniumscholarships.ca