Finding the ‘Right’ Students:
The Search for Predictive
Validity in Applicant Screening
for the Health Sciences
Rosemary Lysaght, Ph.D.
Catherine Donnelly, M.Sc.
Michelle Villeneuve, M.Sc.
School of Rehabilitation Therapy
Background
• Impetus for examination of our admissions
process was the creation of the new masters
entry-level program in 2004
Goals for admission included:
• An evidence-based admissions process
• Increase the applicant pool
• Efficiency in admission process
Overall Aim of Selection
To select students who will:
• Succeed in the academic program
• Perform credibly in professional practice
• Possess the traits of character and ethical
values desired of a professional person
(Nayer, 1992)
Commonly Used Selection
Criteria in Health Sciences
• Pre-admission academic grades
• Discipline-specific aptitude tests (MCAT,
HOAE)
• Interviews
• Written submissions
• Letters of reference
• Prerequisites
(Auriemma, 2002; Salvatori, 2001)
Predictors of academic
success
• Evidence across disciplines supports the
predictive validity of
– Pre-admission academic grades
• No clear support found for
–
–
–
–
–
Discipline-specific aptitude tests
Interviews
Written submissions
Letters of reference
Prerequisites
(Bridle, 1987; Caplan et al, 1996; Howard & Jerosh-Herold, 2000; Lewis & Smith, 2002;
Kirchner et al., 2000; Kirchner & Holm, 1997; Salvatori, 2001)
Predictors of clinical
performance
• Unclear relationship between pre-admission
performance and clinical performance
• Valid outcome measures difficult to identify
• Lack of consistent raters
• Variability across settings
(Howard & Jerosch-Herold, 2000; Kirchner & Holm, 1997; Katz & Mosey,
1980; Tan et al., 2004)
Research Questions
1.
What admissions screening tools best predict
academic performance in a masters level OT
program?
 Selected academic courses and
 Clinical performance
2.
Does undergraduate coursework predict
success in topically-related coursework?
Method
• Analysis of existing data for 128 students
admitted to the OT masters program
– Multiple regression
– Models created for each research question
• Project received approval by the Queen’s
University’s Research Ethics Board
• Sample included 3 cohorts of students (1st three
years of the new professional masters program)
Factors Considered in OT
Admissions
– GPA
– Academic transcript
– Letter of intent
– 2 referee rating forms/letters
– Supporting data (foreign/non-traditional
applicants)
Predictor Variables
• Undergraduate Grade Point Average (GPA) –
from ORPAS
• Letter of Intent (LOI)
• Average rating, 2 raters
• Referee Rating
• 5 point scale, 12 items
• Average total score of 2 external referees
Predictor Variables (con’t)
• Additional ratings were created for each student
based on file review (1 = weak to 5 = exceptional)
Physical Sciences Experience/Preparation
• e.g. Anatomy, physiology, kinesiology (r for 2 raters = .98)
Social Sciences Experience/Preparation
• e.g. Psychology, Sociology, Family Studies (r=.91)
Experience with People with Disabilities/Vulnerable
Populations
• Rated for number, duration, and relevance (r=.79)
• Information drawn from Letter of Intent, referee letters and
experience questionnaire
Criterion Variables
• Program GPA (1st year)
• Communication Skills Practicum
• Targeted course grades in:
 Physical Determinants of Occupation
 Cognitive-Neurological Determinants
 Psycho-Emotional Determinants
Descriptive Statistics
- Predictor Variables
Variable
N
Mean
SD
Undergraduate GPA
Letter of Intent Rating
Referee Rating
129
129
129
3.24
71.70
4.72
.278
12.86
.25
Bkg. Physical Sciences
Bkg. Social Sciences
Bkg. Experience with PWD
127
127
127
3.92
2.90
3.70
1.35
1.66
.88
Descriptive Statistics
- Criterion Variables
Variable
N
Mean
SD
First Year GPA
128
80.29
3.00
Communication Skills Rating
Physical Determinants Grade
112
129
8.44
77.72
1.19
4.05
Cogntive-Neuro Det. Grade
Psycho-Emotional Det. Grade
126
126
78.97
78.89
5.13
5.03
Results – First Year GPA
• Predictor Variables: GPA, Letter of
Intent, Referee Rating
• Model is significant (R2 = .101; F= 4.65, p <
.001)
• GPA is only variable with significant beta
score (p = .005)
• Significant correlations between all three
variables and program GPA
Results – Communication Skills
•
Predictor Variables: GPA, Letter of
Intent, Referee Rating, all experience
ratings,
•
•
•
Model is not significant (R2 = .021; F= .62)
Referee rating is significantly correlated
with performance rating
No background experience ratings were
correlated with performance rating
Results – Physical Determinants
•
Predictor Variables: GPA, all
experience ratings
Result:
•
•
Model not significant (R2 = .009; F=.275)
No significant correlations between any
background experience rating and outcome
Results – Cognitive-Neuro
•
Predictor Variables: GPA, all
experience ratings
Result:
•
•
•
Model is significant (R2 = .14; F=5.0, p = .004)
GPA only variable with significant beta score (p
= .00)
Background in physical sciences was
significantly negatively correlated with outcome
Results – PsychoEmotional Det.
•
Predictor Variables: GPA, all
experience ratings
Result:
•
•
•
Model is significant (R2 = .082; F = 2.6, p< .05)
Social science bkgd only variable with
significant beta score (-.2 )
GPA is significantly positively correlated with
outcome, while SS bkgd is significantly
negatively correlated.
Additional Observations
• The contribution of the undergraduate GPA to all
models was reduced by the addition of the 3rd cohort,
which had significantly higher GPA ratings than the
first 2 cohorts and less spread in scores
• Referee ratings also had small but significant
correlations with
– physical determinants grade (r = .18, p = .02)
– communication skills grade (r = .25, p = .003)
Conclusions
• Findings relative to GPA as a significant predictor of
academic performance in a health science program
supports previous research
• Other positive correlates with 1st year GPA:
– Referee ratings
– LOI ratings
suggest that these measures have some value in the
admissions process.
• Referee ratings have even broader potential value,
given positive correlations with performance in
Physical Determinants & Communications Skills
courses.
Conclusions (con’t)
• No support for the requirement of specific
academic pre-requisites
• Previous experience with PWD/vulnerable
populations did not affect academic grades or
communication skills performance
• No support for the requirement of specific
academic pre-requisites
• Previous experience with PWD/vulnerable
populations did not affect academic grades or
communication skills performance
Discussion
• Fairness of admissions process
– Pre-requisite requirements not justified if predictive validity of courses
not substantiated
– No control over authorship of LOI
• Inherent biases in process
– LOI requirement may bias selection towards persons from the same
culture who highlight issues salient to that culture, females, and strong
writers
– Applicants with strong social science background may be overrepresented if GPA is primary selection factor
• Impact on program
– Elimination of pre-requisites broadens applicant pool, may result in
higher calibre class
– More diversity of students in program/field
Discussion
• Role of Pre-Requisites and other Screening
Tools
– LOI, Referee ratings useful as screen out, rather
than selection criteria?
– Certain pre-requisites may make academic
progress easier for student
– Admissions requirements may have value beyond
identification of best applicants
• Credibility of process and applicant
• Interviews may help form relationships, promote program
Limitations
• Findings limited to one health science program
format within a Canadian context, and may not
generalize
• Other potential screening tools not available for
consideration in this study
• Course grades are subject to unsystematic scoring
errors
• Ratings of background experience and LOI subject to
rater error
References
•
•
•
•
•
Auriemma, D. (2002). Admission methods of professional occupational
therapy programs in the united states. Education Special Interest Section
Quarterly, 12(3), 1-4.
Bridle, M. J. (1987). Student selection: A comparison of three methods...
queen's university occupational therapy program. Canadian Journal of
Occupational Therapy, 54(3), 113-117.
Caplan, R.M, Kreiter, C., & Albanese, M. (1996). Preclinical science
course ‘preludes’ taken by premedical students: do they provide a
competitive advantage? AMJ, 71 920-922.
Howard, L., & JeroschHerold, C. (2000). Can entry qualifications be used
to predict fieldwork and academic outcomes in occupational therapy and
physiotherapy students? British Journal of Occupational Therapy, 63(7),
329-334.
Katz, G.M., & Mosey, A.C. (1980). Fieldwork performance, academic
grades, and pre-selection criteria of occupational therapy students.
American Journal of Occupational Therapy, 34(12), 794 – 800.
•
•
•
•
•
•
Kirchner, G. L., & Holm, M. B. (1997). Prediction of academic and
clinical performance of occupational therapy students in an entrylevel master's program. American Journal of Occupational Therapy,
51(9), 775-779.
Kirchner, G. L., Stone, R. G., & Holm, M. B. (2000). Use of admission
criteria to predict performance of students in an entry-level master's
program on fieldwork placements and in academic courses.
Occupational Therapy in Health Care, 13(1), 1-10.
Lewis, M., Smith, S. (2002). Selection of pre-registration
physiotherapy students: Changing to a more objective process.
Physiotherapy, 88(11), 688 – 698.
Nayer, M. (1992). Admission criteria for entrance to physiotherapy
schools: How to choose among many applicants. Physiotherapy
Canada, 44, 41 – 46.
Salvatori, P. (2001). Reliability and validity of admissions tools used
to select students for the health professions.see comment. Advances
in Health Sciences Education, 6(2), 159-175.
Tan, K. P., Meredith, P., & McKenna, K. (2004). Predictors of
occupational therapy students clinical performance: An exploratory
study. Australian Occupational Therapy Journal, 51(1), 25-33.