Finding the ‘Right’ Students: The Search for Predictive Validity in Applicant Screening for the Health Sciences Rosemary Lysaght, Ph.D. Catherine Donnelly, M.Sc. Michelle Villeneuve, M.Sc. School of Rehabilitation Therapy Background • Impetus for examination of our admissions process was the creation of the new masters entry-level program in 2004 Goals for admission included: • An evidence-based admissions process • Increase the applicant pool • Efficiency in admission process Overall Aim of Selection To select students who will: • Succeed in the academic program • Perform credibly in professional practice • Possess the traits of character and ethical values desired of a professional person (Nayer, 1992) Commonly Used Selection Criteria in Health Sciences • Pre-admission academic grades • Discipline-specific aptitude tests (MCAT, HOAE) • Interviews • Written submissions • Letters of reference • Prerequisites (Auriemma, 2002; Salvatori, 2001) Predictors of academic success • Evidence across disciplines supports the predictive validity of – Pre-admission academic grades • No clear support found for – – – – – Discipline-specific aptitude tests Interviews Written submissions Letters of reference Prerequisites (Bridle, 1987; Caplan et al, 1996; Howard & Jerosh-Herold, 2000; Lewis & Smith, 2002; Kirchner et al., 2000; Kirchner & Holm, 1997; Salvatori, 2001) Predictors of clinical performance • Unclear relationship between pre-admission performance and clinical performance • Valid outcome measures difficult to identify • Lack of consistent raters • Variability across settings (Howard & Jerosch-Herold, 2000; Kirchner & Holm, 1997; Katz & Mosey, 1980; Tan et al., 2004) Research Questions 1. What admissions screening tools best predict academic performance in a masters level OT program? Selected academic courses and Clinical performance 2. Does undergraduate coursework predict success in topically-related coursework? Method • Analysis of existing data for 128 students admitted to the OT masters program – Multiple regression – Models created for each research question • Project received approval by the Queen’s University’s Research Ethics Board • Sample included 3 cohorts of students (1st three years of the new professional masters program) Factors Considered in OT Admissions – GPA – Academic transcript – Letter of intent – 2 referee rating forms/letters – Supporting data (foreign/non-traditional applicants) Predictor Variables • Undergraduate Grade Point Average (GPA) – from ORPAS • Letter of Intent (LOI) • Average rating, 2 raters • Referee Rating • 5 point scale, 12 items • Average total score of 2 external referees Predictor Variables (con’t) • Additional ratings were created for each student based on file review (1 = weak to 5 = exceptional) Physical Sciences Experience/Preparation • e.g. Anatomy, physiology, kinesiology (r for 2 raters = .98) Social Sciences Experience/Preparation • e.g. Psychology, Sociology, Family Studies (r=.91) Experience with People with Disabilities/Vulnerable Populations • Rated for number, duration, and relevance (r=.79) • Information drawn from Letter of Intent, referee letters and experience questionnaire Criterion Variables • Program GPA (1st year) • Communication Skills Practicum • Targeted course grades in: Physical Determinants of Occupation Cognitive-Neurological Determinants Psycho-Emotional Determinants Descriptive Statistics - Predictor Variables Variable N Mean SD Undergraduate GPA Letter of Intent Rating Referee Rating 129 129 129 3.24 71.70 4.72 .278 12.86 .25 Bkg. Physical Sciences Bkg. Social Sciences Bkg. Experience with PWD 127 127 127 3.92 2.90 3.70 1.35 1.66 .88 Descriptive Statistics - Criterion Variables Variable N Mean SD First Year GPA 128 80.29 3.00 Communication Skills Rating Physical Determinants Grade 112 129 8.44 77.72 1.19 4.05 Cogntive-Neuro Det. Grade Psycho-Emotional Det. Grade 126 126 78.97 78.89 5.13 5.03 Results – First Year GPA • Predictor Variables: GPA, Letter of Intent, Referee Rating • Model is significant (R2 = .101; F= 4.65, p < .001) • GPA is only variable with significant beta score (p = .005) • Significant correlations between all three variables and program GPA Results – Communication Skills • Predictor Variables: GPA, Letter of Intent, Referee Rating, all experience ratings, • • • Model is not significant (R2 = .021; F= .62) Referee rating is significantly correlated with performance rating No background experience ratings were correlated with performance rating Results – Physical Determinants • Predictor Variables: GPA, all experience ratings Result: • • Model not significant (R2 = .009; F=.275) No significant correlations between any background experience rating and outcome Results – Cognitive-Neuro • Predictor Variables: GPA, all experience ratings Result: • • • Model is significant (R2 = .14; F=5.0, p = .004) GPA only variable with significant beta score (p = .00) Background in physical sciences was significantly negatively correlated with outcome Results – PsychoEmotional Det. • Predictor Variables: GPA, all experience ratings Result: • • • Model is significant (R2 = .082; F = 2.6, p< .05) Social science bkgd only variable with significant beta score (-.2 ) GPA is significantly positively correlated with outcome, while SS bkgd is significantly negatively correlated. Additional Observations • The contribution of the undergraduate GPA to all models was reduced by the addition of the 3rd cohort, which had significantly higher GPA ratings than the first 2 cohorts and less spread in scores • Referee ratings also had small but significant correlations with – physical determinants grade (r = .18, p = .02) – communication skills grade (r = .25, p = .003) Conclusions • Findings relative to GPA as a significant predictor of academic performance in a health science program supports previous research • Other positive correlates with 1st year GPA: – Referee ratings – LOI ratings suggest that these measures have some value in the admissions process. • Referee ratings have even broader potential value, given positive correlations with performance in Physical Determinants & Communications Skills courses. Conclusions (con’t) • No support for the requirement of specific academic pre-requisites • Previous experience with PWD/vulnerable populations did not affect academic grades or communication skills performance • No support for the requirement of specific academic pre-requisites • Previous experience with PWD/vulnerable populations did not affect academic grades or communication skills performance Discussion • Fairness of admissions process – Pre-requisite requirements not justified if predictive validity of courses not substantiated – No control over authorship of LOI • Inherent biases in process – LOI requirement may bias selection towards persons from the same culture who highlight issues salient to that culture, females, and strong writers – Applicants with strong social science background may be overrepresented if GPA is primary selection factor • Impact on program – Elimination of pre-requisites broadens applicant pool, may result in higher calibre class – More diversity of students in program/field Discussion • Role of Pre-Requisites and other Screening Tools – LOI, Referee ratings useful as screen out, rather than selection criteria? – Certain pre-requisites may make academic progress easier for student – Admissions requirements may have value beyond identification of best applicants • Credibility of process and applicant • Interviews may help form relationships, promote program Limitations • Findings limited to one health science program format within a Canadian context, and may not generalize • Other potential screening tools not available for consideration in this study • Course grades are subject to unsystematic scoring errors • Ratings of background experience and LOI subject to rater error References • • • • • Auriemma, D. (2002). Admission methods of professional occupational therapy programs in the united states. Education Special Interest Section Quarterly, 12(3), 1-4. Bridle, M. J. (1987). Student selection: A comparison of three methods... queen's university occupational therapy program. Canadian Journal of Occupational Therapy, 54(3), 113-117. Caplan, R.M, Kreiter, C., & Albanese, M. (1996). Preclinical science course ‘preludes’ taken by premedical students: do they provide a competitive advantage? AMJ, 71 920-922. Howard, L., & JeroschHerold, C. (2000). Can entry qualifications be used to predict fieldwork and academic outcomes in occupational therapy and physiotherapy students? British Journal of Occupational Therapy, 63(7), 329-334. Katz, G.M., & Mosey, A.C. (1980). Fieldwork performance, academic grades, and pre-selection criteria of occupational therapy students. American Journal of Occupational Therapy, 34(12), 794 – 800. • • • • • • Kirchner, G. L., & Holm, M. B. (1997). Prediction of academic and clinical performance of occupational therapy students in an entrylevel master's program. American Journal of Occupational Therapy, 51(9), 775-779. Kirchner, G. L., Stone, R. G., & Holm, M. B. (2000). Use of admission criteria to predict performance of students in an entry-level master's program on fieldwork placements and in academic courses. Occupational Therapy in Health Care, 13(1), 1-10. Lewis, M., Smith, S. (2002). Selection of pre-registration physiotherapy students: Changing to a more objective process. Physiotherapy, 88(11), 688 – 698. Nayer, M. (1992). Admission criteria for entrance to physiotherapy schools: How to choose among many applicants. Physiotherapy Canada, 44, 41 – 46. Salvatori, P. (2001). Reliability and validity of admissions tools used to select students for the health professions.see comment. Advances in Health Sciences Education, 6(2), 159-175. Tan, K. P., Meredith, P., & McKenna, K. (2004). Predictors of occupational therapy students clinical performance: An exploratory study. Australian Occupational Therapy Journal, 51(1), 25-33.