Pharmacy Curriculum Outcomes Assessment (PCOA) Moderated by: Abby A. Kahaleh, BPharm, MS, PhD, MPH February 16, 2016 Contact Information Abby A. Kahaleh, BPharm, MS, PhD, MPH Curriculum SIG Chair 847-330-4537 (Phone) Akahaleh@Roosevelt.Edu Presenters 1. Elizabeth A. Coyle, Pharm.D., FCCM, BCPS Assistant Dean of Assessment University of Houston College of Pharmacy 2. Tara Jenkins, RPh, PhD Associate Dean of Academic Affairs University of Houston College of Pharmacy 3. Justine Gortney, Pharm.D., BCPS Clinical Assistant Professor Wayne State University College of Pharmacy Program Description According to NABP, the Pharmacy Curriculum Outcomes Assessment (PCOA) is an effective assessment tool for colleges and schools of pharmacy to use for evaluating student performance in the curricula. PCOA facilitates a review of a pharmacy student’s performance from year to year in relation to the entire curriculum and compares the results to national data. The purpose of this program is to provide participants with expertise from administrators and faculty who used PCOA in their institutions. Objectives Discuss the benefits of using PCOA as a non-high stakes APPE-readiness exam Discuss the benefits of using PCOA as a highstakes APPE-readiness exam Compare and contrast different techniques of setting minimal competency when using PCOA Design a remediation plan for students who do not meet minimal competency on PCOA Presenter I Justine S. Gortney, Pharm.D., BCPS Director of Assessment, Division of Pharmacy Clinical Assistant Professor Wayne State University, Eugene Applebaum College of Pharmacy and Health Sciences Pharmacy Institution’s Stakes Attached to Student Performance 5% 8% High -impacts progression Medium-course grade 26% 61% Gortney JS, Bray B, Salinitri F. Am J Pharm Ed 2015; 79(9) Article 137. Low-development plan None-no negative impact N=38 Where We House PCOA: Professionalism Curriculum Event P1 Year P2 Year P3 Year P4 Year Professional Community Service Student Curriculum Student Student Professional Meeting Student Curriculum Student Student Professional Event Student Curriculum Student Student Summative Evaluation ---- PCOA PCOA ---- Wayne State Operational Timeline Education about Process StudentsAugust, November, December PCOA Thursday & Friday Results Receipt and Distribution Evaluated by SEC March-April Mid January Rollout: P2s 2013, P3s 2014 Distributed to Students and Faculty in April Remediation (Student Development) Advisor meetings May; Plans due in late May/Early June With Low Stakes, How Do We Incentivize Students? Social Element- Breakfast the day of exam High Performance on PCOA - Rewards – Recognition at graduation for top two – Letter of recognition from the Dean’s Office for top 10% – Overall examination scores above the college mean will be rewarded with 1% bonus on your final PPS grade Poor performance: Student development plan WSU Results Distribution Plans Student Class Meetings Faculty Meeting Rollout • Review overall class results • Review how to interpret PCOA scorecard • Distribute individual student scorecards • Distribute individual student development plans • P3 only: Review performance change from P2 and reflection • Provide comparison of the following: • Overall school results • Comparison with national • Comparison within class and across classes • Review timelines of student development plans Publish results on dashboard: http://www.cphs.wayne.edu/pharmd/strategic_assessment_2015/pcoa_data_2015_webupdate.pdf Example of WSU P2 to P3 scorecard Name P2 Overall_SS Area_1_SS Area_2_SS Area_3_SS Area_4_SS Overall Percentile Area 1 Percentile Area 2 Percentile Area 3 Percentile 343 377 338 348 330 58 66 54 56 Area 4 Percentile Physiology Biochemistry 54 70 44 P3 369 418 357 329 380 58 85 52 33 53 70 38 Assess the Change in magnitude of Confidence Score P2 to the change in Rank P3 PCOA 26 41 19 -19 50 0 19 -2 -23 -1 0 -6 Confidence Rank 7 6 Past: Student Development Plans / Thresholds Year 2013 P2a 2014 P2a 2015 P2a 2014 P3b (2013 P2) 2015 P3 b (2014 P2) Basic Bio A B - - E F G L Pharm Sci A B I P Q H N O SBAS A B C D J K L M R S A B I Clin Sci - - T U V - aTwo SD below WSU mean bSame as P2 2013 except Clin Sci (current P3 2 SD below P3 mean) WSU Student Development Plan 2013-2015 (Student Driven) Self assessment of the content area – How have you studied this content in the past? – What are your current study habits? – What do you perceive as a limitation to studying or understanding the specific content at hand? Plan for improvement – How will you proceed forward to improve your knowledge and skills in these areas? • Generate a personal action plan that is practical and implementable • Describe the timeframe you propose to do this over – What resources will you use to do this? Self-reassessment of the content area after plan for improvement implemented – What specific methods will you use to assess your progress? (eg. online quizzes in Access Pharmacy) WSU Student Development Plans Modified for 2016 Cut-points modified – Agreed 1 SD below 3yr NABP composite based on curriculum map for P2s & overall score P3s • P2s for Basic Biomed (282) and PharmSci (280) • P3s Overall Score (303) Revised Student Development Plan – Development/rollout being finalized – Faculty and content-structured – Timed during 6 weeks of first APPE block How Have We Used This Data in WSU Curricular Assessment Limitations We Have to Date Using as low stakes; student motivation Most of our data analysis to this point has been in P2s PCOA with P1 GPA: r=0.50, p<0.01 (N=143) PCOA with P2 GPA: r=0.53, p<0.01 (N=142) Early curriculum, science-based classes showed lowmoderate but significant correlations with PCOA scores Lacking NAPLEX correlation data due to date of exam Does not appear on surface to relate to risk for APPE rotation failure; APPE grading scale lacking variability At WSU, How Do We Feel this Has Benefited our Students and Program Provided a starting block for summative evaluations Provided positive feedback for students regarding their knowledge Provided specific assessment data to track our students’ knowledge and benchmark against others – Used specifically during curricular renewal process • Part of a triangulation of data driving “renewal” • Plan on using as a consistency check after “renewal” – ACPE self-study data – Higher Learning Commission (HLC) outcomes measurement Presenters II and III Elizabeth A. Coyle, Pharm.D., FCCM, BCPS Assistant Dean of Assessment University of Houston College of Pharmacy Tara Jenkins, RPh, PhD Associate Dean of Academic Affairs University of Houston College of Pharmacy Utilizing the PCOA as a High Stakes Exam Objectives UHCOP High-Stakes Exam History PCOA Pilot Setting Cut-off Scores Remediation Questions PCOA Project Team Elizabeth A. Coyle, Pharm.D., FCCM, BCPS Catherine L. Hatfield, Pharm.D. Gerida Brown Tara L. Jenkins, RPh, Ph.D. UHCOP Faculty Milemarker Homegrown cumulative didactic exam Started as high stakes for P3’s in 2000 – Included questions from all 3 years of didactic curriculum – Written by faculty – 200 case-based questions – Questions Angoffed – Mean of Angoff score from 200 questions had to be met for minimal competency Julie Szylagyi. AJPE. 2008; 72 (5) Article 101. Milemarker P3’s had to meet minimum competency on Milemarker III to advance to APPEs – Average pass rate from 2000-2012 was 99% P1 & P2’s also took Milemarkers I & II – Formative with remediation and points towards Milemarker III New cases & validated questions were hard to accumulate Julie Szylagyi. AJPE. 2008; 72 (5) Article 101. PCOA Pilot January of 2013 P3 students – All required to take – Formative information only P3 students still had to take Milemarker III in April 2013 Exam Comparisons MILEMARKER III Overall Questions: Biomedical Sciences: Clinical Sciences: Pharmaceutical Sciences: Social/Behavior/Admin: 200 Questions 10.5% (21 questions) 61% (122 questions) 13.5% (27 questions) 15% (30 questions) PCOA 220 (20 “test” questions) 16% (32 questions) 32% (64 questions) 30% (60 questions) 22% (44 questions) Results Comparison Milemarker III PCOA Overall Average (% Correct): 78.9% UH 73%/ National 67% Biomedical Sciences (% Correct): 82.9% UH 73%/National 66% Clinical Sciences (%Correct): 80.4% UH 76%/National 73% Pharmaceutical Sciences (%Correct): 68.9% UH 67%/ National 57% Social/Behavior/Admin (% Correct): 76.5% UH 77%/ National 71% PCOA Benefits External measurement and benchmark of our students’ performance in pharmacy curricula Psychometrically sound questions written by content experts from colleges of pharmacy around the country Ability to trend results longitudinally Does not require faculty time to analyze and write capstone questions Identify Cut off Score For P3 PCOA Which Scores to Use? Percentile rank — the percentage of scores in a distribution that is equal to or lower than the measured score. Raw score — Total number of raw points a test taker receives based on the number of questions answered correctly; typically, for example, 20 correct answers means a raw score of 20 Scaled score — Scores that have been mathematically transformed from one set of numbers (i.e., the raw score) to another set of numbers in order to make them comparable in some way — for example, across different editions, or “forms,” of the same test X Tuan and R Michel. R & D Connections No 16. Sept. 2011 National Scaled Scores (normed reference sample) National Scaled Score 700 50%ile 50%ile 361 50%ile 50%ile 50%ile 50%ile 50%ile 50%ile 357 (scaled normed referenced score change) 0 1 2 3 4 5 Year 6 7 8 UHCOP Cutoff Score * NSS vs. 10% below NSS vs. 1 or 2 STD * PCOA Cutoff score for P3’s is 1 STD from the NSS scale score. The STD will be calculated from the PCOA student scores each year. 3 Year Comparisons P3 Class (N) National Class Scaled Percentile Score Class Ave Scaled Score Standard Deviation Passing Score Number of Remediating Number (%) Number ≥ 90th after 1st < NSS percentile PCOA Number ≤ 20th percentile P3 Class 2013 (N=109) 75th 361 405 41 N/A (pilot) N/A (pilot) 14 (12.8%) 28 1 P3 Class 2014 (N=113) 56th 361 388 41 325 14* 45 (39.8%) 10 11 P3 Class 2015 (N=109) 65th 357 388 47 310 2 23 (21.1%) 22 3 * If we would have used 1 STD cutoff, 11/113 would have remediated High Stakes Exams & Academic Performance Coyle EA, Hatfield CL, Cottreau JM, Brown G, Smesny A. Poster (July 2013) AACP Annual Meeting, Chicago, IL Remediation What kind of remediation do you use? – Retest PCOA? – Homegrown exam? – Tailored remediation? How do remediated students perform on rotations? How often do you remediate? PCOA Remediation Policy for P3’s • Meet minimum competency go on to APPE in May PCOA Jan/Feb ALL P3's • Meet minimum competency go on to APPE in May. • Do not meet minimum competency, receive I in IPPE II. • Follow a remediation plan/take a remediation course once classes end in May. Remediation 1 March/April PCOA • Meet minimum competency start APPE's in August . • Do not meet minimum competency, follow remediation plan/take remediation course. Remediation 2 July PCOA* Remediation 3 Sep/Oct PCOA • Meet minimum competency start APPE's in January. • Do not meet minimum competency, follow remediation plan/take remediation course. After 3rd remediation repeat until student meets 6 year program limit. After 5th remediation I in IPPE II becomes a U. * Not in PCOA normal testing windows Conclusions Adapting PCOA as a high stakes assessment was an easy transition at UHCOP Minimum competency should be set off of the scaled score and not percentile or raw score A clear remediation process is imperative to close the loop for high stakes exams Q&A Discussion Questions 1. Compare and contrast the PCOA exam with other assessment instruments 2. What are the advantages/disadvantages of various remediation plans? 3. Describe examples of successful student/faculty development plans 4. What are the steps for closing the assessment feedback loop based on the PCOA results? Thank You!