Pharmacy Curriculum
Outcomes Assessment
(PCOA)
Moderated by:
Abby A. Kahaleh, BPharm, MS, PhD, MPH
February 16, 2016
Contact Information
Abby A. Kahaleh, BPharm, MS, PhD, MPH
Curriculum SIG Chair
847-330-4537 (Phone)
Akahaleh@Roosevelt.Edu
Presenters
1. Elizabeth A. Coyle, Pharm.D., FCCM, BCPS
Assistant Dean of Assessment
University of Houston College of Pharmacy
2. Tara Jenkins, RPh, PhD
Associate Dean of Academic Affairs
University of Houston College of Pharmacy
3. Justine Gortney, Pharm.D., BCPS
Clinical Assistant Professor
Wayne State University College of Pharmacy
Program Description

According to NABP, the Pharmacy Curriculum Outcomes
Assessment (PCOA) is an effective assessment tool for
colleges and schools of pharmacy to use for evaluating
student performance in the curricula. PCOA facilitates a
review of a pharmacy student’s performance from year to
year in relation to the entire curriculum and compares the
results to national data. The purpose of this program is to
provide participants with expertise from administrators and
faculty who used PCOA in their institutions.
Objectives

Discuss the benefits of using PCOA as a non-high
stakes APPE-readiness exam

Discuss the benefits of using PCOA as a highstakes APPE-readiness exam

Compare and contrast different techniques of
setting minimal competency when using PCOA

Design a remediation plan for students who do
not meet minimal competency on PCOA
Presenter I
Justine S. Gortney,
Pharm.D., BCPS
Director of Assessment,
Division of Pharmacy
Clinical Assistant Professor
Wayne State University,
Eugene Applebaum College of Pharmacy and Health
Sciences
Pharmacy Institution’s Stakes
Attached to Student Performance
5%
8%
High -impacts
progression
Medium-course grade
26%
61%
Gortney JS, Bray B, Salinitri F. Am J Pharm Ed 2015; 79(9) Article 137.
Low-development plan
None-no negative impact
N=38
Where We House PCOA:
Professionalism Curriculum
Event
P1 Year
P2 Year
P3 Year
P4 Year
Professional
Community
Service
Student
Curriculum
Student
Student
Professional
Meeting
Student
Curriculum
Student
Student
Professional
Event
Student
Curriculum
Student
Student
Summative
Evaluation
----
PCOA
PCOA
----
Wayne State Operational Timeline
Education
about Process
StudentsAugust,
November,
December
PCOA
Thursday &
Friday
Results
Receipt and
Distribution
Evaluated by
SEC March-April
Mid
January
Rollout: P2s 2013, P3s 2014
Distributed to
Students and
Faculty in April
Remediation
(Student
Development)
Advisor meetings
May; Plans due in
late May/Early June
With Low Stakes, How Do We
Incentivize Students?

Social Element- Breakfast the day of exam

High Performance on PCOA - Rewards
–
Recognition at graduation for top two
–
Letter of recognition from the Dean’s Office
for top 10%
–
Overall examination scores above the
college mean will be rewarded with 1%
bonus on your final PPS grade

Poor performance: Student development plan
WSU Results Distribution Plans
Student Class Meetings
Faculty Meeting Rollout
• Review overall class results
• Review how to interpret PCOA
scorecard
• Distribute individual student
scorecards
• Distribute individual student
development plans
• P3 only: Review performance
change from P2 and reflection
• Provide comparison of the
following:
• Overall school results
• Comparison with national
• Comparison within class and
across classes
• Review timelines of student
development plans
Publish results on dashboard:
http://www.cphs.wayne.edu/pharmd/strategic_assessment_2015/pcoa_data_2015_webupdate.pdf
Example of WSU P2 to P3 scorecard
Name
P2
Overall_SS
Area_1_SS
Area_2_SS
Area_3_SS
Area_4_SS
Overall Percentile
Area 1 Percentile
Area 2 Percentile
Area 3 Percentile
343
377
338
348
330
58
66
54
56
Area 4 Percentile
Physiology
Biochemistry
54
70
44
P3
369
418
357
329
380
58
85
52
33
53
70
38
Assess the
Change in magnitude of Confidence
Score P2 to the change in
Rank
P3
PCOA
26
41
19
-19
50
0
19
-2
-23
-1
0
-6
Confidence
Rank
7
6
Past: Student Development Plans /
Thresholds
Year
2013 P2a
2014 P2a
2015 P2a
2014 P3b
(2013 P2)
2015 P3 b
(2014 P2)
Basic Bio
A
B
-
-
E
F
G
L
Pharm Sci
A
B
I
P
Q
H
N
O
SBAS
A
B
C
D
J
K
L
M
R
S
A
B
I
Clin Sci
-
-
T
U
V
-
aTwo
SD below WSU mean
bSame as P2 2013 except Clin Sci (current P3 2 SD below P3 mean)
WSU Student Development Plan
2013-2015 (Student Driven)



Self assessment of the content area
– How have you studied this content in the past?
– What are your current study habits?
– What do you perceive as a limitation to studying or
understanding the specific content at hand?
Plan for improvement
– How will you proceed forward to improve your knowledge
and skills in these areas?
• Generate a personal action plan that is practical and
implementable
• Describe the timeframe you propose to do this over
– What resources will you use to do this?
Self-reassessment of the content area after plan for
improvement implemented
– What specific methods will you use to assess your
progress? (eg. online quizzes in Access Pharmacy)
WSU Student Development Plans
Modified for 2016


Cut-points modified
– Agreed 1 SD below 3yr NABP composite based
on curriculum map for P2s & overall score P3s
• P2s for Basic Biomed (282) and PharmSci
(280)
• P3s Overall Score (303)
Revised Student Development Plan
– Development/rollout being finalized
– Faculty and content-structured
– Timed during 6 weeks of first APPE block
How Have We Used This Data in
WSU Curricular Assessment
Limitations We Have to Date

Using as low stakes; student motivation

Most of our data analysis to this point has been in P2s

PCOA with P1 GPA: r=0.50, p<0.01 (N=143)

PCOA with P2 GPA: r=0.53, p<0.01 (N=142)

Early curriculum, science-based classes showed lowmoderate but significant correlations with PCOA scores


Lacking NAPLEX correlation data due to date of exam
Does not appear on surface to relate to risk for APPE
rotation failure; APPE grading scale lacking variability
At WSU, How Do We Feel this Has
Benefited our Students and Program

Provided a starting block for summative evaluations

Provided positive feedback for students regarding their
knowledge

Provided specific assessment data to track our students’
knowledge and benchmark against others
–
Used specifically during curricular renewal process
•
Part of a triangulation of data driving “renewal”
•
Plan on using as a consistency check after “renewal”
–
ACPE self-study data
–
Higher Learning Commission (HLC) outcomes
measurement
Presenters II and III
Elizabeth A. Coyle, Pharm.D., FCCM,
BCPS
Assistant Dean of Assessment
University of Houston College of
Pharmacy
Tara Jenkins, RPh, PhD
Associate Dean of Academic Affairs
University of Houston College of
Pharmacy
Utilizing the PCOA as a
High Stakes Exam
Objectives

UHCOP High-Stakes Exam History

PCOA Pilot

Setting Cut-off Scores

Remediation

Questions
PCOA Project Team

Elizabeth A. Coyle, Pharm.D., FCCM, BCPS

Catherine L. Hatfield, Pharm.D.

Gerida Brown

Tara L. Jenkins, RPh, Ph.D.

UHCOP Faculty
Milemarker

Homegrown cumulative didactic exam

Started as high stakes for P3’s in 2000
– Included questions from all 3 years of
didactic curriculum
– Written by faculty
– 200 case-based questions
– Questions Angoffed
– Mean of Angoff score from 200
questions had to be met for minimal
competency
Julie Szylagyi. AJPE. 2008; 72 (5) Article 101.
Milemarker
P3’s had to meet minimum competency
on Milemarker III to advance to APPEs
– Average pass rate from 2000-2012 was
99%
 P1 & P2’s also took Milemarkers I & II
– Formative with remediation and points
towards Milemarker III
 New cases & validated questions were
hard to accumulate

Julie Szylagyi. AJPE. 2008; 72 (5) Article 101.
PCOA Pilot

January of 2013

P3 students
– All required to take
– Formative information only

P3 students still had to take Milemarker
III in April 2013
Exam Comparisons
MILEMARKER III
Overall Questions:
Biomedical Sciences:
Clinical Sciences:
Pharmaceutical Sciences:
Social/Behavior/Admin:
200 Questions
10.5% (21 questions)
61% (122 questions)
13.5% (27 questions)
15% (30 questions)
PCOA
220 (20 “test” questions)
16% (32 questions)
32% (64 questions)
30% (60 questions)
22% (44 questions)
Results Comparison
Milemarker III
PCOA
Overall Average (% Correct):
78.9%
UH 73%/ National 67%
Biomedical Sciences (% Correct):
82.9%
UH 73%/National 66%
Clinical Sciences (%Correct):
80.4%
UH 76%/National 73%
Pharmaceutical Sciences (%Correct):
68.9%
UH 67%/ National 57%
Social/Behavior/Admin (% Correct):
76.5%
UH 77%/ National 71%
PCOA Benefits

External measurement and benchmark of
our students’ performance in pharmacy
curricula

Psychometrically sound questions written
by content experts from colleges of
pharmacy around the country

Ability to trend results longitudinally

Does not require faculty time to analyze
and write capstone questions
Identify Cut off Score
For P3 PCOA
Which Scores to Use?

Percentile rank — the percentage of scores in a
distribution that is equal to or lower than the measured
score.

Raw score — Total number of raw points a test taker
receives based on the number of questions answered
correctly; typically, for example, 20 correct answers means
a raw score of 20

Scaled score — Scores that have been mathematically
transformed from one set of numbers (i.e., the raw score)
to another set of numbers in order to make them
comparable in some way — for example, across different
editions, or “forms,” of the same test
X Tuan and R Michel. R & D Connections No 16. Sept. 2011
National Scaled Scores
(normed reference sample)
National Scaled Score
700
50%ile
50%ile
361
50%ile
50%ile
50%ile
50%ile
50%ile
50%ile
357 (scaled normed
referenced score change)
0
1
2
3
4
5
Year
6
7
8
UHCOP Cutoff Score
*
NSS vs. 10% below NSS vs. 1 or 2 STD
* PCOA Cutoff score for P3’s is 1 STD from
the NSS scale score. The STD will be
calculated from the PCOA student scores
each year.
3 Year Comparisons
P3 Class
(N)
National
Class
Scaled
Percentile
Score
Class Ave
Scaled
Score
Standard
Deviation
Passing
Score
Number of
Remediating Number (%) Number ≥ 90th
after 1st
< NSS
percentile
PCOA
Number ≤ 20th
percentile
P3 Class
2013
(N=109)
75th
361
405
41
N/A
(pilot)
N/A
(pilot)
14 (12.8%)
28
1
P3 Class
2014
(N=113)
56th
361
388
41
325
14*
45 (39.8%)
10
11
P3 Class
2015
(N=109)
65th
357
388
47
310
2
23 (21.1%)
22
3
* If we would have used 1 STD cutoff, 11/113
would have remediated
High Stakes Exams & Academic
Performance
Coyle EA, Hatfield CL, Cottreau JM, Brown G, Smesny A. Poster (July
2013) AACP Annual Meeting, Chicago, IL
Remediation

What kind of remediation do you use?
– Retest PCOA?
– Homegrown exam?
– Tailored remediation?

How do remediated students perform
on rotations?

How often do you remediate?
PCOA Remediation Policy for P3’s
• Meet minimum
competency go on to
APPE in May
PCOA Jan/Feb
ALL P3's
• Meet minimum
competency go on to APPE
in May.
• Do not meet minimum
competency, receive I in
IPPE II.
• Follow a remediation
plan/take a remediation
course once classes end in
May.
Remediation 1
March/April PCOA
• Meet minimum
competency start APPE's
in August .
• Do not meet minimum
competency, follow
remediation plan/take
remediation course.
Remediation 2
July PCOA*
Remediation 3
Sep/Oct PCOA
• Meet minimum
competency start APPE's
in January.
• Do not meet minimum
competency, follow
remediation plan/take
remediation course.
After 3rd remediation repeat until student meets 6 year program limit.
After 5th remediation I in IPPE II becomes a U.
* Not in PCOA normal testing windows
Conclusions

Adapting PCOA as a high stakes
assessment was an easy transition at
UHCOP

Minimum competency should be set off of
the scaled score and not percentile or raw
score

A clear remediation process is imperative
to close the loop for high stakes exams
Q&A
Discussion Questions
1. Compare and contrast the PCOA exam with other
assessment instruments
2. What are the advantages/disadvantages
of various remediation plans?
3. Describe examples of successful student/faculty
development plans
4. What are the steps for closing the assessment
feedback loop based on the PCOA results?
Thank You!