Program Directors 101 P ando ™ Workshop
Effective Outcomes Assessment
Meredith Davison
John B urns
Objectives
• Upon completion o f this workshop, participants will be a ble to:
• Foster student success without lowering the bar • Distinguish between student outcomes assessment and program evaluation • Describe the key components of an effective assessment or evaluation plan • Develop and implement an effective student outcomes assessment plan Acknowledgements
Thanks to: Bob Philpot, PhD, PA-­C
Matt Dane Baker, DHSc, PA-­C
Bradford W. Schwarz MS, PA-­C
For their contributions to this presentation
PAEA Copyright 2016
1
Why Is It Important?
• Evaluate effectiveness and alignment of curricula • Improve the program and its standing • Inform planning and decision making • Understand the impact of program changes • Highlight program successes • Provide evidence of need when requesting resources • Advise students with a set of learning outcomes
Why Easier f or Us?-­-­-­ARC-­PA!!
• C1.01 E valuation
“The Program must implement an ongoing program self-­assessment process t hat is designed t o document program effectiveness and foster program improvement.”
C1.01 Annotation
• Self-­assessment process
– Collection of data and interpreting evidence
• Student learning
• Program functions and outcomes
– Qualitative and quantitative data
– Careful thought to data:
• Collection
• Management
• Interpretation
PAEA Copyright 2016
2
ARC-­PA
l C3 – Student Evaluation
lFormative evaluation
lSummative evaluation
l C4 – Clinical Site Evaluation
Outcome Assessment:
Student and Program
Ø Should lead to:
lIdentification of strengths and opportunities for improvement
lPlanning and modification of curriculum for enhancement
lReassessment of measurement effectiveness
We Understand Value of Monitoring • Tracks our progress a nd its easy!
– # of Cs in pre-­clinical c oursework
– Attrition rates
– Deceleration rates
– PACKRAT
– EOR Exams
– PANCE
– Graduation rates
PAEA Copyright 2016
3
Definitions
• Outcomes A ssessment
– Answers the question – what (learning) has changed about the student between matriculation and graduation (and who or what is responsible) ?
– Program outcomes specify the measurable knowledge, skill, or behavior you expect students to be able to demonstrate as a result of their participation in the program.
– Assessment uses a feedback loop that involves identifying desired results;; collecting and analyzing relevant information (data);; and then using the findings to make changes to your program. System of Outcomes Assessment
Systematized
Comprehensive
Organized
Useful
Cost Effective
Assessment Cycle*
Develop c learly articulated written statements of learning outcomes
Use results of assessments to improve teaching and learning
Design experiences that provide opportunities to achieve outcomes
Assess student achievement of outcomes
* Figure modified from the Middles States Commission on Higher Education
PAEA Copyright 2016
4
Infrastructure: Culture of Assessment
•
•
•
•
•
Buy in from faculty
Transparency
Follows institutional p rocedures
Resources
Linked to faculty evaluation, b udget planning
• Integrated in syllabi
• Dissemination!
Components
•
•
Outcomes are measureable!
Curricular MAPS
•
Assessment instruments (direct and indirect)
•
•
– Exams
– Rubrics
Internal
External
•
•
•
•
Qualitative and Quantitative
Schedule
Activity and Results (data collection)
Evidence of Analysis of Results (analyze)
•
Evidence of change based on assessment results •
Documentation
5 Stages
• Establish clearly defined expected learning outcomes
• Deliver a curriculum in accordance with the expected outcomes
• Continuously teach, react with, advise, and evaluate students during their educational experience
• Provide multiple measures and a continuous feedback loop that allows intervention strategies that enable students to successfully complete the curriculum
• Assess the outcomes of the program and then reevaluate the expected learning outcomes, curriculum and students, and make program changes as needed.
PAEA Copyright 2016
5
As a Program Director…
• Does the curriculum cover e verything students n eed to know?
Program Level Assessment Exercise
•
•
•
•
•
•
•
•
•
Provide a data collection and distribution system to monitor trends, meet the goals of students, and meet the needs of the community
Prove a process to effectively assess student learning and support services
Provide the necessary information to strengthen programs, promote access, and insure efficient use of resource
Identify needed curriculum and service improvements and revisions
Identify equipment and other resource modifications
Identify the need for alternative delivery methods
Keep faculty and staff aware of changing trends
Identify areas for improvement of teaching, faculty and staff development and services
Compare business and industry’s requirements with your PA Program.
Group Exercise 5-­7 minutes
• Take a moment to review all of the Program level assessment g oals
• Review your assigned g oals;; d iscuss what experience you h ave h ad a s it pertains to this goal a nd what d ata you u sed to achieve this particular goal. • If you d o n ot h ave a p ersonal e xperience to d raw from, be creative!
PAEA Copyright 2016
6
Developing a Plan
Outcome assessment (old)
Learning Learning Learning Learning Learning
Outcome Outcome Outcome Outcome Outcome
1
2
3
4
5
Course
PA1
x
PA2
x
x
x
x
PA3
x
PA4
x
PA5
x
x
x
x
Outcome Assessment (new)
Learning Learning Learning Learning Learning
Outcome Outcome Outcome Outcome Outcome
1
2
3
4
5
Course
PA1
87%
PA2
65%
70%
90%
88%
PA3
75%
PA4
85%
PA5
82%
72%
84%
69%
PAEA Copyright 2016
88%
90
88%
82.5%
89%
7
Focused F eedback to F aculty:
Exam Level
Additionally includes item analysis, breakdown of learning outcomes by category, ie
Focused F eedback to F aculty:
Course Level
Course LO
Assessment
Avg Grade
# items per L O actions
1
Exam 1,2
85%
45
2
Exam 3, Paper 71%
1
Exam 4, 90%
presentation 1
20
3
PAEA Copyright 2016
60
8
Capitalize…
• On the culture of a ssessment we h ave in accredited p rofessional p rograms!
– ……….think of y our c olleagues who t each in liberal arts!
Tracking Longitudinal Data
Exams
Items
BT 1 knowledge
BT 2
application
BT 3
synthesis
Cardiology
146
2951
Group Average
84%
136
998
84%
47
83
81%
35
293
83%
Renal
23
110
78%
What data concerns you in this table? Resources
•
Accreditation Review Commission on Education for the Physician Assistant (2010). Accreditation Standards for Physician Assistant Education, 4th edition. •
Commission on Colleges of the Southern Association of Colleges and Schools, (2010). Principles of Accreditation: Foundations for Quality Enhancement. Author. Available: http://sacscoc.org/princi pl es.as p
Suskie, L.(2009). Assessing Student Learning: A Common Sense Guide, 2nd ed. San Francisco: Jossey-­Bass.
Berk R. (2006). Thirteen Strategies to Measure College Teaching, 1st
ed. Sterling, Va: Stylus Publishing
National Institute for Learning Outcomes Assessment http://www.learning outc ome ass ess me nt.org
•
•
•
PAEA Copyright 2016
9