The Competency Assessment
Toolkit for Professional
Psychology
Catherine L. Grus, Ph.D.,
Associate Executive Director for
Professional Education and Training
Education Directorate
American Psychological Association
THANK YOU
Many thanks to
• Nadya Fouad, University of Wisconsin
• Nadine Kaslow, Emory University
• Benchmarks Workgroups
• Assessment “Toolkit” Workgroup
Overview
• Considerations in the assessment of
competence
• Development of the Toolkit
• Putting the toolkit into practice
Competence …
• “ … the habitual and judicious use of
communication, knowledge, technical
skills, clinical reasoning, emotions, values,
and reflection in daily practice for the
benefit of the individual and community
being served.”
From Epstein and Hundert (2002)
Competence …
• … is developmental depending on stage of
professional functioning
• … context dependent
• … executed according to ethical principles,
guidelines, standards, and values of the
profession and
• … requires public verification
Competencies …

are elements of competence that are ….
– Observable
– Measurable
– Containable
– Practical
– Derived by experts
– And flexible
Assessment of Competence Considerations
• Use of both “formative” and “summative”
•
evaluation methods
Authentic evaluations
• Real world tasks, meaningful activities
• [High] Fidelity: Degree to which
the assessment represents
actual performance
• Employ a range of perspectives (multiinformant) and methods
• Challenge of evaluating attitudes and
values
• Psychometric issues
• Feasibility/costs
• Lifelong self-assessment
What methods are commonly
used to assess students/
trainees?
• In class examinations and course “grades”
• Faculty/supervisor completes rating scale
• Letters of recommendation
• EPPP, state jurisprudence exams
Developing Faculty/Supervisor
Competencies
• Training in “difficult conversations”
– How to “critique constructively”
• Develop awareness of potential dual roles in
evaluation and how that could influence results
– Benefits of having other faculty evaluate students’
performance
– Explicit obligations to make evaluations accurate
• Grade inflation
• Letters of recommendation to “market” vs. evaluate
• Knowledge of a range of assessment tools,
access to resources “toolkits”
Competency Assessment for
“Toolkit” for Professional
Psychology
– Charge from APA Board of Educational Affairs:
Develop a “Toolkit” for professional psychology
– Purpose: Promote broader implementation of
competence assessment and provide information
about application of assessment methods to the
assessment of competence
– Coordinated with Benchmarks Work Group
Benchmarks Core Competencies
Foundational Competencies:
• Professionalism
• Reflective practice
• Scientific knowledge and
methods
• Relationships
• Individual and cultural
diversity
• Ethical and legal
standards and policy
• Interdisciplinary systems
Functional Competencies:
• Assessment
• Intervention
• Consultation
• Research and evaluation
• Supervision
• Teaching
• Administration
• Advocacy
How are the Benchmarks
Organized?
• Core Foundational and Functional
Competencies
• Essential Component: what are the
critical elements of/what
knowledge/skills/attitudes that make up
this competency?
• Behavioral Anchor: what would it look
like if you saw it (essential component)?
Components of Toolkit
• Background and Introduction
• Assessment Method Fact Sheets
• Grid of Assessment Methods and
Competencies/Essential Components Best
used for
Fact Sheet: Portfolio Review
Description
A portfolio is a collection of products, gathered by the person being assessed …
Application
Competency
Applies to1
Predominant Development
Use of2
al Level3
Professionalism
E
F,S
2,3,4
Reflective Practice
E
F,S
1,2,3,4
Implementation
Deciding on form (e.g., web-based or hard copy)
Psychometric Properties
Reliability has not been well established due to the variable content included
in a portfolio
Strengths
Provides a low cost assessment strategy
Challenges
Requires intense commitment of time and labor intensive for the assessors
and the person being assessed
Future directions
Create and implement training modules for assessors and people being
assessed in the portfolio assessment strategy
Written exams
Stand. patient interview
Simulations/role plays
Self-assessment
Record reviews
Ratings Performance
Portfolio reviews
2
1
2
1
2
1
1
2
2
1
2
1
2
2
2
3
2
2
1
2
2
3
3
2
1
2
1
2
2
2
1
3
2
1
Implementation 1
1
1
1
1
1
Progress
evaluation
1
2
2
2
Annual review
3
Consumer surveys
Rating Forms
Intervention Knowledge
2
interventions
Intervention
2
planning
Skills
2
2
Process/outcome data
OSCE
1
1
Case reviews
1
3
3
360 Evaluations
Stand. oral exams
1
2
1
Grid of Possible Assessment Methods
(Toolkit)
Assessment Methods in Toolkit
•
•
•
•
•
•
•
•
360 evaluation
Portfolio
OSCE
Structured Written & Oral
Exams
Case Presentation
Simulation/Role Play
Competence Evaluation
Rating Form
Self-Assessment
• Ratings of live or
•
•
•
•
recorded performance
Standardized Client
Interview
Client/Patient
Process/Outcome
Measure
Consumer Satisfaction
Survey
End of Rotation
Performance Review
Putting it all Together …
Benchmarks
Toolkit
Using the Toolkit and
Benchmarks to Guide the
Assessment of Student Learning
Outcomes
Identify expected
student learning
outcomes using
competency model
(e.g., Benchmarks)
Select
assessment
method(s)
Expected level of
competence attainment
and training plan
developed with trainee
Student demonstrates
problems with
competencies acquisition
Formative
assessment
Student demonstrates
emerging competencies
Summative evaluation to assess
readiness for next level of training
Remediation plan
developed
Training plan
continues
Identify expected
student learning
outcomes using
competency model
(e.g., Benchmarks)
Program Outcome: Students will demonstrate the
following intervention (functional) competencies prior
to going on internship.
1. Knowledge of scientific, theoretical, empirical and
contextual bases of intervention, including theory,
research, and practice
2. Formulates and conceptualizes cases and plan
interventions utilizing at least one consistent
theoretical orientation
3. Clinical skills
4. Implements evidence-based interventions that take
into account empirical support, clinical judgment,
and client diversity (e.g., client characteristics,
values, and context)
5. Evaluate treatment progress and modify treatment
planning as indicated, utilizing established outcome
measures
Identify expected
student learning
outcomes using
competency model
(e.g., Benchmarks)
Select
assessment
method(s)
Written exams
Stand. patient interview
Simulations/role plays
Self-assessment
Record reviews
Ratings Performance
Portfolio reviews
2
1
2
1
2
1
1
2
2
1
2
1
2
2
2
3
2
2
1
2
2
Rating Forms
3
3
2
Intervention
planning
2
1
2
1
2
2
Skills
2
2
1
3
2
1
Implementation
1
1
1
1
1
1
Progress
evaluation
3
1
2
2
2
Consumer surveys
Process/outcome data
2
2
Case reviews
OSCE
1
1
Knowledge
interventions
Intervention
Annual review
1
3
3
360 Evaluations
Stand. oral exams
1
2
1
Grid of Possible Assessment Methods
(Toolkit)
Identify expected
student learning
outcomes using
competency model
(e.g., Benchmarks)
Select
assessment
method(s)
Expected level of
competence attainment
and training plan
developed with trainee
Developmental Level
A. Knowledge of Interventions
Readiness for Practicum
Readiness for Internship
Essential Component:
Basic knowledge of scientific, theoretical,
and contextual bases of intervention and
basic knowledge of the value of evidencebased practice and its role in scientific
psychology
Essential Component:
Knowledge of scientific, theoretical,
empirical and contextual bases of
intervention, including theory, research,
and practice
Behavioral anchor:
Articulates the relationship of EBP to the
science of psychology
Identifies basic strengths and weaknesses of
intervention approaches for different problems
and populations
Behavioral Anchor:
Demonstrates knowledge of interventions
and explanations for their use based on EBP
Demonstrates the ability to select
interventions for different problems and
populations related to the practice setting
Investigates existing literature related to
problems and client issues
Writes a statement of one’s own theoretical
perspective regarding intervention strategies
Identify expected
student learning
outcomes using
competency model
(e.g., Benchmarks)
Select
assessment
method(s)
Expected level of
competence attainment
and training plan
developed with trainee
Student demonstrates
problems with
competencies acquisition
Formative
assessment
Student demonstrates
emerging competencies
Remediation plan
developed
Training plan
continues
Remediation Plan
Competenc
y Domain/
Essential
Component
s
Problem
Behaviors
Expectations
for Acceptable
Performance
Trainee’s
Responsibilitie
s/
Actions
Supervisors’/
Faculty
Responsibilities
/
Actions
Timeframe
for
Acceptable
Performance
Assessmen
t
Methods
Dates of
Evaluatio
n
Consequence
s
for
Unsuccessful
Remediation
Identify expected
student learning
outcomes using
competency model
(e.g., Benchmarks)
Select
assessment
method(s)
Expected level of
competence attainment
and training plan
developed with trainee
Student demonstrates
problems with
competencies acquisition
Formative
assessment
Student demonstrates
emerging competencies
Summative evaluation to assess
readiness for next level of training
Remediation plan
developed
Training plan
continues
For more information:
http://www.apa.org/ed/graduate/competency.html
Coming soon:
Kaslow, N. J., Grus, C. L., Campbell, L. C., Fouad,
N. A., Hatcher, R. L., & Rodolfa, E. R. (in press)
Competency Assessment Toolkit for
Professional Psychology. Training and
Education in Professional Psychology.