Competency Workshop
Medbiquitous
May 2011
Susan Albright
Workshop Objectives
By the end of this workshop, participants will
be able to:
 Describe a process for developing
competency frameworks
 Identify challenges and potential barriers
to developing and implementing
competency frameworks
 Identify opportunities available with the
implementation of competency
frameworks in an electronic system
Agenda







Introductions
Part 1 – Background on the Tufts
Experience

Activity

Discussion
Part 2 Assessment
Break
Part 3 – Developing the system
Discussion
Summary
Why Competencies?

“Competency-based education is
characterized by the development of clearly
defined and explicitly published
competencies, mapping the curriculum to
achieve competencies, and assessment
process matched to competencies.”
(Ron Harden, Medical Teacher 2002 #24 Vol 2
What is Competency Based
Education




CBE is an institutional process that moves education from
focusing on what academics believe graduates need to know
(teacher-focused) to what students need to know and be able
to do
CBE is focused on outcomes (competencies) that are linked to
workforce needs as defined by the profession.
CBE necessitates complex assessment through portfolios or
experiential learning assessement in field experience
Large skill sets are broken down into competencies which
sequestial levels of mastery
CBE




Competencies may reinforce one
another from basic to advanced as
learning progresses
Continual refinement of competencies is
essential as this a process not a product
Consistency in expectations across the
curriculum
Consistency in grading
http://www.ceph.org/pdf/Competencies_TA.pdf
Council on Education for Public Health
Competency Frameworks






ACGME
Scottish Doctors
Tomorrow’s Doctors
The Tuning Project
CanMeds
MSOP AAMC
ACGME Competency Domains






Patient care
Medical knowledge
Practice-based learning and
improvement
Interpersonal and communication skills
Professionalism
Systems-based practice
AAMC Learning objectives




The first of the four attributes is that physicians must be altruistic.
There are seven learning objectives, including the objective that before
graduation, the student can demonstrate compassionate treatment of
patients and respect for their privacy and dignity.
The second attribute is that physicians must be knowledgeable; one of
the six learning objectives is that the student can demonstrate
knowledge of the normal structure and function of the body and of
each of its major organ systems.
The third attribute is that physicians must be skillful; one of the eleven
learning objectives is that the student have knowledge about relieving
pain and ameliorating the suffering of patients.
The last attribute is that physicians must be dutiful; one of the six
learning objectives is that the student have knowledge of the
epidemiology of common maladies within a defined population, and the
systematic approaches useful in reducing the incidence and prevalence
of those maladies.
Planning Process at the Tufts
Cummings Veterinary School
One year + planning process
involving:
•Educational Dean
•Chair of Rotation Directors
•All Rotation Directors
•Support Workshops provided by
the Tufts Center for Teaching and
Learning
Part 1
The Tufts Cummings Veterinary
School Experience





Start with Competency Statements of
accrediting body – AVMA
Localize to School Wide graduating
competencies for clinical rotations
Workshops on CBE and competency
writing
Build rotation/clerkship Competencies
Produce Grading Rubrics
Vet School Experience




Match each rotation back to the school wide
competencies
All competencies come back to curriculum
committee for approval
Informatics had not been identified by any
rotation – curriculum committee added these
competencies
Curriculum committee added the preclinical
competencies
Helpful Tips






Provide a good model
Provide support – workshops
Do not insist on uniformity
Make sure grading rubrics match back
to competencies
Identify it as a work in progress
Hard part was getting the attention of
the rotation directors
Helpful Tips


One person brings it all together
Work with external bodies – curriculum
committee to provide overview
How to write Competencies





Understand the level (school-wide,
course, content)
Action Words
Verbs followed by nouns
Measureable and/or observable
Performance based
How to write….




Consider relationships up and down the
framework
Do not use evaluative words in the
competency (this will come later)
Who owns the competency? Who can
create? Who can modify?
Feel free to use terms like ‘outcomes’,
learning objectives
Part 2

Assessment and grading rubrics
How rubrics help:

For Courses:



Rubrics enable faculty to clearly communicate expectations for student performance to
students. They support more consistent and objective assessment of student work.
They also facilitate detailed feedback to students. When areas of strength and
weakness in performance are noted, students more easily comprehend the reasoning
behind their professor’s assessment of their work.
A rubric is a scoring guide that seeks to evaluate a student's performance based on
the sum of a full range of criteria rather than a single numerical score
.
For Student Outcomes Assessment:

Rubrics facilitate quality conversations among faculty about student learning, shorten
the time it takes to do SOA, and contribute to the validity and reliability of the
assessment process.
- Stevens and Levi, Ch. 2
www.uni.edu/chfasoa/IntroductiontoRubrics.ppt
Developing a Rubric cont.

Scale
How well or poorly has the student done on the
task?





Sophisticated, competent, partly competent, not yet competent.
Exemplary, proficient, marginal, unacceptable.
Advanced, Intermediate, Novice.
Distinguished, proficient, intermediate, novice.
Accomplished, developing, beginning. (Huba and Freed, 2000)
Note: Consider whether a scale needs more than three levels. Some
research indicates that information about student learning obtained
from a three-level scale is comparable to that obtained from a fivelevel scale. More levels typically means more time spent on
assessment.
Developing a Rubric cont.

Dimensions



Dimensions break down a task into components
and identify the importance of these
components.
Dimensions are descriptive, not evaluative (e.g.,
“organization” not “good organization”).
Dimensions help students see that the work
they are doing is multidimensional and draws on
multiple abilities.
Developing a Rubric cont.

Description of the Dimensions




Rubrics should contain at least a description of the
highest level of performance.
With experience and in response to the complexity of the
performance, descriptions of all levels can be written.
Students need not fit cleanly into a single category. On
oral presentation skills, a student might speak in a clear
voice but lack eye contact.
Descriptions for each level help students see that the
work they are doing does have varying levels of
achievement. There is a difference between poor and
excellent work.
Examples of Grading Rubrics

handout
Competency AssessmentTUSK mock-ups
Planning Process
After the competencies and
grading rubrics TUSK had a series
of meetings with :
•Educational Dean
•Chair of Rotation Directors
•All Rotation Directors
•Iterative process of
moving current process to
Online version integrated
with TUSK
• Create Powerpoint ”story
board” which developer
used as functional spec
Process to get to competency
assessment in TUSK

TUSK meetings with vet school leaders






Understand business processes
Translate these to the online world
Develop powerpoint story board to achieve
agreement of functional spec
Iterate over 7 months
Check in with all rotation directors
Test and train


Pilot test in two clerkships
Release to all clerkships
View the Process through Every
Lens

Course Administrators




Faculty Assessors






See each assessor’s view as well as summary
Able to add comments
Able to override scores
Student

View the results
Compilation of all clerkship results

Access final grade (allow resubmit)


Able to select/deselect students
See picture of student
Easy access to form
Course Director


Create form
Link to competencies
Link Faculty to students
Registrar
Competency Assessment
Features









Prepare TUSK to accept competencies
Link to School-wide/course competencies
Configurable form for each course
Multiple assessors
Access to Registrar
Show scores or only feedback
Multiple rubrics per form
Override score/grade
Student retains access to scores over time
Overview
Admin View
I.
a)
b)
c)
Director View (con’t)
III.
Populate Competency Data
Populate Competency
Assessment Form & Data
Other Assessment Form Data
c)
d)
e)
Faculty View
II.
a)
b)
c)
Access to the Assessment IV.
Selecting Additional Students
Assessment Form
Individual Student Results
Completed Faculty/Staff
Form
Summary Form
Student View
a)
Accessing Assessment
Results
Director View
III.
a)
b)
Assessment Results for All
Students
Completion Report
Registrar View
V.
IV.
View final grades
I. Administrator View
Creating and Populating
Competency
Assessment Data
Done at the School Administrator level
POPULATE COMPETENCY
DATA
Step 1: Create Competency Category
Modify Competency Category
Add Sub-Competency Levels (if applicable)
Sort Competencies
Sort Child Competencies
Done at the Course Admin level
POPULATE COMPETENCY
ASSESSMENT FORM DATA
Step 2: Create Assessment Form(s)
Note: The “View Reports” button is only visible to the course director
“Balance Weights” still needs to be further discussed/clarified
Populate Assessment Form Metadata
Grading is numeric. There
will be an option in the
grade book to give an
equivalent letter or pass/fail
grade
Scoring Range defaults to
‘No’
Score Display:
• Score & Feedback
• Feedback Only
Scoring Value:
• Max value = 25
Frequency:
• Max No. = 10
Step 3: Populate Fields and Data

Question type:







Scaling
Scaling w/Sub-question
Single Select
Single Select w/Sub-question
Multi-level questions (options with subquestions)
Link competencies to questions
Free-text feedback option
Add Field: Scaling
Make text boxes bigger
Linking Competencies
Scaling w/ Sub-Questions
Add Field: Single Select
Single Select w/ Sub-level Questions
Done at the Course Admin level
OTHER COMPETENCY
ASSESSMENT FORM DATA
Considerations





Display student images?
Display elective information?
Display summary comments
Multiple assessors?
Who selects assessors?
Considerations?





Use Numeric scores?
Display scores to students or just
comments?
Display scores to assessors?
360 degress review?
What else?
Link Faculty/Student
II. Faculty View
Getting to the form
Accessing the Assessment(s)
Accessing the Form
Evaluator: J. Berg
Assessment Form – Scaling
Assessment Form – Single Select
III. Director View
Viewing assessment
results
Assessment Results – All Students
Assessment Results – Individual Student
Note: To view a form completed by a
faculty/staff click their name
Note: When ready, click here to fill out a final
assessment form for the student
Completed Form by Faculty/Staff
Summary Form
Click to view student’s patent logs
in a pop-up window
Hover over score to
see actual individual
scores
This is at the bottom of the form
Student’s Patient Log Summary
This opens up as a pop-up window
Completion Report
Send Email Reminders
IV. Student View
Access via the Grade
Book
Accessing Assessment Form
Note: Link to view final assessment form
completed by the course director
Accessing Assessment Form
J. Berg happens to also be the course director
V. Registrar View
View Final Grades
Grades by Course
Step 1: Select the course
Grades by Course
Step 2: Select the time period
Part 3

Building the Framework
First steps




A set of clear definitions
Use cases
A survey of Canadian, US and UK health
professional education organizations
(24 respondents)
See definitions_and_usecases.pdf for
details (medbiquitous.org)
Comparison of published
frameworks












Accreditation Council on Graduate Medical Education (ACGME)
Competencies8
Acute Care Nurse Practitioner Competencies
American Association of Critical-Care Nurses (AACN) General Patient
Care Competencies
American Society of Health-System Pharmacists
CanMEDS 20059
CanMEDS Specialty Competencies
Good Medical Practice (UK)
Good Medical Practice (USA)
The Scottish Doctor7
Tomorrow's Doctors
Tuning (MEDINE)
Women’s Healthcare Competencies
DEVELOPMENT OF A
CONCEPTUAL MODEL
A competency object . . .
Outcomes for
Clinical Skills
Relates to other
competency objects . . .
Which can relate to
external resources . . .
Learning
Object
Assessment
Performance
data
A competency object . . .
Current Specifications
Outcomes for
Clinical Skills
Relates to other
competency objects . . .
Which can relate to
external resources . . .
Learning
Object
Assessment
Performance
data
Specs and Schemas Available
At:


http://www.medbiq.org/working_group
s/competencies/index.html
Open license, membership not required
Competencies –
individual
competency
definition
A standard format for a Competency Object
single competency
definition, including an
identifier, categories
and references.
Competencies –
an interrelated
set of
competencies
A framework for
establishing the
relationships among
competencies in a
single framework
Competency
Framework
Competencies –
evidence of
achievement
A format for
documenting
achievement of a
milestone or
competency.
Educational
Achievement
Competencies –
cross-mapping
A format for mapping
competencies from one
framework to
competencies from
another framework.
Working draft
Being
implemented by
Tufts
May be
implemented by
VA
Working draft
Being
implemented by
Tufts
May be
implemented by
VA
Work to begin in
July 2010
Informal proposal
from competency
working group
(planned work)
Essential for tying
curricula to competencies
and enabling competency
based education.
Essential for integrating
existing competency sets
into curriculum
management systems.
This will be important for
schools tracking
competence in a portfolio
and for residency
programs tracking
competence.
Should we arrive at the
point of having a national
competency framework,
this would be important
for schools that want to
see how local
competencies fit into the
national picture.
Spec needs Analysis of the Problem
from all perspectives

Student View




Faculty view





How is student doing
Grade/Rate/Comment on student
View proof of learning
See full range of expected learning outcomes
Registrar


How am I doing?
What do I have to do?
How can I show my learning
Track grades
Admin View



What are the competencies
Where are competencies taught - content/courses/themes/rotations/clinics
Where are the gaps
Need tools to Enter, Store, view, Grade
Medbiquitous Competency Object
Standard
Copyright Medbiquitous consortium 2010 All rights reserved May 20 2010 p.11
What Info Needs to be
brought Together





School wide competencies
Course level competencies/learning
objectives
Content addressing competencies and
learning objectives
Element of time – when and how much
How are competencies assessed
Where does TUSK get the
Data




Schedule
Content linked to schedule (upon
upload)
Curriculum hours pulled from schedule
Keywords linked to content
Competencies
How are they taught?
External
Competency
Framework
Course
competencies
Course
competencies
Content
repository
w/
metadata
School Wide
Competencies:
Competency
Framework
Content
competencie
s
Where are they
assessed?
When?
(Links to
schedule)
Where are
they taught ?
PBL
lecture
s
Competency
Assessment
rotation
s
skills
knowledge
attitudes
Schedule – Curriculum View
First year courses
Course Summary view
Methods to Display Complex Data
complex data displays
References





Medical Teacher: Outcomes-Based Education Themed Issue Vol 29
Number 7 September 2007
Lessons Learned from the Competency-Based Curriculum Initiative at
Baylor College of Dentistry. McCann, Ann L.; Babler, William J.; Cohen,
Peter A., Journal of Dental Education, v62 n2 p197-207 Feb 1998
The new formal competency based curriculum at Indiana University
School of Medicine, Litzleman and Cottingham, Academic Medicine Vol
82, No. 4 April 2007
An educational blueprint for the Brown Medical School, Smith MD
Learning objectives for medical student education--guidelines for
medical schools: report I of the Medical School Objectives Project.
AAMC Acad Med. 1999 Jan;74(1):13-8.