C.E.L.T
Developing a Comprehensive Faculty
Evaluation System
Jeff Schlicht, Ph.D.
Associate Professor
Department of Health Promotion and Exercise Sciences
schlichtj@wcsu.edu
Measurement vs. Evaluation
Measurement… is the process of systematically
assigning numbers to the individual members of a
set of objects or persons for the purpose of
indicating differences among them in the degree to
which they possess the characteristics being
measured.
Robert L. Ebel, “Measuring Educational
Achievement”, 1965
Measurement vs. Evaluation
Evaluation is the process of interpreting a
measurement by comparing it to a value (or
set of values) that represent an agreed upon
standard. This process involves judgment.
A Comprehensive System =
Measurement and Evaluation
In order to develop a comprehensive faculty
evaluation system, we must:
1) Develop technically good (valid &
reliable) measurement instruments
2) Agree upon a university standard for
desirable faculty performance
How Do We Currently Measure
Faculty Teaching Performance?
In the HPX department we use:
1) Student rating forms
2) Peer observation (in classroom)
Are either of these forms of measurement valid
and/or reliable?
The answer is 1) we don’t know and 2) no
Student Ratings of Faculty Performance
The development of a valid and reliable
questionnaire, i.e. a student rating form, is a
technical process that has a body of science
behind it.
Our department does not know who created
the current rating form, and whether it has
ever been tested for validity and reliability.
Student Ratings of Faculty Performance
Dos and Don’ts
DO include a balanced response scale
• equal number of positive and negative responses
• mirror-image opposites
– Strongly agree agree disagree strongly disagree
DO develop items (questions) important to faculty AND
students
DO conduct field tests of the items ( = collect data) to do
reliability and validity analyses
DO develop norms (a year’s worth of data) to allow fair
and appropriate interpretation (i.e. comparison)
DO consider using a professionally designed student
rating form
Student Ratings of Faculty Performance
Dos and Don’ts
DON’T
• Have two items in one question (doublebarreled response item)
• Simply hand back a summary sheet of student
ratings to instructor – include personal
consultation to foster instructional improvement
• Consider individual student responses in faculty
evaluation
Peer Observation
The short answer: DON’T do it for personnel decisions
(can be useful for faculty enrichment)
The addition of another faculty member or administrator
into the classroom ALWAYS alters the classroom
dynamic in such a way as to provide an abnormal view
of what usually happens there
THOSE DATA ARE NEVER VALID
Peer Observation
If you must do it for faculty evaluation consider:
• multiple visits by multiple (4) peers
– 8 visits = adequate sampling
– develop valid, reliable checklist
• train observer team in use
– prepare students, instructor
– schedule post-observation conference
• videography
8 Steps to a Comprehensive Faculty
Evaluation System
Step 1: Define the faculty role model
• What should be evaluated?
–
Start at the department level
•
•
Have faculty write down everything they do at WCSU
to create a master list of responsibilities
Determine faculty roles
– What are the things we are accountable for?
•
Assign the items from the master list of faculty
responsibilities to the various faculty roles
8 Steps to a Comprehensive Faculty
Evaluation System
Step 2: Determine faculty role model parameters on a
University (system?) level
• How important should each role be?
–
Static vs. dynamic role model
•
•
Static = fixed weights
Dynamic = range of weights
This is missing from our current system, and therefore we
do not control subjectivity as well as we could
8 Steps to a Comprehensive Faculty
Evaluation System
Step 2: Determine faculty role model parameters
on a University (system?) level
• Static Role Model
–
–
–
–
Load Credit = 50%
Creative Activity = 25%
Productive Service = 15%
Professional Service = 10%
8 Steps to a Comprehensive Faculty
Evaluation System
Step 2: Determine faculty role model parameters
on a school, then University (system?), level
• Dynamic Role Model
–
–
–
–
Load Credit
Creative Activity
Productive Service
Professional Service
Minimum
Maximum
35%
20%
5%
0%
75%
50%
45%
40%
8 Steps to a Comprehensive Faculty
Evaluation System
Step 2: Determine faculty role model parameters
on University (system?) level
• Dynamic Role Model
–
The advantage of a dynamic role model is that it
allows various levels of the University to play to
their strengths
•
Allow each department to set dynamic range values,
then select extreme max. and min. as University
standards
8 Steps to a Comprehensive Faculty
Evaluation System
Step 3: Use observable or otherwise documentable
achievements/products/performances to define all the
roles in the faculty role model
• Teaching (student interactions that promote learning)
–
–
–
–
–
Content appropriateness/currency
Instructional design skills
Instructional delivery skills
Instructional assessment skills
Course management skills
8 Steps to a Comprehensive Faculty
Evaluation System
Step 4: Define weights for each component of a single
role – departments should be allowed to have their
own weights, as long as they fit into the University
structure. This embeds department values into the
evaluation design:
•
Teaching
–
–
–
–
–
Content appropriateness/currency
Instructional design skills
Instructional delivery skills
Instructional assessment skills
Course management skills
min
max
10%
10%
20%
20%
0%
40%
30%
60%
50%
30%
8 Steps to a Comprehensive Faculty
Evaluation System
Step 5: Identify where you will collect the data to evaluate the roles
•
•
Use sources that have first hand experience with the performance in
question – preferably multiple sources of input
Teaching
–
Content appropriateness/currency
•
•
–
Instructional delivery skills
•
•
•
–
syllabus review by peer
Self-review
Student evaluation
Peer evaluation???
Self-review
Course management skills
•
•
Chair or administrative assistant evaluation
Self-review
8 Steps to a Comprehensive Faculty
Evaluation System
Step 6: Define weighting for information provided
by different sources
• Teaching
min
max
–
–
–
–
Students
Self
Peer
Chair
15%
15%
15%
15%
50%
40%
30%
30%
8 Steps to a Comprehensive Faculty
Evaluation System
Step 7: Determine how to gather information
• Questionnaire
• Interview
• Checklist
8 Steps to a Comprehensive Faculty
Evaluation System
Step 8: Design data collection instruments
Evaluation for Promotion and Tenure
Evaluation is the process of interpreting a measurement by comparing it to
a value (or set of values) that represent an agreed upon standard. This
process involves judgment.
In a system like the one described today,
evaluation is an on-going process involving
multiple sources and evaluators. We are
constantly comparing ourselves against the
WCSU Faculty Role Model, so we have
immediate feedback about our performance.
Evaluation for Promotion and Tenure
Promotion and tenure decisions can be based
on comparing our performance against a
pre-determined, minimum required
composite Faculty Role Model score. In
other words, prior to becoming eligible for
advancement, we would already know the
outcome.
Comprehensive Faculty Evaluation
Practical steps we can take to improve the system:
• Initiate a discussion about what it means to be a
faculty member at WCSU in the 21st century
• Discover new ways to measure faculty performance,
tests those ways to make sure they are valid and
reliable, and create a feedback loop that uses that
information to enhance faculty performance
• Lobby the union and the University to more actively
support faculty enrichment
Final Thoughts About Faculty Evaluation
Why should we do it?
• Promotion and tenure is important, but…
the true value of assessment lies in its ability
to direct faculty enrichment. If evaluation is
not tied to enhancing faculty performance, it
will be perceived as punitive, a hurdle, rather
than as a path toward self-improvement.
Acknowledgements
These materials are summarized from the ideas of
Raoul A. Arreola, University of Tennessee
Health Science Center, presented at the Center
for Educational Development and Assessment
professional enrichment seminar in Orlando, FL.
March 10-11, 2008, and available in the text:
“Developing a Comprehensive Faculty Evaluation System,”
3rd edition, Arreola RA, Anker Publishing Co., Bolton, MA 2007
C.E.L.T
Developing a Comprehensive Faculty
Evaluation System
Jeff Schlicht, Ph.D.
Associate Professor
Department of Health Promotion and Exercise Sciences
schlichtj@wcsu.edu