E l i Committee
Evaluation
i
Report
p
to the Faculty Senate
September 13,
13 2011
Points of agreement between the Ad Hoc
and
dE
Evaluation
l ti C
Committees
itt
• Both committees seek:
– The use of some alternative to the current
reporting of “department, school, or university
averages, quintiles, and categorical statements
such as average
average, above average
average, etc.
etc ”
– The use of department‐ or program‐specific
questions on our student evaluations of teaching
effectiveness.
ff ti
This
Thi practice
ti iis already
l d possible
ibl and
d
in use by some programs, but more widespread
adoption could improve SPOT usefulness across
campus;
– The use of multiple measures of teaching
effectiveness, of which SPOTs should be just one of
many considerations
id ti
for
f reappointment,
i t
t tenure,
t
and promotion decisions;
More points of agreement
• Both committees affirm:
– The rights of faculty members to comment
upon and contextualize their SPOT results in
their RTP materials;
– The need for greater uniformity in peer
evaluation across the university;
– The potential value in inviting representatives
of respected private student evaluation
vendors to campus to show us what they
offer.
Comparison of Q16 results across departments, from Spot 2006
This dataset shows 29
departments, 6 of which
have zero responses: ACG,
BUS, GCA, HON, ISO and
MBA.
XX
What do we observe?
1. Most of BIO’s are 6 & 7
ratings.
2. AAT has only 4 and
above.
3. Over 50% of IDS’ are 7.
4. MAT has one of the
most flat distributions.
5. Dept X has very few 6
& 7.
6. HST, GRA, and FST
have no 1s.
By Yishi Wang, Department of
Mathematics and Statistics
Plotting Q16 by department
X
By Yishi Wang, Department of
Mathematics and Statistics
An individual instructor’s summary results, questions 1‐15
By Jim Blum, Department of
Mathematics and Statistics
Sample department‐wide summary using 2006 SPOT results
By Jim Blum, Department of
Mathematics and Statistics
A view of multiple sections of the same course
By Jim Blum, Department of
Mathematics and Statistics
A depiction of UNCW’s current SPOT process
By Kevin
Violette, ITSD
Recommendation 1: Eliminate the global question (Q16 from
UNCW SPOTS)
SPOTS).
• The Evaluation Committee has conducted
quantitative analyses of large volumes of SPOT
data in 1995 and 2007 (and a UNCW statistician
has done so again in 2011). All found that the vast
majority of the fifteen SPOT items are highly
correlated with Q16.
• Without
With t a global
l b l measure, faculty
f lt and
d decision
d ii
makers must consider fifteen items simultaneously.
• Professional organizations that specialize in course
evaluations endorse the use of overall measures of
effectiveness.
Recommendation 2: For RTP, Q1 to Q15 are not reported with
any department,
department school
school, or university averages,
averages quintiles,
quintiles and
categorical statements such as above average, average, etc.
• Some benchmark is necessary for understanding
how to interpret SPOT results.
• Committee members would like to see only
disciplinary benchmarks provided, thus agreeing
somewhat
h with
i h the
h recommendation
d i in
i spirit.
ii
• UNCW statisticians have found that SPOT results
can differ sharply between disciplines, making
school and university benchmarks potentially
h
harmful
f l to
t some faculty
f lt members.
b
Recommendation 3: Q1 to Q15 be reported for RTP only as a
comparison
i
over time
ti
for
f that
th t particular
ti l iinstructor.
t t
• Comparisons
p
over time are p
potentiallyy useful,, and we
may be able to create new ways of displaying SPOT
results that will enable such comparisons more easily,
but limiting results to this form would create an
unmanageable task for decision makers.
p
over time could create incentives for
• Comparisons
attempts to manipulate performance, and may show
trends difficult for faculty members to explain.
• Without the benchmarks discarded in recommendation
#2, comparisons over time would be especially difficult
to manage.
Recommendation 4: That some questions on the SETEs be
t il d to
tailored
t specific
ifi departmental
d
t
t l missions
i i
and
d expectations.
t ti
• Faculty members and departments already
have the option of adding items to the SPOT
form.
form
• The MBA program, for one example, currently
takes advantage of this possibility
possibility.
Recommendation 5: Quantified SETEs (UNCW) cannot be
weighted more than 25% to 33% as an assessment for teaching
effectiveness for purposes of RTP.
• Evaluation Committee members agree
g
that student
evaluations of teaching effectiveness should indeed be
one of multiple measures of many aspects of teaching.
• This
Thi recommendation
d ti presumes a fformula,
l h
however,
the remaining components of which are unclear.
• We resist assigning numerical weights to any single
component of the RTP documents, so that the RTP
committee may exercise its own judgment as to the
relative importance of each component regarding each
individual candidate.
Recommendation 6: That the qualitative/written comments from
student evaluations be more systematically administered and reported in
order for them to be used more effectively in RTP decision.
• Without a statistician or ethnographer to
handle the tremendous amount of
information this would deliver to RTP for each
candidate, this recommendation would create
an avalanche of additional material for no sure
benefit.
• Even with trained help
help, the interpretation of
qualitative data is subjective.
Recommendation 7: That the individual faculty member has a right to formally
comment, explain, or respond to student evaluations (either quantitative or
qualitative/written comments) for purposes of RTP
RTP, and that these comments
comments,
explanations, or responses be formally included as part of the SETE reporting process.
• The Evaluation Committee agrees
g
with the spirit
p
of this recommendation but does not see the
need for any new policy to this effect.
• The UNCW Faculty Handbook
Handbook, Section IV
IV.A.
A lists
the materials that tenure and promotion
candidates may include in their files. Item 3
includes this sentence, among others, “In
addition, the candidate may provide: additional
SPOT reports received prior to the previous two
two‐
and‐one‐half years, and/or his or her own
qualitative interpretation of SPOT results.”
Recommendation 8: That the process of peer evaluation of
teaching be more uniform between departments and schools
within UNCW in order for them to be used more effectively in RTP
decisions.
• The Evaluation Committee agrees with this
recommendation and hopes to examine this
issue (which has been approached in the past)
as soon as the SPOT issue is resolved.
• We hope to approach this issue alongside the
RTP committee so that our recommendations
will be in concert with that committee’s
practical needs.
Recommendation 9: That peer evaluations be weighted at least
equal to student evaluation of teaching (quantified question and
student written comment section) for purposes of RTP.
• Th
The Evaluation
E l ti C
Committee
itt agrees with
ith th
the spirit
i it off thi
this
recommendation, but rejects its reliance upon the
concept
p of weighting
g
g measures of teaching.
g
• We endorse, instead, multiple measures of teaching,
with peer evaluation viewed as at least as significant as
student
t d t evaluations,
l ti
alongside
l
id other
th iindicators
di t off
teaching effectiveness.
• The UNCW Faculty Handbook states, “it
it is strongly
suggested that peer and student evaluations be given
similar emphasis in personnel recommendations.”
Recommendation 10: That UNCW investigate the IDEA Center’s evaluation system for
student evaluation. However, no global question should be used, and any qualified
SETE process should never be weighted more than 25 to 35% as an indicator of faculty
teaching effectiveness.
• The Evaluation Committee accepts the first sentence of
this recommendation and rejects the second sentence.
• We propose that we invite the IDEA Center and IOTA
Solutions to campus for informational forums.
• If we trust the IDEA Center’s extensive research, then
we trust their conclusion that global questions are
valid.
valid
• We again reject the weighting of specific measures of
teaching. We have two measures weighted in these
recommendations,
d i
accounting
i for
f 50 to 66% off the
h
teaching portion of RTP’s assessment, but we have no
indication of what the other 34 to 50% of the
assessment might
i h entail.
il
The Evaluation Committee’s recommendations
Recommendation 1: For RTP purposes, SPOT results
should be reported
p
with onlyy disciplinary
p
y benchmarks.
Recommendation 2: In the 2011‐12 academic year, the
Faculty Senate Evaluation Committee should be
charged
h
d with
ith bringing
b i i greater
t uniformity
if
it to
t the
th process
of peer evaluation of teaching throughout the
university.
Recommendation 3: In the fall semester of 2011, IDEA
Center and IOTA Solutions representatives should be
invited to campus to meet with members of the Faculty
Senate so that we may consider the potential benefits
and drawbacks of moving to an off‐site evaluation
provider.
Definitions of Information Highway, Remote, and Web‐
E h
Enhanced
d Courses
C
(f
(from UNCW R
Registrar’s
i t ’ website)
b it )
•
•
Instructional Method Codes
(bl k) ‐ In‐person.
(blank)
I
F
Face‐to‐face
t f
instruction
i t ti (real
( l time
ti
and
d reall people
l with
ith
instructor in the same room as the students).
•
WEBE ‐ Web‐based with some in‐person class meetings
•
OLASY ‐ Fully Online – asynchronous. Internet or Web‐based asynchronous
(instructional materials made available on the Internet and available 24 hours a
day every day—not time‐dependent slots).
OLSYN ‐ Fully Online – synchronous.
synchronous Internet or web
web‐based
based SYNCHRONOUS
(instruction made available on the web or internet in time‐dependent slots).
REMOT ‐ Non‐web based at remote site. Television, videocassette, CD, DVD, etc.
(non‐Web‐based instructional materials distributed to the student at a remote
site).
site)
INHWY ‐ Interactive real‐time video (e.g., NCREN / NC Information Superhighway).
Some members of the class at a remote studio with monitors used to convey two
way audio and video signals
•
•
•