Students’ Evaluation of
Educational Quality (SEEQ)
Dr. Mary Benbow
Associate Professor,Department of Environment and Geography
University of Manitoba
Resources to Examine Evaluations
• My web site
• http://home.cc.umanitoba.ca/
~benbow/seeq.html
• Centre for
Educational
Advancement at
Curtin University,
Australia.
• “Students Rating
Teaching” by Mark
Lawall
• Many, many peerreviewed research
articles
– A selection is
provided in the
bibliography
Using the SEEQ
• Student evaluations – what we know
• Student Evaluation of Educational Quality (SEEQ)
• Using the SEEQ
Student Evaluation:
What We Know
• A lot of research (some apparently contradictory
and/or inconsistent)
• Difficult to compare different instruments
• Many useful reviews:
– Students Rating Teaching
– Other articles (e.g. Ory, 2001)
Development and Use of Student
Evaluations
• Introduced in the 1960s
– Students sought “a stronger voice”
– Today viewed as one component of accountability
• Over 90% of institutions of higher education use
student evaluations of some sort
• Comprehensive scheme for student evaluations
began at the U of M in the academic year 1996-97
(Policy 425)
– September 2002, Senate approved voluntary posting of
SEEQ results on U of M website
Courses: Requirements, Levels and Sizes
• Electives tend to get
higher ratings than
required course
– majors more favourable
than minors
• Higher level courses tend
to get higher ratings
• There are differences
between the disciplines
• Class size has little effect
overall
Instructors and Students
• Student ratings positively correlated to those of
alumni
– positive correlation between students and colleagues
• Lower ratings for less experienced instructors
and teaching assistants
– relationship between research productivity and ratings
is either positive or nil
• Higher ratings where there is prior interest
• Slightly higher ratings where gender of instructor
and students is the same
Administration and Use
• Diagnostic not prescriptive
– Need to follow uniform
procedures
– higher evaluations if followed
by a short speech
– higher also if not anonymous
– higher if the instructor stays in
the room
• Most important: not the only
source of information
concerning teaching
Student Evaluation of
Educational Quality (SEEQ)
• Widely researched
– very high degree of reliability (r = 0.88 - 0.97)
• Based on psychometric analysis (teaching not
content)
– correlate well with a wide range of measures of learning
outcome
– correlates well with instructors’ self ratings
The development of the
SEEQ
• Developed Herbert Marsh of the
University of Western Sydney in
late 1970s
– Internationally recognised expert in
psychometrics
• SEEQ has been exhaustively
researched
– Statistical tests over 13 years, from
approximately 50,000 courses and
almost 1 million students
SEEQ Factors
Questions developed from interviews with faculty and students
about what constitutes effective teaching
•
•
•
•
•
Learning
Organization
Individual Rapport
Breadth
Assignments
•
•
•
•
Enthusiasm
Group Interaction
Examinations
Overall Ratings
Teaching as a multi-dimensional activity
SEEQ Questions
• Formative questions:
– Identifies specific aspects of each factor
– Organization:
– Instructor’s explanations were clear
• Summative overall questions:
– 30. Compared with other courses I have had at the U. of M., I would
say this course is:
– 31. Compared with other instructors I have had at the U. of M., I
would say this instructor is:
– 32. As an overall rating, I would say this instructor is:
Student Evaluation of
Educational Quality:
Using the SEEQ
• Little research on summative use
• Ratings are a reliable source of data
• Data must then be examined and evaluated
– Examine the data to create assessment
– Followed by decisions and examination of change
Using the SEEQ
•
•
•
•
•
Understanding your priorities
Undertake a self-evaluation of teaching
Looking at your evaluations
Identify solutions
Making changes and smart decisions
Your Priorities
• Related to your Teaching Philosophy
– Describes the principles that direct your
teaching style and activities
• Identify which factors and elements
are most important to you
– Are your priorities consistent those of
your discipline, department or institution?
– Can you establish a reason for your
priorities
• Prior to examining your evaluations,
it is important to assess which matter
most to you
Self-Rating of Teaching
• The SEEQ correlates well with self ratings
and trained observer ratings
• Useful to consider our assessment of
teaching
– Collect evidence of teaching and assemble in a
teaching dossier or portfolio
– Consider our assessment using a self-rating form
based on the SEEQ
– “Most of us await student reactions to our courses
with both eagerness and trepidation” (Delucchi
and Pelowski, 2000)
Strongly Disagree Neutral
Disagree
1. I have found
the course
intellectually
challenging and
stimulating
2. I have
learned
something
which I
consider
valuable
3. My interest in
the subject has
increased as a
consequence of
this course
4. I have
learned and
understood the
subject
materials of this
course
Agree




Strongly
Agree




Identify the
most common
responses you
expect from
your class
Self-Analysis:
Looking at your course evaluations
• Are there any surprising
responses?
• Which surprises are
most relevant?
• Do the written
comments reveal any
more information?
• You can also use your
statistical summaries of
evaluation data (but
differences of <0.05 are
not considered
significant)
Analysis framework
• What special features reflect the subject, the
students or other contextual factors?
• Eliminate any factors that are not appropriate to
your intentions in teaching this class.
– What are the main strengths of your teaching?
– Which are the main weaknesses in your teaching?
• How do these findings differ from your Self-Rating
Survey
Other Useful Data on the SEEQ
• Students Enrolled and Students Responding can
indicate attendance (e.g. 62%)
• Expected Grade can be compared to the actual
grade distributions
• Reason for taking course can reveal how many
are required to take a course
“There’s always
one”
!?
• Look at the response
sheets and identify which
responses are the outliers
• Look at their written
comments
• Look at the Student and
Course Characteristics
• What should we do
about that one or two
reponse(s) that stand
out, perhaps
anomalously?
• Are the outliers only in one
question or subject area?
(e.g. use of humour)
• Are they relatable (Level of
interest (37) & Learning
questions)?
Using the SEEQ:
Making changes
• Using your priorities and self rating decide where
changes would be most useful
• Need to identify sources of information
– Choose information sources to suit your teaching and
learning style
– Start small (quick, easy, cheap) before investing in large,
long-term changes
Select key areas for improvement
• Which are the two or three areas that
received the lowest ratings?
– What strategies for improvement could
you use that would be appropriate to your
particular situation?
• Could the Tips to Improve Academic
Teaching help you in this?
• Would further personal guidance or
investigation be needed to help you
improve your knowledge and skills?
Making Changes
• Start small before investing in long-term
changes
• Why go it alone?
– Could there be some changes that could be
accomplished by group action?
• It takes time to create a new habit
– When you try something new, it may feel
uncomfortable at first
– Give yourself reminders to try some new techniques
or ideas
A good place to start are the
“Tips to Improve Academic
Teaching” organized in 8
sections referring to each factor
12. Instructor gave lectures that facilitated taking notes
Strongly
Strongly
Disagree Disagree Neutral
Agree
Agree
9
18
3
13
7
This question comprises part of the Organization
factor: choose Organization & Clarity SEEQ Factor 3
Statistical Analysis of SEEQ Results
• SEEQ data cannot be analyzed or summarized
statistically by administrators
• But, individuals can analyze their own data
• The Statistics Advisory Group recommends the
following techniques
• Can be undertaken by hand or by using a
computer spread sheet
Averages, Medians and Grouped
Questions
• Find the average response for each question
• Find the median response for each question
• Find the average for each subject grouping or
FACTOR (e.g. average response for all four
questions under “Learning”)
• 1. I have found the course intellectually
challenging and stimulating
N/A
1 Strongly Disagree
2 Disagree
3 Neutral
4 Agree
5 Strongly Agree
0
0
0
3
20
2
• Average = (0*1)+(0*2)+(3*3)+(20*4)+(2*5) =
99/25 = 3.96
• Median = 4
• 1. I have found the course
intellectually challenging and
stimulating
• 2. I have learned something which I
consider valuable
• 3. My interest in the subject has
increased as a consequence of this
course
• 4. I have learned and understood the
subject materials of this course
Av. 3.96
Med. 4.0
Av. 4.16
Med. 4.0
Av. 4.08
Med. 4.0
Av. 4.24
Med. 4.0
Learning factor: Average = 4.11, Median = 4.0
You may wish to provide summaries to indicate
improvements in your teaching
1. I have found the
course intellectually
challenging and
stimulating
2. I have learned
something which I
consider valuable
3. My interest in the
subject has
increased as a
consequence of this
course
4. I have learned and
understood the
subject materials of
this course
‘00-’01
‘01-’02
‘02-‘03
3.6
3.6
3.7
3.8
3.9
3.8
3.7
3.7
3.8
4.0
4.0
4.3
You can also graph changes in averages (in a teacher
dossier, include an explanation)
5
4.5
4
3.5
3
Q. 30
Q. 31
Q.32
2.5
2
1.5
1
0.5
0
2000-01
2001-02
2002-03
2003-04
Additional Questions in the SEEQ
• At Curtin University (Perth, Western Australia)
• Staff engaged in the 1995 pilot of SEEQ were concerned
that it might not accommodate forms of teaching and
learning other than conventional face-to-face lecturing
• Teaching staff across the disciplines have subsequently
developed factors and questions to complement the SEEQ
form.
SEEQ Questions Examples:
The computer laboratory
1. The laboratory was an
essential part of the unit.
2. Laboratory-practical
exercises reinforced the
main points of the lecture.
3. By applying theoretical
concepts in the laboratory,
my understanding of the
material improved.
4. The time and effort required
for computer-based
laboratory work was
reasonable.
Written Comments
• Type them up
– Include course title,
number, year, number of
students, etc.
– What additional depth do
they provide for the
question responses?
– e.g. if you score well in
Organization the
comments may bring up
a clear outline, good
notes, web pages, etc.
• Note: Not to be used for
tenure or promotion
Students’ Evaluation of
Educational Quality (SEEQ) :
Conclusions
• Use your priorities and self rating to decide where
changes would be most useful
• Innovations to improve teaching are everywhere
– Use the library, NetDoc or Google Scholar to find great
information and ideas
• The SEEQs are not the only source of information
about your teaching
– Develop your teaching dossier
– Develop your teaching philosophy and goals