Student Evaluation of Teaching
Past, Present, and Future
Donghui Zhang, Liaoning University
melinda_shy@163.com
Yuxiang Liu,
St. John’s University
liuy@stjohns.edu
Questions about Student Evaluation of Teaching (SET)
A. Is SET a valid method to measure
teaching effectiveness?
B. What instruments have been available
for such measurements?
C. What are the SET policies and
procedures?
D. What issues that SET is facing today and
what’s ahead for SET?
2
Methods Used to Seek Answers to the Questions
A.Detailed literature review
B.SET survey of 500 colleges
C.Analysis of SET policies and
procedures collected from 50 colleges
3
Literature Review: Purposes of SET
Purposes of SET :
A. The formative function, i.e., SET used
for improving instruction.
B. The summative function, i.e., SET
used for personnel decisions.
C. Guide students in their selection of
courses and instruction.
4
Literature Review: Stages 1 - 4
In 1993, Centra summarized SET development
into the following four historical stages:
1. Late 1920s to 1950s, Remmers and his colleagues at Purdue
University did their pioneer work.
2. The 1960s witnessed faculty’s voluntary use of SET for
improvement purposes.
3. The 1970s was regarded as the golden age of research on SET.
Studies were designed to answer such questions as: 1) Does SET
really measure teaching effectiveness? and 2) Can SET help
improve teaching?
4. The1980s to early 1990s, the continuing refinement of research
findings and a series of reviews and meta-analyses
substantiated findings on important issues.
5
Literature Review: Stage 5 of SET
The next stage, from mid-1990s to the present, can be
labeled as the expanded applications of technology
and the transition from paper to the online mode of
SET.
A. In 2000, 2% of colleges used the institution-wide online SET
(survey of 200 most wired institutions, by Hmieleski).
B. In 2002, increased to 10% (survey of 256 colleges, by Hoffman).
C. In 2005, up to 15% (survey of 100 colleges, by the Faculty
Center at the Brigham Young University).
D. In 2010, up to 27% (survey of 296 colleges, by Liu).
6
Literature Review: Instruments
Some widely-used SET instruments include:
A.
B.
C.
D.
E.
F.
G.
H.
Instructional Assessment System (IAS), University of Washington, Seattle.
Instructor and Course Evaluation (ICE), Southern Illinois University.
Instructor and Course Evaluation System (ICES), University of Illinois, Urbana.
Instructional Development and Effectiveness Assessment (IDEA) System,
Kansas State University.
Purdue’s Cafeteria System, Purdue University.
Students’ Evaluation of Educational Quality (SEEQ), University of Western
Sydney.
Student Instructional Rating System (SIRS), Michigan State University.
Student Instructional Report (SIR, the revised version called SIR II), the
Educational Testing Services (ETS).
7
Literature Review: Dimensions of SET
Some common dimensions in SET instruments:
A. Organization/Clarity
B. Instructor enthusiasm
C. Group interaction
D.Individual rapport
E. Breadth of coverage
F. Examinations/Grading
G.Assignments/Readings
H.Overall rating
8
SET Policies and Procedures
A. All SET results should be anonymous.
B. The SET system should ensure that each individual
student can fill out the form only once.
C. The instructor shouldn’t be present in the classroom
or site when students fill out the form.
D. SET should end before grades are disclosed to
students.
E. SET results shouldn’t be disclosed to instructors
before grades are submitted.
F. Student comments on the paper form should be
typed.
9
Issues Facing SET Today
Issues facing both online and paper-form SET:
• How can the SET data be effectively used to improve
teaching and learning?
Issues facing paper-form SET:
• It takes too much time and resources for data
collection and report.
• If student comments are scanned into image files, the
anonymity cannot be guaranteed.
Issues facing online course evaluation
• The response rate is low.
10
Survey Results: Response Rates for Online SET
• 100% if SET is mandatory.
• Around 80% if a link is set up between course
grades and online SET: If a student has not filled
out the online SET form, the system
automatically leads him to the online SET when
he checks his grade; he has the option to
decline.
• 40 – 60% if four or more reminders are used.
• 20 – 45% if three or fewer reminders are used.
11
Available SET Software Packages
A. Blue/Evaluation by eXplorance:
http://www.explorance.com/prod_evaluation.asp
B. Class Climate by Scantron: http://www.scantron.com/classclimate/
C. Digital Measures:
http://www.digitalmeasures.com/CourseResponse/index.htm
D. EvaluationKit: http://www.evaluationkit.com/
E. Online Course Evaluations by Gap Technologies:
http://service.onlinecourseevaluations.com/
F. Remark Web Survey by Gravic Inc.:
http://www.gravic.com/remark/websurvey/
G. SIR II by ETS: http://www.ets.org/
H. SurveyDig by Runner Technologies:
http://www.runnertech.com/surveydig_home.html
12
Questions and Discussion
THANK YOU!
13