SRI Report Recommendations
Meeting notes/Committee report
Friday 10/18/2010
Present: Eric Ball, George Donovan, Mark Potter, Kamran Sahami, David Sullivan
I.
II.
Confidence intervals. The present reporting practice is to include the number of
respondents (N), mean (AVG), and standard deviation (SD) for each item on student
evaluations. We recommend also including the confidence interval in reports for each of
the two items in section I of the SRI. With confidence intervals, there is a greater
likelihood of avoiding conclusions that overlook sample size.
Machine-generated verbal cues. Large standard deviations and/or large confidence
intervals render means less reliable as indicators. Arreola (2007, p. 120) recommends
machine-generated verbal cues as an aid to the reviewer, for example:
On item 6, the standard deviation was 1.3. This can be interpreted to mean that
there was CONSIDERABLE DISAGREEMENT among the students on this item
and thus the average response value should NOT be interpreted as representing a
consensus rating by the class.
And
On item 18, the average response was 3.1, and the standard deviation was .04.
This can be interpreted to mean that the students rated you as being
MODERATELY HIGH on this item and there was a HIGH degree of consensus
among the students in this rating.
We recommend machine-generated comments of this nature in the report for both items
in Section I of the SRI.
III.
We were able to come to agreement to a point regarding the choice of norm groups. We
are in agreement that comparisons made with the use of norm groups should be fair and
appropriate. We know from the literature on student ratings (Berk, 2006, Cashin, 1995,
Marsh and Roche, 1997; McKeachie, 1997) that certain biases exist in how students rate
courses. Overall:
a. Small classes receive higher ratings than large classes;
b. Upper division classes receive higher ratings than lower division classes;
c. Electives and classes where students had a prior interest in the subject matter
receive higher ratings than required courses.
d. Humanities courses receive higher ratings than science and math courses.
The choice of appropriate norm groups is a means of controlling for these biases. Berk
(2006) recommends using subject-matter discipline and program level (lower-division,
upper-division, masters) norms for the fairest comparisons. In addition, although we do
not know of studies that demonstrate bias in online versus face-to-face classes, in the
interest of comparing “like-to-like” we agree that online courses should be compared
only to online courses and face-to-face courses should be compared only to face-to-face
courses.
The following table presents several different options along with the rationale and the
committee view(s):
Norm
How the norm
appears on the
current report
format
CALL MEAN
Rationale
Committee View
This is the closest possible
comparison of “like-tolike.” It will allow, for
example, one section of
ENG 1010 to be compared
to the average ratings from
all sections of ENG 1010.
Yes. We agree that CALL MEAN
should be reported. However, this
will not be a useful norm for courses
without multiple sections and for
courses offered infrequently.
Courses in the
same program at
the same level
CRS PREFIX
LOWER MEAN or
CRS PREFIX
UPPER MEAN
This norm comes close to
comparing “like-to-like,”
and it aligns with Berk’s
suggestion of using
subject-matter discipline
and program level as bases
of comparison.
Courses offered
in the same
department
DEPT MEAN
Courses offered
in the same
school
SCHOOL MEAN
Courses offered
across the college
COLLEGE MEAN
This norm groups together
all courses—upper and
lower division; general
studies, electives, and
major requirements; small
courses and large courses—
taught in a department.
This norm does as above,
with DEPT MEAN, and in
addition groups together
courses of very different
subject matter.
Same as SCHOOL MEAN
Maybe. There may still be
distinctions among lower or upper
level courses within a program,
wherein some courses are required
and some are elective courses. This
norm does not fully control for bias
“c” identified above. The FETF
should explore allowing
departments to identify for
themselves finer distinctions
within their program offerings.
No. We do not feel that this is an
appropriate and fair basis for
comparison.
All courses that a
faculty member
has offered
FACULTY MEAN
All sections of the
same course over
a defined period
of time
This norm averages the
instructor’s ratings from all
courses taught.
No. We do not feel that this is an
appropriate and fair basis for
comparison.
No. We do not feel that this is an
appropriate and fair basis for
comparison.
No. Each faculty member’s
combination of courses is different
and thus will be differently subject
to the above mentioned biases.