COMPARISON OF FCQ RESULTS FOR SUMMER 2006 (ONLINE) VS. SUMMER 2005
(PAPER)
Perry Sailor
University of Colorado at Boulder
Office of Planning, Budget, and Analysis
September 2007
Summary
Background and procedure. Although online administration of FCQs has been an available
option for Boulder instructors since Spring 2003, the vast majority have always chosen to
administer FCQs in class on paper forms (with the exception of online courses). However,
because of time demands on the FCQ staff caused by the planned introduction of a new FCQ
form in Fall 2006, the Office of Planning, Budget, and Analysis (PBA) made online
administration mandatory for all FCQs in Summer 2006 for that term only. This provided the
opportunity for a natural experiment on the effect of mode of administration (paper vs. online) on
ratings and response rates.
Instructor ratings, course ratings, and response rates from summer 2005 (paper administration)
and summer 2006 (online administration) were compared. We did two sets of comparisons
between terms: (1) all sections, and (2) restricted to sections of the same course taught by the
same instructor in each term.
Key results
Response rate: The average response rate was considerably lower when the FCQ was
administered online, compared to paper administration.
Ratings of course and instructor: The effect of online administration on instructor and
course ratings differed by course level. For undergraduate sections, average ratings
were slightly lower with online administration, “slightly” meaning sufficient to move a
section 2 to 6 percentile points lower in the distribution of section ratings. For graduate
sections, the opposite was true, with course and instructor ratings slightly higher when
FCQs were administered online than on paper.
Variation in ratings: For undergraduate sections, variation within sections in course and
instructor ratings were larger when the FCQ was administered online. For graduate
courses, variation was higher with online than with paper administration for course
ratings, but lower for instructor ratings. As with average ratings, all differences between
paper and online variation were small.
Section size: The negative effect of online administration ratings was larger in bigger
sections; there was little or no effect on instructor or course ratings in sections of less
than 20 students, but a bigger one in sections of 40 or more.
Instructor type: We compared results for tenured/tenure track instructors (TTT),
teaching assistants (TA), and others. Differences among the three groups were small
and inconsistent for course and instructor ratings. For response rates, the negative
effect of online administration was much less pronounced for TTT instructors than the
others, but was still sizable.
Discipline College: Unlike other colleges, average instructor ratings and course ratings
in Engineering were actually higher with online administration than with paper, and
online response rates were almost the same as paper (although it was because paper
Paper vs. Online FCQ Administration
Page 2 of 10
response rates were low, not because online response rates were high). However,
these results were based on only 9 sections.
Details
There were 677 sections in summer 2006 (online), and 644 sections in summer 2005 (paper)1.
Section means, standard deviations, and return rates from the two terms were compared. The
unit of analysis was the section – that is, the means and SDs were calculated over 677 (2006)
and 641 (2005) section means, not over the 7,404 (2006) and 11,282 (2005) individual forms in
the sections. Table 1 shows results for all sections. “Effect size” is the difference between the
mean from the online administration and the mean from the paper administration, divided by the
standard deviation of the paper administration. Thus, it is the difference between the two
modes of administration, expressed in units of standard deviation.
Table 1. Summer 2005 (Paper) and 2006 (Online) results for all sections.
2005
2006
(Paper) (Online)
Measure
Mean
Mean
Difference Effect Size
Instructor Rating Section Mean
3.45
3.36
-.09
-.19
Course Rating Section Mean
3.27
3.24
-.03
-.06
Response Rate
73%
48%
- 25% pts. -1.00
Instructor Rating Section SD
0.66
0.73
.06
.18
Course Rating Section SD
0.76
0.81
.05
.17
N sections
641
677
N forms
11,282
7,404
All of the online vs. paper differences were statistically significant, except course rating.
However, with the exception of response rate, where there was a very large difference, the
effect sizes are small, all less than 0.2 standard deviation. Because statistical significance is
largely a function of sample size – with large samples, even tiny differences of no practical
significance can be statistically significant -- effect size is a more meaningful measure. (See
http://www.colorado.edu/pba/ia/statrules/eff_size.html for more information on effect sizes.)
There were 250 cases over the two terms in which the same course was taught by the same
instructor, once using paper FCQ administration, once using online. All further analyses in
this report include only these 250 cases, involving the same instructors teaching the
same courses in both terms. Table 2 shows the overall results from these sections:
1
Note: Some 2005 FCQs were administered online; these were eliminated from analyses.
D:\98881671.doc 7/12/2016 5:57:52 AM
Paper vs. Online FCQ Administration
Page 3 of 10
Table 2. Summer 2005 (Paper) and 2006 (Online) results for sections of the same course
taught by the same instructor in both terms.
Measure
Instructor Rating Section Mean
Course Rating Section Mean
Response Rate
Instructor Rating Section SD
Course Rating Section SD
2005
2006
(Paper) (Online)
Mean
Mean
Difference Effect Size
3.49
3.40
-.09
-.20
3.32
3.28
-.04
-.09
72%
48%
- 24% pts.
-.97
0.65
0.74
.09
.29
0.75
0.82
.06
.21
Again, the effect sizes for course and instructor ratings are small, although slightly larger than in
the comparison of all sections reported in Table 1. How meaningful is a difference of .09 points
(instructor rating) or .04 points (course rating)? Applying the same mean ratings to the same
courses using paper-administered FCQs in fall 2005, the difference between a 3.49 mean
instructor rating and a 3.40 would be tantamount to a move from the 56th percentile to the 50th.
The difference between a 3.32 course rating and a 3.28 would be a move from the 66th
percentile to the 64th.
Box plots of course and instructor ratings for 2005 (paper) and 2006 (online) are attached in the
appendix (Figure 1), as are box plots of response rates (Figure 2).
Differences by section size
Table 3 shows course ratings, instructor ratings, and response rates for the two administration
types broken down by section size, involving only the subset of 250 sections of the same course
taught by the same instructor in both terms. There appears to be a bigger effect on instructor
ratings in larger sections; that is, online administration vs. paper makes a bigger difference for
instructor ratings as section size gets larger. The same is true for course ratings, where virtually
the entire difference between paper and online ratings is for sections of 40 or more students.
The effect sizes for response rate are much larger for sections of 10-19, 20-39, and 40 or more
students than for very small sections of 1-9 students. Unfortunately, the biggest section in the
sample was less than 100 students, so we couldn’t check to see if differences between paper
and online ratings continued to grow as section sizes reached the 200, 300, or even 400 or
more students that occur in fall and spring terms.
Table 3. Ratings for summer 2005 (paper) vs. summer 2006 (online), by class size, for sections of the same course
taught by the same instructor in both terms.
N in Class
1-9
10-19
20-39
40-97
N
12
83
121
34
All
250
Instructor Rating
Paper
Online
ES
3.68
3.63
-.08
3.53
3.46
-.16
3.52
3.42
-.24
3.26
3.10
-.33
3.49
D:\98881671.doc 7/12/2016 5:57:52 AM
3.40
-.20
Course Rating
Paper
Online
3.55
3.58
3.36
3.33
3.34
3.30
3.09
2.97
3.32
3.28
ES
.05
-.07
-.09
-.28
-.09
Response Rate
Paper
Online
ES
57%
41%
-.36
75%
50%
-.89
74%
48%
-1.24
66%
47%
-.90
72%
48%
-.97
Paper vs. Online FCQ Administration
Page 4 of 10
Instructor Ratings by Section Size
Average Section Rating
Summer 2005 (paper) vs. Summer 2006 (online)
3.8
3.6
3.4
Paper
3.2
Online
3.0
2.8
1-9
10-19
20-39
40-97
Section Size
Course Ratings by Section Size
Summer 2005 (paper) vs. Summer 2006 (online)
Average Section
Rating
3.8
3.6
3.4
Paper
3.2
Online
3.0
2.8
1-9
10-19
20-39
40-97
Section Size
Response Rate by Section Size
Summer 2005 (paper) vs. Summer 2006 (online)
Average Response
Rate
80%
70%
Paper
60%
Online
50%
40%
1-9
10-19
20-39
Section Size
Differences by class level
D:\98881671.doc 7/12/2016 5:57:52 AM
40-97
Paper vs. Online FCQ Administration
Page 5 of 10
As table 4 shows, online instructor and course ratings were actually higher than paper in
graduate level classes, the opposite of undergraduate classes. Effect sizes were small. Return
rate differences between paper and online administration were similar for graduate and
undergraduate classes.
Table 4. Ratings for summer 2005 (paper) vs. summer 2006 (online), by class level.
Level
Grad
Undergrad
N
17
233
Instructor Rating
Paper
Online
ES
3.67
3.71
.12
3.48
3.38
-.22
All
250
3.49
3.40
-.20
Course Rating
Paper
Online
3.53
3.56
3.30
3.26
3.32
3.28
ES
.07
-.09
Response Rate
Paper
Online
ES
76%
45%
-.84
72%
48%
-1.00
-.09
72%
Instructor Ratings by Course Level
Summer 2005 (paper) vs. Summer 2006 (online)
Average Section
Rating
3.8
3.7
3.6
Paper
3.5
3.4
Online
3.3
3.2
Grad
Undergrad
Course Level
Course Ratings by Course Level
Summer 2005 (paper) vs. Summer 2006 (online)
Average Section
Rating
3.6
3.5
3.4
Paper
3.3
Online
3.2
3.1
Grad
Undergrad
Course Level
D:\98881671.doc 7/12/2016 5:57:52 AM
48%
-.97
Paper vs. Online FCQ Administration
Page 6 of 10
Response Rate by Course Level
Summer 2005 (paper) vs. Summer 2006 (online)
Average Response
Rate
80%
70%
Paper
60%
Online
50%
40%
Grad
Undergrad
Course Level
Differences by instructor type
Table 5 shows ratings and response rates for the two administration types broken down by
instructor type (tenured/tenure track (TTT), teaching assistant (TA), and other). Differences in
effect sizes among the three groups are small and inconsistent for course and instructor ratings.
For return rates, mode of administration made much less difference for TTT instructors than for
TAs and others. Recall that the effect size for mode of administration was likewise less for very
small sections than for larger ones; so we checked to see if perhaps more TTT instructors
taught very small sections. However, this was not the case.
Table 5. Ratings for summer 2005 (paper) vs. summer 2006 (online), by instructor category.
Category
TTT
TA
Other
N
68
32
150
Instructor Rating
Paper
Online
ES
3.59
3.53
-.17
3.27
3.15
-.20
3.50
3.39
-.26
All
250
3.49
3.40
-.20
Course Rating
Paper
Online
ES
3.46
3.44
-.05
2.98
3.04
.12
3.33
3.25
-.18
3.32
3.28
-.09
Response Rate
Paper
Online
ES
68%
52%
-.52
68%
39%
-1.26
75%
48%
-1.23
72%
48%
Differences by Discipline College
Engineering sections did not show the same pattern as other colleges; average instructor
ratings and course ratings were actually higher with online administration than with paper
administration. Moreover, unlike every other college, online response rates in engineering were
almost the same as paper (albeit because paper response rates were low, not because online
response rates were high). However, the reader should note that these results are based on
only 9 sections in each condition, and so should be regarded cautiously.
D:\98881671.doc 7/12/2016 5:57:52 AM
-.97
Paper vs. Online FCQ Administration
Page 7 of 10
Online administration was associated with a particularly large negative gap in course and
instructor ratings in business sections.
We checked to see if engineering sections tended to be particularly small or business sections
particularly large, since class size was related to mode-of-administration effect (see Table 4);
neither was the case.
Table 6. Ratings for summer 2005 (paper) vs. summer 2006 (online), by discipline college.
Instructor Rating
Paper
Online
ES
3.48
3.40
-.24
3.50
3.40
-.22
3.36
3.14
-.47
3.64
3.60
-.12
3.49
3.52
.07
*
*
*
*
*
*
3.58
3.58
.00
College
N
Arch & Plan
6
Arts & Sci
180
Business
27
Education
10
Engineering
9
Journalism
2
Law
4
Music
12
All
250
3.49
3.40
-.20
Course Rating
Paper
Online
3.34
3.32
3.31
3.28
3.17
3.00
3.57
3.52
3.28
3.51
*
*
*
*
3.56
3.44
3.32
3.28
ES
-.10
-.07
-.33
-.13
.70
*
*
-.26
-.09
*Too little data to for meaningful comparison; however “All” includes these colleges.
Instructor Ratings by College
3.7
3.6
3.5
Paper
3.4
Online
3.3
3.2
Discipline College
D:\98881671.doc 7/12/2016 5:57:52 AM
us
ic
M
n
En
gi
ne
er
in
g
Ed
uc
at
io
Bu
sin
Sc
i
&
Ar
ts
Pl
a
&
Ar
ch
es
s
3.1
n
Average Section Rating
Summer 2005 (paper) vs. Summer 2006 (online)
Response Rate
Paper
Online
ES
71%
60%
-.92
73%
47%
-1.04
76%
52%
-1.71
92%
45%
-5.87
53%
51%
-.06
*
*
*
*
*
*
65%
48%
-.65
72%
48%
-.97
Paper vs. Online FCQ Administration
Page 8 of 10
Course Ratings by College
Summer 2005 (paper) vs. Summer 2006 (online)
3.6
3.5
3.4
Paper
3.3
Online
3.2
3.1
3.0
us
ic
M
n
En
gi
ne
er
in
g
Ar
ts
Ed
uc
at
io
Bu
sin
Sc
i
&
Pl
a
&
Ar
ch
es
s
2.9
n
Average Section Rating
3.7
Discipline College
Response Rates by College
100%
90%
80%
70%
60%
50%
40%
Paper
Discipline College
D:\98881671.doc 7/12/2016 5:57:52 AM
us
ic
M
n
En
gi
ne
er
in
g
Ed
uc
at
io
Bu
sin
Sc
i
&
Ar
ts
Pl
a
&
Ar
ch
es
s
Online
n
Average Response Rate
Summer 2005 (paper) vs. Summer 2006 (online)
Paper vs. Online FCQ Administration
APPENDIX
Figure 1.
Course, Instr rating distributions Summer 05 and Summer 06
|
4 +
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
+-----+
+-----+
3.75 +
|
|
|
|
|
|
|
|
+-----+
|
|
|
|
|
+-----+
|
|
|
|
|
|
|
|
|
|
|
*-----*
|
|
3.5 +
|
|
|
|
|
|
*-----*
|
|
|
|
|
| + |
|
|
|
*-----*
*-----*
|
|
| + |
|
|
|
|
|
|
|
|
|
3.25 +
| + |
| + |
+-----+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
+-----+
3 +
+-----+
+-----+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2.75 +
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2.5 +
|
|
|
|
|
|
|
0
|
|
|
|
0
|
|
|
|
0
|
2.25 +
|
|
0
|
|
|
|
0
|
|
|
|
0
|
|
|
|
2 +
0
|
0
|
|
0
|
0
0
0
0
|
0
0
0
0
1.75 +
0
0
|
0
0
|
0
0
*
0
|
0
1.5 +
0
0
|
0
0
*
|
0
0
|
0
0
0
1.25 +
|
0
0
*
0
|
0
|
*
*
0
1 +
*
|
0
*
0
|
0
|
0.75 +
*
|
*
------------+-----------+-----------+-----------+----------Mode (Yr)
Paper(05)
Online (06) Paper (05) Online (06)
var
Course
Course
Instr
Instr
Top edge of box = 75th percentile
Bottom edge of box = 25th percentile
Middle horizontal line = Median (50th percentile)
+ = Mean
Vertical lines = Extent of data (to 1.5 interquartile ranges)
0 = Data values 1.5 to 3 interquartile ranges
* = Data values more extreme than 3 interquartile ranges
D:\98881671.doc 7/12/2016 5:57:52 AM
Page 9 of 10
Paper vs. Online FCQ Administration
Figure 2.
Return rate distributions Summer 05 and Summer 06
Variable:
retrate
(Return rate)
Schematic Plots
|
1 +
|
0
|
|
|
|
|
|
0
0.9 +
+-----+
0
|
|
|
|
|
|
|
|
|
|
|
|
0.8 +
*-----*
|
|
|
|
|
|
|
|
|
|
| + |
|
0.7 +
|
|
|
|
+-----+
|
|
|
|
|
|
|
0.6 +
|
|
|
|
+-----+
|
|
|
|
|
|
|
|
0.5 +
|
|
|
|
|
*--+--*
|
|
|
|
|
|
|
|
0.4 +
|
|
|
|
|
+-----+
|
0
|
|
0
|
0.3 +
0
|
|
|
|
0
|
|
0
|
0.2 +
|
|
0
|
|
0
|
|
|
0.1 +
0
|
|
|
|
0
|
|
0 +
*
0
------------+-----------+----------Mode (Yr)
Paper(05)
Online (06)
Top edge of box = 75th percentile
Bottom edge of box = 25th percentile
Middle horizontal line = Median (50th percentile)
+ = Mean
Vertical lines = Extent of data (to 1.5 interquartile ranges)
0 = Data values 1.5 to 3 interquartile ranges
* = Data values more extreme than 3 interquartile ranges
D:\98881671.doc 7/12/2016 5:57:52 AM
Page 10 of 10