Is it Live or is it Internet?
Experimental Estimates of the
Effects of Online Instruction on
Student Learning
David Figlio, Northwestern U
Mark Rush, U Florida
Lu Yin, American Institutes for Research
Background
• Two major trends affecting higher education
– Declining state and local appropriations  more
fiscal challenges, increasing demands for
“efficiency” (new NRC panel)
– Rapid improvements in technology
• As a consequence, millions of students are
now taking classes online
What do online classes look like?
• Two major types:
– Innovative, highly interactive classes aimed at
exploiting special nature of the internet and modern
technology
– Traditional lectures presented in an online format
• While door #1 is the type of class advocated by
learning scientists (NB: I’m working with
colleagues to develop this type of class in high
school science), door #2 is what most universities
are doing.
What is the causal question of
interest?
• Do students learn more in a traditional lecture
when it is broadcast on the internet than when
it is presented in a standard lecture hall?
• In principle, the results could be positive or
negative:
– Pro: increased flexibility; no need to rely on
others’ notes; ability to annotate/rewatch lectures
– Con: increased barriers to student-faculty
interaction; reduced ability to ask just in time
questions; incentives to defer work
What is the causal question of
interest? (part 2)
• Are there heterogeneous effects of internetbased traditional lectures on students?
• Why interesting?
– Some groups may face additional communication
barriers (e.g., language minority students)
– Some groups may have lower self-regulation skills
(e.g., college-aged men; relatively lowerachievers)
What is the evidence to date?
• Almost nothing
• Mainly small-scale case studies; unsurprisingly,
almost all studies are based on type 1 of internetbased class rather than type 2 of internet-based
class
• Somewhat larger studies have poor treatmentcontrol contrast
• Considerable need for experimental evidence on
both types of internet class, but especially the
broadcast-traditional-lecture type  this study
What is the ideal experiment?
• Treatment and control students should be taught in tandem
– Same instructor, same exams, same lectures, same supplementary
material, same readings
• Pure randomization to live vs. internet treatment
– No opt-out from experiment
• No opportunities for contamination of treatment and control
– Live students cannot view internet lectures; internet students cannot
attend live lectures
• Ample opportunities for detecting heterogeneous effects and
improving external validity
– Dozens of experiments in different courses at different institutions,
with within-course randomization
– Cluster-randomized design; clustering on subgroups to ensure
sufficient sample size to detect subgroup-specific effects
Fidelity to ideal experiment
• Treatment and control students should be taught in tandem
– Same instructor, same exams, same lectures, same supplementary
material, same readings
• Pure randomization to live vs. internet treatment
– No opt-out from experiment (instead: randomization of volunteers)
• No opportunities for contamination of treatment and control
– Live students cannot view internet lectures; internet students cannot
attend live lectures (live students may potentially watch with friends)
• Ample opportunities for detecting heterogeneous effects and
improving external validity (nope: just one class, no group cluster)
– Dozens of experiments in different courses at different institutions,
with within-course randomization
– Cluster-randomized design; clustering on subgroups to ensure
sufficient sample size to detect subgroup-specific effects
Threats to
internal and external validity
• (1) Volunteers, rather than pure
randomization
– How representative are volunteers of the potential
study population at the institution? [external
validity]  Table 1
– Knowledge that this is an experiment might lead
to differential attrition of live vs. internet [internal
validity]  Table 2 contrasts; bounding exercise in
Table 3
Threats to
internal and external validity
• (2) Fidelity of randomization and lack of treatment
contamination
– Do people drop from experiment post-randomization?
[internal validity]  as 15 students assigned to “live”
dropped from the experiment, Table 3 compares results
treating defectors as “live” versus dropping from study
– Do “live” students view internet version and do “internet”
students attend live lecture? [internal validity]
• Door guards checked IDs so we know that no “internet” students
attended live lectures
• Live students did not have online access, but could have accessed
via friends’ log-ins  Figure 1 shows that “live” students attended
substantially more lectures than “live+internet” non-volunteers
Threats to
internal and external validity
• Intro economics might be special
• The university in question might be unusual
• The particular instructor may translate
well/poorly to the internet platform
Lots of hand-wringing
And also a call for more experiments in other
subjects and settings and with a design aimed
at detecting heterogeneous treatment effects
Threats to
internal and external validity
• (1) Volunteers, rather than pure randomization
– How representative are volunteers of the potential
study population at the institution? [external validity]
 some differences; volunteers had higher GPAs and
lower SAT scores and mom was less likely to be a
college grad.
– Knowledge that this is an experiment might lead to
differential attrition of live vs. internet [internal
validity]  no evidence of differential attrition: 6 live,
10 online attriters, and no differences between them.
Bounding exercise (giving attriters scores of 0 or 100
on missed exam) shows tight bounds when
considering attrition.
Threats to
internal and external validity
• (2) Fidelity of randomization and lack of treatment
contamination
– Do people drop from experiment post-randomization? [internal
validity]  no differences in results when we treat defecting
volunteers as “live” versus when we drop them from the study
– Do “live” students view internet version and do “internet”
students attend live lecture? [internal validity]  can’t know for
certain about live students viewing lectures online, but it looks
like live students definitely attend more live lectures than those
with the choice. Note that there is a “professional” private notetaking and tutoring service that is very popular and available to
all students regardless of live/online. Other studies at the
institution on cramming indicate that many internet coursetakers don’t view all (or most, or sometimes any) of the lectures
either
Threats to
internal and external validity
• Intro economics might be special
• The university in question might be unusual
• The particular instructor may translate
well/poorly to the internet platform
This is important, and there’s nothing we can
do about this except to call for more
experiments
Results
• Small insignificant positive estimated effects of
live-only vs. internet-only instruction; statistically
significant (mostly due to larger coefficients
rather than smaller standard errors) when
conditioning on covariates
• Positive estimated effects are largest for Hispanic
students (sig) and Asian students (not sig); male
students (not sig) and students with relatively low
SAT scores (not sig)  larger sample sizes and
cluster randomization might have helped detect
differences here
Conclusions
• There might be efficiencies to exploit with
internet-based traditional lectures, but there
is no free lunch
• The results of this experiment should be
interpreted as a first piece of evidence;
responsible universities should be slow to
implement this policy change despite the
momentum and push for this. We need many
more – including larger -- experiments!