Mining Data from Randomized Within-Subject Experiments in an Automated Reading Tutor
Joseph E. Beck and Jack Mostow
Project LISTEN (www.cs.cmu.edu/~listen), Carnegie Mellon University
Funded by National Science Foundation and The Heinz Endowments
Experiments embedded in the Reading
Tutor help evaluate its decisions in tutoring
decoding, vocabulary, and comprehension
Research question: What
tutorial decision does the
experiment investigate?
Decoding: What type of help is most effective
for helping students learn to decode words?
Vocabulary: Does a brief introduction to a
word’s meaning before a story help the student
to learn the word and comprehend the story?
Before student starts to read story,
Trial context: In what situation
does the tutorial decision occur?
Randomized decision: The
Reading Tutor chooses at
random among plausible
alternative actions. Each such
choice starts an experimental
trial. Randomizing the decision
allows causal attribution.
Student is reading story
Reading Tutor identifies vocabulary words in story.
Student is reading story and clicks on a word for help
Reading Tutor
randomly picks
half of vocabulary
words to explain
Reading Tutor
randomly selects
which type of help
to provide
Rhymes with
“saw”
Trial outcome: We define the
outcome of each decision based
on subsequent student behavior
– a much finer-grained and
more copious source of data
than post-test scores.
Comprehension: Does inserting generic whquestions help students comprehend stories?
“Draw”
Reading Tutor
randomly
decides whether
to insert a whquestion
…
Examine student performance on a future encounter
of the word. Does the student ask for help? Does
the tutor accept the word as read correctly?
While student reads story,
assess comprehension of story.
After student finishes story,
assess retention of vocabulary.
Student continues
reading story
Two outcome
measures
During the story,
student encounters
cloze questions
N=15,187 cloze questions
Logistic regression model
Analysis: Aggregating over
many such trials can tell not only
which choices work best, but
when and for whom.
N = 189,039 help events
Rhyming help is most effective overall.
For hard words, best to just tell the student the word.
N = 5,668 vocabulary words tested
Explaining words helps for both within-story
comprehension probes and after-story vocabulary
questions – but effects interact with reading level.
This work was supported by the National Science Foundation under ITR/IERI Grant No. REC-0326153. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the National Science Foundation or the official policies, either expressed or implied, of the sponsors or of the United States Government.
Independent variables
# preceding wh- questions
# preceding cloze questions
# recent wh- questions
# recent cloze questions
Time since prior question (sec)
…
Helps/Hurts

=

=

…
p
0.023
0.074
0.036
…