Assessing Whether There is an Effect on Test Taking for Students
with Different Language Abilities When Given Visual Cues
Valerie Cousineau
Kevin Doubleday
Ruoyu Huang
1
Executive Summary
Trianna Oglivie is from the Department of Speech, Language & Hearing Sciences. She and her team
are interested in identifying whether visual cues have an effect on the math scores of students with
varying English proficiency levels, namely, typically-developing, native English-speakers (NEs),
specific language impairment (SLI) or English-Learners (ELs). The analysis is post-hoc and utilizes
data which was previously obtained through a standardized math assessment. The test featured
questions that incrementally increased in difficulty as the child progressed and had visual cues
associated with each question. Ms. Oglivie and her team rated the visual cues as either containing
high or low visual content, with high content visual cues theoretically being more helpful to the
student. Ms. Oglivie’s main interest is in assessing whether there is an interaction between the group
taking the test, the level of question difficulty and the content of the visual cue given. Her
hypothesis is that questions with high visual content will help the students in both the SLI and EL
group to perform better overall on the test than they otherwise would have, as compared with NEs
who should not need help from the visual content. We used logistic regression to model the
relationship between answering a question correctly given some language group, question difficulty,
and visual content.
2
Detailed Summary
Ms. Oglivie is conducting post-hoc research involving a math exam given to children ages ___ to
___. The exam has seven sub-tests, covering a variety of math subject areas. Students answer
questions sequentially until a certain number of questions are missed in a row. Students begin on
different question numbers within the exam based on their established math skill level.
2.1 The research question
Ms. Oglivie is concerned that students with lower English language ability may do worse on a math
exam, despite the fact that their math skills may be quite good. She wants to ensure that a math
exam is testing math, not English language skills. She believes that visual aids may help those
students with lower English language skills answer more math questions correctly. Additionally,
difficultly of the question may determine how effective the visual aid is. Ms. Oglivie wants to
develop a model that estimates the probability of answering a question correctly given the content of
the visual aid, English language ability group of the student, and question difficulty along with the
interaction of these three variables.
2.2 Variable scales
Our prediction model includes three variables, namely, visual aid content, and English language
group, and question difficulty. Visual aid content is rated as “Low” or “High” based on a visual
assessment. English language group has three levels, typically developing native English speaker
(NE), specific language impairment (SLI), and English-learners (EL). A substantial proportion of
the questions at the beginning and end of the exam were not answered so selection of questions to
include in the analysis was done by pairing question of similar difficulty, but with opposite visual
content designations, on “Low” and one “High”. Question difficulty was classified as “Easy” or
“Hard” where the easy questions are located toward the beginning of the exam and a substantial
proportion of students provided an answer. A similar procedure was used to classify the hard
questions, but selection was done toward the end of the exam.
Questions from the geometry and numeration portions of the exam were included in this analysis.
2.3 Statistical analysis
The outcome of interest is the probability of answering a question correctly, hence we will use
logistic regression to estimate these probabilities. Terms for visual content, question difficulty, and
language group along with their pairwise interactions and three way interaction will be included in
the model.
3
Results
3.1 Logistic model
The estimates from the logistic model are found in Table 1.
Table 1. Estimates from the logistic model for answer a question correct.
Parameter
Estimate
95% Confidence
Log(odds correct) Interval
Intercept
1.20
-Question Difficulty
-2.98
(-3.88, -2.08)
(Ref=Easy)
Visual Aid (Ref=Low)
-0.39
(-1.13, 0.36)
Language Group NE
1.27
(0.31, 2.22)
(Ref=Group SLI)
Language Group EL
-0.02
(-0.83, 0.80)
(Ref=Group EL)
Difficulty*Visual
1.67
(0.49, 2.85)
Difficulty*Group NE
-0.62
(-1.88, 0.65)
Difficulty*Group EL
0.15
(-1.02, 1.33)
Visual*Group NE
-0.15
(-1.40, 1.11)
Visual*Group EL
0.84
(-0.28, 1.96)
Difficulty*Visual*Group NE
-0.01
(-1.72, 1.70)
Difficulty*Visual*Group EL
-0.98
(-2.62, 0.65)
p-value
-<0.001
0.31
0.01
0.97
0.01
0.34
0.80
0.82
0.14
0.99
0.24
We see that the estimates for question difficulty, language group NE, and the interaction between
difficulty and visual aid are significant (p < 0.05). Positive estimates indicate an increase in the
probability of answering a question correctly for the given characteristic, relative to the reference
category. For instance, the question difficulty estimate of -2.98 indicates that for those in the same
language group working on questions with similar visual content presented, there is a decreased
probability of answering the question correctly if the question is difficult versus if the question is
easy. The estimate for the question difficulty, visual aid interaction of 1.67 indicates that for difficult
questions, the probability of answering correctly increases with the high content visual aid versus
low content. Additionally, since this interaction is statistically significant, the visual aid content
modifies the effect of the question difficulty on chances of answering correctly. In other words, the
relationship between answering a question correctly and visual content provided is different for easy
and hard questions. See Table 2 for examples.
3.2 Converting log(odds) to probability
Odds is defined as the probability of an event divided by one minus the probability of the event:
𝑂𝑑𝑑𝑠(𝐸𝑣𝑒𝑛𝑡 𝐴) =
𝑃(𝐸𝑣𝑒𝑛𝑡 𝐴)
1 − 𝑃(𝐸𝑣𝑒𝑛𝑡 𝐴)
We can solve for the probability of the event and find:
𝑃(𝐸𝑣𝑒𝑛𝑡 𝐴) =
𝑜𝑑𝑑𝑠(𝐸𝑣𝑒𝑛𝑡 𝐴)
1 + 𝑜𝑑𝑑𝑠(𝐸𝑣𝑒𝑛𝑡 𝐴)
The logistic model estimates the log(odds of an event) so we must exponentiate the estimates before
solving for the probability. Table 2 contains the probabilities of all combinations of question
difficulty, visual aid content, and English language group.
Table 2. Probabilities of answering question correct.
Visual
Question Language Estimated Probability
Content Difficulty Group
of Correct Answer
High
Easy
SLI
0.83
High
Easy
NE
0.93
High
Easy
EL
0.65
Low
Easy
SLI
0.81
Low
Easy
NE
0.90
Low
Easy
EL
0.78
High
Hard
SLI
0.33
High
Hard
NE
0.44
High
Hard
EL
0.40
Low
Hard
SLI
0.20
Low
Hard
NE
0.28
Low
Hard
EL
0.18
Of note is that high visual content helped students in the SLI and NE language groups answer easy
questions correctly, but only by a small margin (SLI: 0.83 vs. 0.81; NE: 0.93 vs. 0.90). All three
language groups were helped by the high visual content on the hard questions (SLI: 0.33 vs. 0.20;
NE: 0.44 vs. 0.28; EL: 0.40 vs. 0.22).
3.3 Future study design recommendations
For future study designs, it would be ideal if both the question difficulty (where the question is given
to the child on the test) and the visual content associated with it were randomized. Additionally, the
students should take an exam where they answer all questions on the exam so as to minimize
missing answers.