Eileen Blair
Assignment 1
Threats to Internal and External Validity
Validity refers to the degree to which something measures what it is suppose to measure.
When conducting a study, there are often extraneous and confounding variables that can possibly
threaten external and internal validity. I will be conducting a pre-experimental design study, in
which I will be assessing the affects of the Smartboard on the ESL students’ language skills and
anxiety level. This study will be a one group-, pretest-posttest design. One single group will be
pretested, exposed to Smartboard instruction for 4 weeks, followed by a posttest. The pre-test
and post-test are given to the teacher, in which the teacher observes and measures the students
across many factors, such as comfort level in the classroom, ability to use the English language
to communicate socially, and evidence of anxiety. There are ten factors each student will be
measured on prior to the instruction, and post instruction. In conjunction to the pre-test and post
test, the students will take a survey after the Smartboard instruction to measure their personal
attitude towards the effects of Smartboard. The symbolic design for this study is OXO. This
study will take place in 6th grade ESL English Language Arts class in a low –income, urban
school in Brooklyn, New York.
The threats to internal and external validity differ upon the type of study being
performed. The possible threats to internal validity when performing a study include history,
maturation, testing-pretesting sensitization, instrumentation, mortality, statistical regression,
differential selection of subjects, and selection-maturation interaction.
History: The internal threat of history refers to the concept that an unplanned event or
circumstance can take place. This threat is valid in the case of my study because
unplanned events can have an effect on the outcome. (An example may be if there are
technical issues with the Smartboard.)
Maturation: The internal threat of maturation refers to the concept that an individual can
lose interest over time. I do not believe this threat is valid in the case of my study because
the study is taking place over a short period of time. The study also requires the students
to be involved with the Smartboard, and active throughout the study.
Testing-Pretesting Sensitization: The internal threat of testing-pretesting sensitization
refers to the concept that the students’ posttest is affected by the pretest. This idea implies
that the students can memorize the questions or answers. However, in the case of my
study the students are not taking the pre-test or posttest, the teacher is evaluating the
students instead. Therefore, this threat is not valid in this study.
Instrumentation: This internal threat of validity refers to the reliability and validity in the
way the instruments are administered. Since the teacher is administrating the pre-test and
posttest, measuring each student on a number of factors, the measurement can be
considered fairly subjective. While the teacher is very familiar with the students, and I
believe has the ability to assess the children, the subjectivity of the instruments leads
instrumentation to be an internal threat of validity.
Mortality: The internal threat of mortality refers to the idea that students may drop out of
the study. In the case of my study, I do not believe students will drop out, however
students may move away with their families, or change schools. This possibility makes
mortality an internal threat of validity.
Statistical Regression: The internal threat of statistical regression refers to the idea that
those with high/low scores regress to the mean on re-testing. The group being used for
this study is fairly large, consisting of 24 students. I do believe that those with high or
low scores may regress to the mean on the retesting. However, the teacher may also
impact the regression, as she will be evaluating the students for the posttest. Due to these
issues, statistical regression is a threat to internal validity for this study.
Differential –Selection of Subjects: This refers to the idea that groups may be different
before the study begins, ultimately affecting the independent variable. Since there is only
once group for this study, and the group is one entire class, I do not believe that this
threat will be likely. However, if a child moves away or changes schools, it is possible
the group may not remain the exactly the same. If this is the case, than differential –
selection of subjects may pose as an internal threat of validity.
Selection-Maturation Interaction: This refers to the concept that different students will
lose interest at different paces. The entire group may not be interested throughout the
study. I believe this idea is a threat to validity in the case of my study of ESL students
and the Smartboard
External Validity is defined by the overall generalizability of the study. When conducting
a study, there are many factors that can threaten external validity. The external threats include
ecological validity, generalizable conditions, pre-test treatment, selection treatment interaction,
specificity of variables, multiple treatment, treatment diffusion, experimenter effects, and
reactive arrangement/participant effects.
Ecological Validity: This refers to the generalizability of the treatment to different
environments. While the Smartboard-infused instruction can be used with different
environment, the factors to be measured would differ. This treatment is studying the
relationship between ESL students and the Smartboard, which is a specific environment.
Therefore, ecological validity is a threat to the external validity of this study.
Generalizable conditions: This refers to the extent that the study can be replicated and
receive the same results. Since the pre-test and posttest are measured by the teacher, and
fairly subjective, generalizable conditions may pose as external threat to validity. While
the overall findings should be fairly similar, the study may not be exactly the same.
Pre-test treatment: This refers to the idea that participants react differently to the
treatment because they have been pre-tested. This does not pose as a threat to external
validity because the students are not taking part in the pre-test or posttest. This strictly
conducted by the teacher.
Selection-Treatment Interaction: This refers to the idea that the sample is non-random or
consists of volunteers. This does pose as a threat to external validity, as this study is made
up on an ESL class, not random participants.
Specificity of Variables: This refers to the idea that variables are poorly operationalized.
This may pose as a threat to external validity, as it not certain the Smartboard can be used
to teach the English Language, decrease anxiety, and increase student participation and
involvement. While I believe it is feasible, it is not certain.
Multiple Treatment: This refers to the idea that participant receive multiple treatments in
succession. The students will only receive Smartboard infused instruction in 3 of their 5
class periods per week. This means that while the Smartboard instruction is the treatment,
the students will be subjected to other types of instruction or treatment throughout the
week. This means that multiple treatment does pose as an external threat to validity.
Treatment Diffusion: This refers to the idea that different groups receiving different
treatment will communicate. There is only one ESL class in the sixth grade receiving
treatment, however they may discuss with native English-speaking class, or other ESL
grades. Therefore, treatment diffusion may pose as a threat to external validity.
Experimenter Effects: This refers to the idea that the researcher can present potential
threats, either actively or passively. This does pose as a threat to external validity, as the
teacher’s prior knowledge and opinion of each student may impact how she pretests and
posttests each student.
Reactive Arrangement/Participant Effects: This refers to a number of factors, regarding
how the study was conducted.
Hawthorne Effect: This refers to the idea that the students will change their
behavior because they are aware they are being studied. I do believe this may
pose as threat to external validity. The students will not be told directly that they
are being studied, are not part of the pre-test and posttest, and are not aware of
the factors being measured; however their parents must sign a consent form. If a
parent informs the student that they are being studied, the student may alter
his/her behavior.
Compensatory Rivalry Effect: This refers to the idea that the control group or
experimental group may compete and alter their normal behavior. This is not a
threat as there is only one group in this study.
Placebo Effect: This is defined as a reaction to a placebo, and resulting in the
production of expected side effects. This is not a threat to external validity, as
there is no placebo involved in this study.
Novelty Effect: This is defined as the tendency for performance to initially
improve when new technology is implemented. The improvement in performance
is not because of any progress in learning or achievement, but in response to
increased interest in the new technology. This effect is difficult to judge because
the purpose of this study is to measure the impact of technology on the ESL
students’ ability to learn and participate. Therefore, I believe that this may pose
as a threat to external validity.
References
O’Connor-Petruso, S. (2010). Descriptive Statistics Threats to Validity [PowerPoint slides].
Retrieved from http://bbhosted.cuny.edu/webapps/portal/