Experimental Research
Methods in Language
Learning
Chapter 5
Validity in Experimental Research
Leading Questions
• ‘A measure is not valid if it is not reliable.’ Why
do you think this can be the case?
• In academic research, why do you think we
need to pay attention to research validity?
• Do you know various types of research
validity? If so, what are they?
Validity
• Validity is related to the accuracy,
correctness and legitimacy of the
measurements and observations made
during data collection.
• Validity is also related to the soundness of the
inferences made on the basis of the data
collected.
• To consider validity, we need to consider
theories, how a construct is defined, how
instruments are developed and used and
how data are analyzed and interpreted.
Reliability
• Reliability is often related to the issue of the
consistency of research instruments,
observations, or measurements of a construct.
• An experimental study cannot be considered
valid if it uses unreliable data from
participants to analyze and answer the
research questions.
• Example, using a reliable and valid reading
score to claim how someone is good at
writing is invalid and inappropriate.
Reliability
• The reliability of instruments is concerned with
the degree to which the results of a
questionnaire, test or other measuring
instrument are consistent.
• The reliability of the research result of a study
is concerned with the degree to which the
research result (e.g., the difference between
experiment and control groups) is likely to
reappear.
Internal and External Validity
• Internal validity is closely related to what is
considered and done during research
processes.
• External validity is related to a generalization
of the study to other participants and settings
Internal Validity
• Is concerned with the logic of the causal-like
relationship between the independent and
dependent variables under examination.
• Is related to the extent to which other
confounding variables influence the research
outcomes.
• The less confounding variables interference,
the better the internal validity of the study
• prerequisite of external validity
Threats to Internal Validity
Two major groups of threats that need to be
considered:
• Threats related to research participants (e.g.
selection bias, history effect, maturation,
attrition)
• Threats related to research instruments and
procedures (e.g. experimenter/researcher
effect, testing effect, instrumentation effect,
statistical regression effect )
Threats to Internal Validity
Threats relevant to research
participants
• Selection bias: When there are major preexisting differences between participants in
the treatment and control groups.
• The history effect: when a specific situation or
event that takes place during an
experimental study and interferes the
dependent variable of interest.
• The maturation effect: biological, cognitive or
psychological developments that occur
naturally within the participants of the
experimental study.
Threats relevant to research
participants
• The attrition or mortality effect: when there is an
imbalance in the loss of participants across
comparison groups.
• The Hawthorne effect: when experimental
participants change their behaviour due to the
mere fact of the experiment taking place,
rather than the specific treatment of the
experiment).
• The Placebo effects: when experimental
participants believe that they are receiving a
special treatment that improve their current
condition.
Threats relevant to research
participants
• The John Henry effect: when participants in
the control group invest more effort in their
learning to compete with those in the
experimental group.
• The demoralized effect: how students in the
control group feel that they are not being
treated fairly.
• The diffusion effect: situations in which the
participants in an experimental group share
what the details of the special treatment with
those in the control group.
Threats relevant to research
instruments/procedures
• The experimenter/researcher effect: the
unintentional influence of the experimenters
on the research outcome.
• The testing effect: when researchers use the
identical pretest and posttest.
• The instrumentation effect: when researchers
use different instrument for measuring the
dependent variable for the pretest and
posttest.
Threats relevant to research
instruments/procedures
• The novelty effect: the innovative look of a
treatment or method that may excite
learners, thereby causing them to be
enthusiastic about the treatment.
• The statistical regression effect: when
participants with extreme scores (e.g., the
highest or lowest) in the pretest achieve
scores in the posttest which are closer to the
mean score (i.e., the average group score).
External Validity
• generalizability of the inferences made on the
basis of an experimental finding to other
learners (with similar characteristics) and other
settings
• whether the study can be useful for other
people -- the importance of the problem
being addressed by an empirical study
Specific Aspects of Validity
• Construct, content, criterion-related, and
statistical validities are related to the internal
validity of the study.
• Construct Validity: related to research
constructs
• The construct validity of a research instrument
concerns the extent to which an instrument
measures what it is intended to measure.
• The construct validity of research concerns the
question of whether the inferred causal-like
results support the theory behind the research.
Specific Aspects of Validity
• Content Validity: the extent to which sample
behaviors or abilities are relevant to and
representative of the construct being defined.
• Criterion-related Validity: an instrument should
have a strong relationship with other instruments
that measure the same or similar construct.
• Statistical Validity: appropriate statistical
analyses leading to the inferences and
conclusions; use of the probability value (e.g. p
< 0.05)
Specific Aspects of Validity
• Predictive Validity: the level of predictability
of current test scores, or reported behaviours
to future scores, or behaviors.
• Predictive validity is related to the external
validity.
• Face Validity: is not validity in the sense in
which the term has been used so far.
• Face validity is related to the appearance of
a research instrument being used or an
experiment being conducted.
Discussion
• In your view, why is it difficult to separate
reliability from validity?
• Which one do you think more important for
an experimental study: internal validity versus
external validity? Why?
• If you had to choose only one of the following
types of validity for your study (namely
construct validity, predictive validity, or
statistical validity), which one would you
choose? Please explain your reasons.