Validity
RMS – May 28, 2014
Measurement Reliability
• The extent to which a measurement gives
results that are consistent
Measurement Validity
• The degree to which an assessment measures
what it is supposed to measure
Construct validity
• The extent to which an operationalization of a
construct (i.e., practical tests developed from
a theory) actually measure what the theory
says they do
– Does an IQ questionnaire actually measure
intelligence?
Content Validity
• Systematic examination of the test content to
determine whether it covers a representative
sample of the behaviour domain to be
measured
– Does the IQ questionnaire have items covering all
areas of intelligence known in the literature?
Face validity
• An estimate of whether a test appears to
measure a certain criterion (no guarantee that
it actually does!)
– Related to content validity, but you don’t have to
know all the theories to make the judgment
Criterion validity
• Are the test and a criterion variable(s)
representative of the same theoretical
construct related
– Compares the test with other measures already
held to be valid
• IQ tests validated against measures of academic
performance (the criterion)
Assignment 2 – Variables &
Measurement
1. Recap your research question and hypotheses
2. Identify the independent and dependent variables of
interest for each hypothesis. For each variable:
a. How could you measure the variable (i.e., what is its
operational definition)?
b. Should it be continuous or discrete?
c. What is its scaling type?
d. What descriptive statistics are appropriate?
e. How would you estimate reliability and validity of
measurement?
QUESTIONS ABOUT ASSIGNMENT
2?
EXPERIMENTAL VALIDITY
Validity
• Experimental validity – the soundness of the
experimental design
– Not the same as measurement validity (the
goodness of the operational definition)
• Internal validity
• Construct validity
• External validity
Internal Validity
• Concerns the logic of the relationship
between the independent and dependent
variables
• It is the extent to which a study provides
evidence of a cause-effect relationship
between the IV and the DV
Confounding variables
• Error that occurs when the effects of two
variables in an experiment cannot be
separated – results in a confused
interpretation of the results
• Example
– Group A (IV = 0)
– Group B (IV = 1)
– Behaviour observed
• Want to say that changes in behaviour are due
to IV
How do you know what could be
confounding?
• Need to make judgments as you design the
experiment
• Be particularly careful with subject variables
Construct Validity
• Extent to which the results support the theory
behind the research
– Would another theory predict the same
experimental results?
• Hypotheses cannot be tested in a vacuum
– The conditions of a study constitute auxilliary
hypotheses that must also be true so that you can
test the main hypothesis
Example
• H1: Anxiety is conducive to learning
• Participants selected on basis of whether they bit their
fingernails (sign of anxiety)
• Observe how fast they can learn to write by holding a
pencil in their toes (a learning task)
• Did not just test impact of anxiety of learning
– Tested that fingernail biting is a measure of anxiety (H2)
– Tested that writing with toes is a good learning task (H3)
• If either is false, you could have negative results even if
the main hypothesis is true
Similarities
Construct Validity
• Try to rule out other
possible theoretical
explanations of the results
• Must design a new study to
help you choose between
competing theoretical
explanations
Internal Validity
• Try to rule out alternative
variables as potential causes
of the behaviour of interest
• May be able to redesign the
study to control for the
source of confounding
External Validity
• How well the findings of an experiment
generalize to other situations or populations
– Strictly speaking results are only valid for other
identical situations
– Can be hard to know which situational variables
are important
• Ecological validity is related
– Extent to which an experimental situation mimics
a real-world situation
Statistical Validity
• Extent to which data are shown to be the result of a causeeffect relationship rather than an accident
– Does the relationship exist or was it caused by pure chance
•
•
•
•
Similar to internal validity
Notion of power (is n big enough?)
Is measure accurate?
Is the statistical test appropriate?
– 3 kinds of lies: lies, damn lies, and statistics
• Statistical test establishes that an outcome has a certain
low probability of happening by chance alone
– No guarantee that it’s not a random error in sampling or
measurement
So we did the green study again and got no
link. It was probably a – RESEARCH
CONFLICTED ON GREEN JELLY BEAN/ACME
LINK; MORE STUDY RECOMMENDED
- xkcd.com/882/
Threats to Internal Validity - 1
• Events outside the laboratory (history):
– Can occur when different experimental conditions are
presented to subjects at different times
• Example: Feelings of depression (failure condition on
Monday, success condition on Wednesday)
– What if Monday was rainy and Wednesday was sunny?
• Maturation:
– A source of error in an experiment related to the
amount of time between measures
– Subjects may change between conditions because of
naturally occurring processes
• Aging doesn’t just occur with people!
Threats to Internal Validity - 2
• Effects of Testing:
– Changes caused by testing procedure (not processes
unrelated to the experiment)
• Learn how to do the task, learn the style of a test
• Regression Effect:
– tendency of subjects with extreme scores on a 1st measure
to score closer to the mean on a second testing
– Not a perfect correlation between 2 variables
• SAT and GPA or repeated SAT
• Arises when there is an error associated with the measurement of
the variables (e.g., students know some answers and make
lucky/unlucky guesses at the rest)
• Random error – that part of the value of a variable that can be
attributed to chance
Threats to Internal Validity - 3
• Mortality:
– The dropping out of some subjects before an
experiment is completed
– Selective subject loss
– Can introduce bias depending on the reason they
are dropping out
• Selection:
– Any bias in selecting groups can undermine
internal validity
Threats to Construct Validity – 1
• Difficult as there are an indefinite number of
theories that may account for a given
relationship
• Strategy: ask whether alternative theoretical
explanations of the data are less plausible
than the theory believed to be supported by
the research
Threats to Construct Validity - 2
• Loose connection to theory and method
– The anxiety/learning example
– Often research suffers from poor operational
definition of theoretical constructs
• Ambiguous effect of IV
– Experimenter can control all reasonable confound
variables
– Participant may compromise result by seeing the
situation differently than the experimenter
Human-subject research
• Good subject tendency:
– Participants want to help the research achieve
their goals (and may not understand the goals)
• Evaluation apprehension:
– Tendency of participants to alter their behaviour
to appear as socially desirable as possible
Threats To External Validity
• Other subjects
– Are the participants truly representative of the sample
to which you are trying to generalize?
• Other times
– Would the same experiment conducted at another
time produce the same results?
– Caution: the web is rapidly changing
• Other settings
– Lab vs field
– Structured vs unstructured environments
Reading: Exp. Challenges in Cyber
Security
•
•
•
•
What’s the research problem?
What was their approach?
What types of validity do they discuss?
Hypotheses?
– IV, DV
– Measurement validity
• Internal validity
• Construct validity
• External validity