Quantative Research
Power, Control, & Validity Notes
page 1
Parker, R.M. (1990). Power, control, and validity in research. Journal of Learning Disabilities,
23, 613—620.
Internal Validity

Refers to the extent to which error variance is experimentally controlled

Experimental control may be attained through 5 common methods
1. Random assignment
o Not possible to randomly assign students to a treatment based on a pre-existing,
predetermined condition
2. Hold extraneous variables constant or restrict their range
o Holding variables constant reduces their variation and therefore their
covariation with outcome variables
3. Include extraneous variables in the design to measure their effects
o To include the variables, you need to be able to measure them
4. Employ methods of statistical control
o Statistical manipulation of data to reduce extraneous variation
5. Matching subjects in the treatment and control groups on contaminating extraneous
variables
o Matching is effective if: (a) the researcher can identify and measure likely
contaminating variables, (b) the number of variables to be matched is small (2 to
3), and (c) match at an individual-level
Threats to Internal Validity
Finding often can be explained by “rival hypotheses” – an alternative explanation that
accounts for the findings just as well or even better than the research hypothesis
Quantative Research
Power, Control, & Validity Notes
page 2
1. history
o extraneous events that occur prior to outcome assessment and correlates with
the dependent variable
o in a small town, the orthodontist only came on Wednesdays and that was the day
we had oral reading fluency tests
2. maturation
o naturally occurring developmental changes
o a time series study of on-task behavior of kindergarteners (boys vs. girls)
3. testing
o pretests may affect the posttest
o a survey test on class expectations
4. instrumentation
o deterioration or changes in the accuracy of instruments, devices, or observers
o statewide writing test raters – 1st essay versus 500th essay
5. statistical regression
o grouping subjects on the basis of extreme scores; because the scores cannot go
any lower or higher, they have to move towards the middle
o group is composed of all students that got a score of 1 (out of 6) on their math
problem-solving test
6. mortality
o the loss of subjects during research; are you dealing with hardy survivors or are
the early–leavers different than the stayers
o all students that completed the new behavior management program showed less
non-confrontational theft behaviors, because the early-leavers were arrested
for shopping lifting
Quantative Research
Power, Control, & Validity Notes
page 3
7. selection
o selection is problematic when the experimental group is composed of volunteers
or people selected by preferences
o the magnet school does not provide bussing and both parents must volunteer in
the school one day per week
8. interactions with selection
o threats 1 through 5 may interact with #7
o the example is maturation and selection – a six month first grade reading
intervention where boys are compared to girls
9. direction of causal influence
o it is not clear whether A causes B or B causes A
o emotionally disturbed students have academic deficits – did the emotional
disturbance cause the academic deficit or did the academic deficit cause the
emotional disturbance
10. diffusion or imitation
o the treatment and control groups communicate and the groups become
indistinguishable
o in study hall the treatment kids share the better way of solving math problems
with friends from the control group
11. compensatory equalization
o when the treatment is desirable, administration may give the control group
extra money to compensate
o all control teachers get $500 extra to spend on curriculum
12. compensatory rivalry
o competitive motivation – John Henry effect
o our track team had old sweats versus the rich private school
Quantative Research
Power, Control, & Validity Notes
page 4
13. resentful demoralization
o the experimental treatment is obtrusive, preferential, and desirable; therefore,
the control feel resentful and demoralized
o one class gets new laptops, the other gets cool binders
14. local history
o idiosyncratic events accruing during group data-gathering sessions; enthusiastic
presentations versus bland
o the experimental grant writing class gets the full-professor with lots of grants,
while the control class gets a graduate teaching fellow
External Validity

refers to the degree to which research findings can be generalized across time,
setting, and persons
o people participating need to be representative of the population of interest

types of experimental sampling plans
1. simple random samples
o all participants have equal probability of being included in the experimental
group
2. systematic samples
o every Nth sampling plans
3. stratified random samples
o randomly selected from pre-defined strata in the population
4. cluster samples
o random or stratified sampling over time
Quantative Research

Power, Control, & Validity Notes
page 5
types of non-experimental/non-probability designs
1. convenience samples
o samples chosen because they are readily available
2. purposive samples
o samples chosen by subjective judgment of researchers
3. quota samples
o samples chosen because someone belongs to specified groups
Threats to External Validity
Failing to obtain an appropriate sample results in the inability to generalize the study’s
findings beyond the study
1. interaction among treatments
o multiple treatments may have a cumulative effect and may not generalize when
treatments are given individually
o Reading Mastery phonics instruction versus one short vowel workbook page
2. interaction of testing and treatment
o the pretest may increase or decrease the subject’s responsiveness or sensitivity
to the treatment
o students timed in the mile at the beginning of PE and then at the end; grades
depend on improvement
3. interaction of selection and treatment
o participants that are volunteers or purposively chosen may have traits that don’t
generalize to the population of interest
o treatments for low income children chosen from children living in the University
Housing complex
Quantative Research
Power, Control, & Validity Notes
page 6
4. interaction of setting and treatment
o treatments conducted in one environment can be replicated in other traditional
settings
o behavioral interventions administered by trained doctoral students versus by
teachers after watching a video tape
5. interaction of history and treatment
o special circumstances that happen during the treatments that will not be
replicated in other situations
o research involving teacher sensitivity to student’s verbal threats at a school
where violence has occurred
6. generalizing-across-effect constructs
o the degree to which a treatment found to cause one outcome will also produce
another outcome
o do students whose academic skill deficits get remediated display less behavior
problems
Statistical Conclusion Validity

the appropriate use of statistics to derive accurate conclusion

two types of error – Type 1 and Type 2 errors
o Type 1 error (also called alpha  ) – defined as the incorrect rejection of the null
hypothesis when it is actually true; you say there is a difference when there
isn’t
o Type 2 error (also called beta ß) – when the null hypothesis is not rejected, even
though it is not true; you say there isn’t a difference when there is
o When you set an alpha level you are choosing a level of probability for making a
Type 1 error
Quantative Research
Power, Control, & Validity Notes
page 7
o When choosing a smaller alpha value to minimize the probability of Type 1 error
means increasing the probability of Type 2 error
Threats to Statistical Conclusion Validity
1. low statistical power
o APA requires that authors report effect size differences
o does the author report the alpha level
2. fishing and the error rate problem
o running multiple stats inflates the alpha level
o did the author complete follow-up statistics using a Scheffe, Tukeys, or
Bonferroni test
3. low reliability of measures
o low reliability increases error variance and reduces statistical power
o did the author report reliability of “teacher-made test”
4. low reliability of treatment implementation
o mostly due to the lack of standardized procedures when implementing treatment
o did the author adequately describe the treatment so that it can be replicated
5. random irrelevancies in the experimental setting
o you think the outcome variable was affected by aspects of the experimental
setting other than the treatment
o did the author mention limitations
6. random heterogeneity of respondents
o when respondents are different (heterogeneous) on variables that will affect
the outcome variables
o were subjects blocked on variables/factors
Quantative Research
Power, Control, & Validity Notes
page 8
Construct Validity

describes the variable of interest

is the variable adequately defined and accurately measured
o instruments, procedures, manipulations, and methods

a valid construct must be operationally defined
o does the author suggest a cause and effect relationship when the statistics do
not support a causal relationship
Threats to Construct Validity
1. inadequate preoperational explications of constructs
o when constructs are poorly defined, the measurement of those constructs are
suspect
o construct like “love of reading” or “happiness” or “friendship”
2. mono-operational bias
o the author uses only 1 measure to define a construct
o defining reading using only a literal comprehension test
3. mono-method bias
o tests and methods become so intertwined that other measures are ignored
o student’s ability to read and the OSATs
4. hypothesis-guessing within experimental condition
o participants alter their behavior by guessing the purpose of the study
o especially problematic in survey research
5. evaluation apprehension
o subjects present themselves in a favorable way
o if the evaluator is also an administrator that officially evaluates a staff member
Quantative Research
Power, Control, & Validity Notes
page 9
6. experimenter expectancies
o researcher influences participants’ behavior and, thus, the outcome behavior
o the experimenter should not be the person implementing the intervention
7. confounding levels of constructs and constructs
o when the independent variable has multiple levels, but not all of those variables
affect the dependent measure equally
o all the components of the reading intervention differentially affects the reading
test
8. generalizing across time
o how long it took for the outcome effect to appear and at what level was it
manifested
o how long did it take for the students to reach the reading threshold score on
the DIBELS
9. interaction of procedures and treatments
o participants who gain something tangible react differently than those that don’t
o students that get a laptop for note-taking versus students that just get
notebook paper