Study Questions for Chapter 2 of Shadish et al.
“Statistical Conclusion Validity and Internal Validity”
p. 33 Shadish et al. introduce the concept of validity by telling the story of a famous
horse. Briefly tell this story.
p. 34 Validity is a property of _______________________________________
p. 35
________________________ theory says that a claim is true if it corresponds to
the world.
p. 35 ________________________ theory says that a claim is true if it belongs to a
coherent set of claims.
p. 35. _________________________ is a theory of truth that says that a claim is true if it
is useful to believe that claim.
pp. 35-36 Shadish et al. identify three dominant theories of truth.
Name and explain each of these theories. Which of these three theories do Shadish et al.
endorse as the one that is most likely correct? Which of these three theories do Shadish
et al. consider to be "compromised"?
pp. 37-39 What are the four validity types identified by Shadish et al that correspond to
four questions that researchers face when interpreting causal studies? For each type of
validity, state the corresponding question that researchers face when interpreting causal
studies (see page 39).
p. 39 __________________________________________ are specific reasons why we
can be partly or completely wrong when we make an inference about covariance, about
causation, about constructs, or about whether the causal relationship holds over variations
in persons, settings, treatments, and outcomes.
p. 40 What is the main method that Shadish et al. recommend for ruling out potential
threats to validity?
p. 40 If threats to validity cannot be ruled out by design controls, Shadish et al.
recommend that researchers identify and explore the role and influence of the threats in
the study. In doing this, three questions are critical. What are these three questions?
pp. 40-41 Shadish et al. give an example about how their "three critical questions" could
be used to identify and explore "history" as a threat to the validity of the WIC program.
Discuss this example in detail, explaining how each of the three questions is addressed by
Shadish et al.
p. 41 Shadish et al. state that because all researchers have difficulty criticizing their own
work, what is the most common source of identified threats to the validity of studies?
p. 41 Why do Shadish et al. prefer design control over statistical control to rule out
threats to validity?
p. 41 Without a theory that identifies possible threats to validity, most researchers rely on
___________________________________ to identify such threats.
p. 42 Statistical conclusion validity concerns two related statistical inferences. What are
these two inferences?
p. 43 Most criticisms of NHST reduce to two. What are these two?
p. 43. In the debate about NHST, how do most parties prefer that results be reported?
pp. 45-48 Suppose that of friend of yours has designed an experiment and estimated its
power, which is .30.
(a) What is the probability that this study will fail to find a significant effect, assuming
that an effect really exists?
(b) What is the power that is generally recommended for experiments in psychology?
(c) What practical advice should you give to your friend?
pp. 46-47 Be able to name, explain, and give concrete examples of the methods to
increase power listed in Table 2.3 on pages 46 and 47 of Shadish et al.
pp. 46-47 Name, explain, and give concrete examples of THREE methods (other than
increasing the level of alpha) to increase power, that are listed in Table 2.3 of Shadish et
al.
p. 49 What are the two reasons that Shadish et al. believe that a complete reporting of
results in research reports is unlikely:
p. 49 What is the likely outcome of the practice of incomplete reporting of research
results?
p. 49 Unreliability always ______________________________ univariate relationships.
p. 49 What is the effect of unreliability if three or more variables are involved?
p. 49 Name any TWO remedies suggested by Shadish et al. for unreliability of
measurement.
pp. 49-50 Floor and ceiling effects are examples of what threat to statistical conclusion
validity?
pp. 49-51 How does restriction of range affect power and bivariate relationships?
p. 50 What are floor and ceiling effects? Give an example of each.
p. 50 Why is it undesirable to dichotomize or trichotomize continuous variables?
p. 50 If you are thinking about using a particular measure, but are afraid that it may have
floor or ceiling effects in your subjects, how could you deal with this?
p. 51 Suppose you are conducting a field experiment in which features of the
experimental setting may fluctuate and adversely affect the treatment. If you cannot
control these factors, but suspect that they exist, what do Shadish et al. suggest that you
do?
p. 51 Suppose that your subjects are very heterogeneous. What effect will this have on
your ability to detect an effect? How do Shadish et al. suggest that you might deal with
this problem?
pp. 52-53 Explain the maxim that "we cannot prove the null hypothesis". What might
you do if you need to know if two treatments are equivalent?
p. 53 What form of validity refers to inferences about whether observed covariation
between A and B reflects a causal relationship from A to B in the form in which the
variables were manipulated or measured?
p. 54 To support inferences that A causes B, what three things must the researcher show?
p. 54 Campbell (1986) proposed re-labeling internal validity as...
p. 54 Explain the implications of each of the first three words in the term "local molar
causal validity"
p. 54 What is a shorter term for "local molar causal validity?
p. 54 Mackie (1974) stated "Typically we infer from an effect to a cause (inus condition)
by eliminating other possible causes." Shadish et al. say that another term for these
"other possible causes is"
p. 55 What is a bidirectional or reciprocal causal relationship? Give a concrete example.
pp. 56-57 Define and give an example of each of the following threats to internal
validity: Selection History Maturation
p. 56 What is the most effective (although not always the most practical) design strategy
for eliminating selection bias?
p. 56 In a field study without random assignment, how might you reduce the plausibility
of history as a threat to internal validity?
p. 57 In a field study without random assignment, how can you reduce maturation as a
threat to internal validity?
p. 57 If research participants are selected or select themselves because they had scores
that were higher or lower than average, what threat to internal validity (besides selection)
becomes more plausible?
p.58 If the participants in a study are selected precisely because they have extreme scores
on a variable, what is the best solution?
p. 59 What procedures involving measurement at multiple time points can help identify
and reduce regression effects?
p. 59 A teacher has received a list of students who received a D average during the last
quarter. Her job is to select half of these students and tutor them. If their grade goes up
to C during the next quarter, she will receive a bonus. If the teacher wants to maximize
her chances of receiving a bonus, how can she use the principle of "regression to the
mean" to select the students she will tutor?
p. 59 What is another, shorter name for "experimental mortality"?
p. 60 Explain the difference between the validity threats of (1) testing and (2)
instrumentation. Give a concrete example of each.
p. 61 Which treats to internal validity are eliminated by randomization?
Which additional threats are eliminated by randomizing subjects AND measuring groups
equivalently?
p. 62 Given random assignment, inferential problems about causation arise only in TWO
situations. What are they? What are the ways to manage these two problems?
p. 62 What is "response burden"? Give a concrete example.