Study Questions for Chapter 3 of Shadish et al.
“Construct Validity and External Validity”
p. 65 What are the “twin problems” of construct validity?
p. 65 What do the initials in “utos” stand for?
p. 65 _____________________________ validity involves making inferences from the
sampling particulars of a study to the higher order constructs they represent.
p. 65 According to Shadish et al, what are four reasons that construct validity is
important?
p. 66 Shadish et al state that construct validity is fostered by careful attention to four
procedures. What are the first two of these procedures? Give a concrete example of how
a researcher might carry out these two procedures in an experiment.
p. 67 Deciding which features are prototypical depends on the context in which a
construct is to be used. Give a concrete example, other than the examples given by
Shadish et al.
p. 68 Two researchers both say that the participants in their study are “Hispanics,” but
use different ways of operationalizing who are “Hispanics.” This is an issue regarding
what kind of validity? What problems might ensue?
pp. 70-71 Be able to give examples that illustrate issues of construct validity in regards to
(a) outcome measures, (b) persons/participants, (c) treatment, (d) settings.
p. 72 Shadish et al think that in general researchers should probably pay increased
attention to construct validity across p________________ and s_____________.
Why do they think this is a good idea?
p. 72 Give an example from psychology or education of how a construct label might be
applied to the relationship among the variables in a study.
p. 73 You do not have to memorize the names of all the threats to construct validity in
Table 31. However, if you are given the name of one of these threats, be able to explain
it and give an example.
p. 74 Marks (2000) has suggested that researchers make four common errors in
explicating constructs. What are these four mistakes? Give an example of each, other
than the examples given by Shadish.
p. 77 What is meant by reactive self-report changes? What concrete “remedies” do
Shadish et al recommend for reducing the effects of such distortions of self-report?
p. 78 Rosenthal and Rosnow (1991) suggested many ways to reduce problems of
reactivity. Describe any three of their suggestions.
p. 82 Explain what can be learned from the study by R. Feldman (1968) about the
conceptualization of constructs during research.
p. 83 ____________________ concerns inferences about the extent to which a causal
relationship holds over variations in persons, settings, treatments, and outcomes.
p. 83 The researchers of the Transitional Employment Training Demonstration with
retarded participants questioned whether the results were broadly generalizable. What
form of validity were they concerned about? What were some of the specific limitations
that these researchers say to the generalizability of the study’s findings?
p. 86 Tests of the extent to which a causal relationship holds over variations in persons,
settings, treatments, and outcomes are conceptually similar to statistical tests of
__________.
p. 87 Shadish et al. list five threats to external validity. You do not have to memorize the
names of the five. However, if you are given a name, be able to explain it and give a
concrete example.
p. 89 What is a “stakeholder”?
p. 89 What strategy do Shadish et al. recommend for ensuring that likely questions about
generalizability over outcomes are anticipated in the study design?
pp. 90-91 Explain what is meant by “constancy of effect size” and “constancy of causal
direction.” Which of these two do Shadish et al favor as a criterion of the
“generalizability” of an effect? Why?
pp. 91-92 Why is random sampling of persons, treatments, outcomes and settings so rare
in most social science research?
p. 92 What is meant by “purposive sampling of heterogeneous instances? What are its
benefits?
pp. 96-102 In no more than five sentences, summarize the position of Shadish et al
regarding the priority of one type of validity over another.
p. 100 What do Shadish et al suggest as a “feasible option” for an index of the priority
that a particular type of validity has in a particular study?