First the multi choice, then the complex one.




Reliability: 0.77
Average: 17.35/22 (78.86%)
High score: 21/22 (95%)
Low Score: 11/22 (36%)
2014
13.3 (60.6%)
17 (77%)
8 (36%)

This type of validity reflects the extent to
which the study accurately measures what it
purports to measure.
a. Construct (65%)
b. Internal
c. External
d. Conclusion

This type of validity reflects the extent to
which the findings of a study can be applied to
other people, places, or times.
a. Construct
b. Internal
c. External (85%)
d. Conclusion

This type of validity reflects the extent to
which the data in a study supports a proposed
relationship.
a. Construct
b. Internal
c. External
d. Conclusion (25%)

This type of validity reflects the extent to
which the study establishes evidence of a
cause-effect relationship.
a. Construct
b. Internal (64%)
c. External
d. Conclusion

The independent variable is the
a. Cause variable
b. Effect variable
c. Variable the experimenter manipulates
d. Variable reflecting the study outcome
e. a and c (95%)
f. b and d

The dependent variable is the
a. Cause variable
b. Effect variable
c. Variable the experimenter manipulates
d. Variable reflecting the study outcome
e. a and c
f. b and d (95%)

Which of the following terms can be used
most interchangeably (i.e. they mean similar
things)
a. Cross-sectional and longitudinal designs
b. Cross-sectional and repeated measures designs
c. Longitudinal and repeated measures designs
(46%)

Which of the following types of research is
concerned with information from only one
variable at a time?
a. Descriptive (90%)
b. Relational
c. Causal

Which of the following types of research is
concerned with how variables interact with
each other?
a. Descriptive
b. Relational (100%)
c. Causal

Which of the following types of research is
concerned with the effect of one variable on
another?
a. Descriptive
b. Relational
c. Causal (100%)

Let’s say that as GRE score goes up, graduate
GPA goes down. What kind of relationship is
this?
a. No relationship
b. Positive
c. Negative (77%)
d. Curvilinear

Say you find that socio-economic status and
mood state were very strongly correlated.
Does this mean they are causally related?
a. Yes
b. No (77%)

What is the difference between the sampling
model and the proximal similarity model?
a. The sampling model is more difficult to obtain
external validity with.
b. The proximal similarity model considers not just
people but settings and times. (35%)
c. The proximal similarity model forces us to consider
random sampling more frequently.
d. All of the above.

When you have a construct in mind, and do
something in an experiment to try to
represent that construct, you _____________
it.
a. model
b. operationalize (78%)
c. validate
d. generalize

It looks like the researcher included a lot of features
in the way they represented the construct that are
argued for in the research. This suggests good
a. Face validity
b. Content validity (11%)
c. Predictive validity
d. Concurrent validity
e. Convergent validity
f. Discriminant validity

The researcher showed that the operationalization of
the construct behaved similarly to another similar
operationalization, thus demonstrating good
a. Face validity
b. Content validity
c. Predictive validity
d. Concurrent validity
e. Convergent validity (11%)
f. Discriminant validity

You think that the construct sounds right. This
means it probably has reasonable
a. Face validity (56%)
b. Content validity
c. Predictive validity
d. Concurrent validity
e. Convergent validity
f. Discriminant validity

You are concerned that a construct has been measured so
frequently that the measurement itself has provided practice
at the task being tested. You are concerned with
a. Inadequate preoperational explication
b. Mono operation bias
c. Mono method bias
d. Interaction of different treatments
e. Interaction of testing and treatment (78%)
f. Restricted generalizability
g. Confounding constructs and levels of constructs
h. Social threats

You are concerned that a construct has not been sufficiently
understood by the researchers, and so is not well
operationalized. You are concerned with
a. Inadequate preoperational explication (67%)
b. Mono operation bias
c. Mono method bias
d. Interaction of different treatments
e. Interaction of testing and treatment
f. Restricted generalizability
g. Confounding constructs and levels of constructs
h. Social threats

What is the recommended way of controlling
for (reducing) social interaction threats?
a. Add one or more control groups
b. Use random assignment
c. Use blind/double blind procedures (33%)
d. Use random selection

What is the recommended way of controlling
for (reducing) single group threats?
a. Add one or more control groups (78%)
b. Use random assignment
c. Use blind/double blind procedures
d. Use random selection

What is the recommended way of controlling
for (reducing) multiple group threats?
a. Add one or more control groups
b. Use random assignment (33%)
c. Use blind/double blind procedures
d. Use random selection
A professor was curious about the impact of student diet, especially sugar
and caffeine, on test performance. She invited students from her classes to
participate in a study before their mid-term exams. Forty students accepted
the invitation to participate. The professor randomly assigned them to her
experimental groups. The students in one group were told not to change
their food and drink consumption during the week before their exams but to
record everything they ate and drank in a diary. The students in the second
group were also asked to record their nutritional intake in the same kind of
diary, but were also given unlimited free coffee at the campus coffee shop.
The students in the third group received both free coffee and free donuts at
the coffee shop, and again were asked to record everything they ate and
drank in a diary. The professor planned to use the average of their mid-term
grades in her classes as the dependent variable.

a) the independent variable
 Diet (3 levels: normal vs caffeine vs sugar and
caffeine) (100%)

b) the dependent variable
 Test performance (100%)

c) the study design using design notation
R
X1 O
R
X2 O
R
X3 O
(80%)

From the rubric questions:
 What is the relationship being studied?
▪ A professor was curious about the impact of student
diet, especially sugar and caffeine, on test performance
▪ This will provide the starting and finishing point for all
critiques (the next would too, usually, but as you’ll see...)
 What are the study findings, if any?
▪ None – no data collected yet.

What is the sampling strategy?
 Well, she is a professor, and “She invited students from her
classes to participate in a study before their mid-term
exams. Forty students accepted the invitation to
participate.”
 SO:
1.
2.
3.
It is non-probabilistic
It is a sample of convenience
It is a volunteer study (not all will have volunteered)

To what population/time/setting is the study being
generalized/targeted (either implied or actual)? (Look for author
wording confirming the attempted generalization)
 A professor was curious about the impact of student diet,
especially sugar and caffeine, on test performance
▪ The only qualifiers are that she is interested in students’ diets, and she
is interested in its effects on test performance only.
▪ So there is something about population (students) and something
about setting/timing (effects on test performance – so being applied to
when they are taking a test)
▪ The test performance will be taken care of by construct validity (it’s one of the
constructs in the study, so doesn’t need to be considered here)...so we’ll restrict
ourselves to the population issue here.
▪ Also all recorded their food consumption in a diary (part of study
settings)

What does the sampling strategy and the actual time and
settings of the study imply about the merits of any
generalization found in (b) above? (Is there a good match, or
do you see some mismatches?)
 Ok, now we get to the crux of it. Does allowing volunteers from her
classes in some way provide a biased sample for the study, such that
these students do not adequately represent other students? Or in
other settings?
▪ Clearly there are potential problems here. It is very likely that this sample has
systematic differences from the general population due to
▪ All participants are from one college, one department, one instructor’s classes
▪ All participants are volunteers
▪ All recorded their food intake (general population does not, explicitly anyway)

To what extent do you think there is a problem of
generalization? In other words, do you think the relationship
under investigation (see 1.B.), or the results of the study (see
1.C.) would have potential to change given any of the issues
raised in 2.C?
 To answer this question, we have to consider:
▪ What moderator variables might have been unwittingly introduced by the sampling
strategy
▪ Whether these moderators could interact with the study variables
 So what moderators do we have?
▪ SES? Education level?
▪ Interest in research? Responsibility?
▪ Overall awareness/recall of diet?
Based on one college etc.
Based on volunteerism.
Based on diaries.

But ultimately the goal is to provide a rationale as to
whether your moderators would in fact interact with the
study findings (have the potential to change them)
 SES? Education level?
▪ Could this change the results of a study looking at diet and performance?
▪ (e.g. if these students were more or less than typically capable, would they be more
or less susceptible to effects of sugar and/or caffeine on test performance? Well –
maybe there’s something to do with test anxiety here? Or competence? Or both?)
 Interest in research? Responsibility?
Should be
considered
but a
definitive
answer is
not needed
for an “A”
▪ Could this change the results of a study looking at diet and performance?
▪ (e.g. if people are more interested in research, are they more likely to be the kind of
disciplined student whose preparation for the test is diligent enough to allay crises of
confidence? Or are they likely to be precisely the people who have such crises of
confidence, and therefore be more likely to participate...and be more sensitive to
the effects of caffeine/sugar dosing)

What are the constructs in the study intended to measure?
 State the idea(s) behind the dependent measures
 State the idea(s) behind the independent measures

Test Performance:
 Likely, a capacity to produce competent academic work in a stressful
situation.
 Type of work/test? Duration?

Diet (especially sugar and caffeine):
 Probably getting at energy highs and lows, and memory and cognition type
effects, and the supposed effects of S & C
 Sugar and caffeine...what are the supposed effects? When do they exert their
influence? Long term? Short term? Dose?

Likely you’d have further questions about these 2 from a “full paper”

How is each construct in the study operationalized?
 State how dependent measures are operationalized
 State how independent measures are operationalized

Dependent measure:
 Average of mid-term grades

Independent measure:
 Influenced via free coffee, doughnuts (depending on group
assignment) at campus coffee shop.
 Self-reported via diary

What are the strengths and/or weaknesses of the observed
operationalizations?
 Consider the types of validity: face, content, etc.
 Consider the laundry list of threats.
 Do so for both constructs!

Mid-term grades representing test performance
 Has face validity...looks right. Details? What kind of test was the mid-
term? This was only one kind of test...how hard was it? Multi-choice?
Essay? Timed? Untimed? Level of class (es)? Need more details!

Diet and free coffee/doughnuts
 Did they actually partake? Did all of them? On a consistent basis? What we
really expect to see are elevated levels of caffeine and or sugar in the diet
(for the treatment groups). Is this going to achieve it?
 No control of dose, so potentially very weak operationalization.
 Self-report has recall/honesty issues


Is the construct validity issue as described in previous slide
likely to alter the results of the study?
Midterm grades:
 Could. What does the effect depend on? Exam stress and it’s
response? If so, an easy test might mean a null response, a difficult
test an exaggerated one (unable to focus…more stress..etc).

Diet:
 Could: If the resultant dosing was too low on average across either
group, then null results might come out. Might also get great
variability within groups.
 If accuracy of diary method is serious, might also get discrepancies (if
factoring into analysis by controlling for reported level of dose)
Should be considered, but a definitive answer
is not needed for an “A”

What is the study’s design?
R
R
R
X1 O
X2 O
X3 O

Does the study’s design establish temporal
precedence?
 Yes.

Does the study’s design establish covariation
of cause and effect?
 Yes, partly and potentially.
▪ There is an available comparison of normal versus raised
coffee intake. (However, of course, they may not have
drunk more)
▪ The potential for increased sugar intake is only available
when paired with potential increased caffeine intake –
so covariation is compromised for that variable.

To what extent does the study’s design control for
alternate interpretations of the causal relationship?
Does it control for:
 Single group threats?
▪ yes
 Multiple group threats?
▪ Yes (random assignment – and 40 seems an adequate number,
though you could argue this.).
 Social interaction threats?
▪ Nope.

Using your response to A-D, to what extent is the
assertion that the relationship under investigation is
causal a reasonable one?
 While groups were probably equal at the start of the
experiment, the fact that they all attended the same
classes suggests they may have become aware of the
alternate conditions of the experiment.
 The nature of the conditions was such that numerous
social issues could have arisen.

In the event of observed weak internal validity, can
you suggest an alternative causal interpretation?
 Can the social issues change the results of the
experiment?
▪ Possibly.
▪ Frustration and feeling cheated could conceivably result in people
in the control group over-compensating by trying harder, or feeling
aggrieved and getting extra stressed and doing poorly.
▪ Note the different pattern of results these possibilities would result in.
Should be considered, but a definitive answer
is not needed for an “A”