WattsSEDU7006-8-1
0
NORTHCENTRAL UNIVERSITY ASSIGNMENT COVER SHEET
Learner: Stephen W Watts
THIS FORM MUST BE COMPLETELY FILLED IN
Academic Integrity: All work submitted in each course must be the Learner’s own. This
includes all assignments, exams, term papers, and other projects required by the faculty mentor.
The known submission of another person’s work represented as that of the Learner’s without
properly citing the source of the work will be considered plagiarism and will result in an
unsatisfactory grade for the work submitted or for the entire course, and may result in academic
dismissal.
EDU7006-8
Dr. Theresa Thonhauser
Quantitative Research Design
1 Exploring the Logic of Experimental
Design
Assignment: Answer the following questions: (a) Jackson (2012) even-numbered Chapter
Exercises (p. 244). (b) What is the purpose of conducting an experiment? How does an
experimental design accomplish its purpose? (c) What are the advantages and disadvantages of
an experimental design in an educational study? (d) What is more important in an experimental
study, designing the study in order to make strong internal validity claims or strong external
validity claims? Why? (e) In an experiment, what is a control? What is the purpose of a control
group? Of single or multiple comparison groups? (f) What are confounds? Give an example of a
design that has three confounds. Describe three ways to alter the design to address these
confounds and explain the advantages and disadvantages of each. (g) What does “cause” mean
and why is it an important concept in research? How are correlation and causation related? (h)
You are a researcher interested in addressing the question: does smiling cause mood to rise (i.e.,
become more positive)? Sketch between-participants, within-participants, and matchedparticipants designs that address this question and discuss the advantages and disadvantages of
each to yielding data that help you answer the question. Describe and discuss each design in 4-5
sentences. Length: 5-7 pages (app. 350 words per page)
Faculty Use Only
Stephen: Thank you for your work on this paper! You have done a nice job on this first assignment.
Please see my comments throughout your paper for some additional feedback. FYI: I use the NCU rubric
for all my grading. You can find the rubric in our syllabus or the Writing Center. Thanks!
Theresa Thonhauser
Content Score: 95%
Writing Score: 93%
July 16, 2012
Running head: WattsSEDU7006-8-1
1
Exploring the Logic of Experimental Design
Stephen W. Watts
Northcentral University
WattsSEDU7006-8-1
2
Exploring the Logic of Experimental Design
Jackson Chapter Exercises
#2. In the health magazine depression study there are two confounds that might have
impacted the study and the observed findings. Due to the length of the study, history can be an
intervening factor since myriad events outside of the study treatment could effect each given
subject. Mortality could also be a huge confounder in this study. Half of the original subjects
showed improvement, and a great proportion of the other half dropped out of the study. If most
of the subjects that dropped out were the most depressed, it is impossible to determine whether
the program had the effect the researchers claimed. There was also no control group!
#4. A major motivator in research is to answer questions regarding what causes what to
happen; we want to better predict (and control) what happens in our world. When evaluating any
claim that an outcome is caused by some input, the root of the analysis should be based in
internal validity (Trochim & Donnelly, 2008). “Internal validity refers to the accuracy of
conclusions about cause and effect” (Cozby & Bates, 2012, p. 69).
A naïve but tempting motive to conduct research is to find universal truth. External
validity determines how well and the extent to which findings of a study can be generalized
beyond the bounds of the study, or to “other persons in other places and at other times” (Trochim
& Donnelly, 2008, p. 34). Universal truth would have the greatest generalizability and supreme
external validity, but the effects of chance, imperfect conceptualizations, and deficient
understanding of associations makes determining it extremely unlikely, if not impossible.
#6. The biggest similarity between a within-subjects design and a matched-subjects
design is that they are both statistically powerful, because of minimizing the variability between
participants. In the within-subjects design this is due to the same participants acting as both
WattsSEDU7006-8-1
3
control and experimental groups. For the matched-subjects design variability is minimized
because subjects are matched based on relevant measures important to the study.
There are several differences between the designs. In the within-subjects design half as
many subjects are required because each participates in all conditions, where with the matchedsubjects design, participants are randomly assigned to control and experimental groups. This
randomization and separation minimizes participants realization of what the experiment is about
and changing their response, or demand characteristics, in the matched-subjects design; a
possible confound in the within-subjects design. Another potential confound with the withinsubjects design is the carryover or testing effect of subjects being measured multiple times;
something that does not occur in a matched-subjects design.
Answers to Assignment Questions
What is the purpose of conducting an experiment? How does an experimental
design accomplish its purpose? The primary purpose of conducting an experiment is to allow
the “researcher to establish a cause-and-effect relationship” (Jackson, 2012, p. 19) by
manipulating a variable, controlling the environment, and observing the results. By controlling
the environment, experiments allow the researcher to see what happens depending on the state of
the independent variable. The purpose of the experimental design is to ensure that the
environment, and the control and experimental groups are as similar as possible, to determine if
when X happens then Y happens; when X does not happen, neither does Y (Trochim &
Donnelly, 2008).
What are the advantages and disadvantages of an experimental design in an
educational study? A true experiment has strong internal validity. Strong internal validity
allows the researcher to determine cause-and-effect. All that is required for strong internal
WattsSEDU7006-8-1
4
validity is control over the environment of the study, statistically equivalent control and
experimental groups, and a large enough sample. Unfortunately, the stronger the internal
validity, the more it limits the ability of the researcher to generalize the results outside of the
context in which it is measured. A true experiment may have limited or weak external validity
for this reason.
Experiments are about control (Jackson, 2012); controlling variables through direct
manipulation (Cozby & Bates, 2012). Cause-and-effect is easier to ascertain since extraneous
variables in the experiment are kept constant (Creswell, 2009). This allows the researcher to
determine whether a treatment is the cause of an outcome and not some other factor. There are
times or situations, however, where an experiment is not advised or ethical. Experiments are not
conducted in real-world settings because the real-world is messy, and controlling for extraneous
variables would be too difficult. Experiments have limited external validity because they
measure conditions in an artificial environment that may not be generalizable to the real-world.
Under some circumstances conducting an experiment is unethical. If the researcher is testing a
drug that can ease or eliminate the pain and suffering of subjects, or cure a malady, but only
administers it to the experimental group, while withholding it from the control group it is unjust,
and therefore unethical. What about random selection and assignment, as well as control
groups? Would those be easy or difficult to setup for an educational study?
What is more important in an experimental study, designing the study in order to
make strong internal validity claims or strong external validity claims? Why? Valid
research consists of the interplay between internal validity and external validity. Internal and
external validity exist on a continuum and have a negative correlation. A study does not either
have or not have internal or external validity; these constructs are not binary. Further, as the
WattsSEDU7006-8-1
5
internal validity of a study increases, making it more determinant, the external validity tends to
decrease making it less extendable. Conversely, the higher the generalizability the less likely a
study will engender a causal relationship. If a researcher is choosing to do an experiment, he or
she is also choosing to emphasize internal validity because that is the nature of an experiment; to
determine the effects of an independent variable on dependent variables.
In an experiment, what is a control? What is the purpose of a control group? Of
single or multiple comparison groups? In an experiment it is essential to measure the outcome
of what happens when treatment is given, while also measuring the outcome of what happens
when treatment is not given. By comparing the results of the outcomes, a determination is made
regarding whether the treatment produced the desirable outcome or not. The group that does not
receive the treatment, or the situation of not receiving the treatment, is considered the control.
The control group in some experiments receives an alternate treatment. In either case, the
control acts as the baseline from which differences in outcome are measured.
A single comparison uses a control and experimental group in order to determine if a
treatment produces a different outcome than does non-treatment. A multiple comparison group
allows multiple treatments to be compared with non-treatment and with each other. A wellknown multi-group experiment is the control-placebo-experimental design. By using the placebo
group, participation expectation effects are controlled, while a true measure of the treatment
effect can be gained between the placebo and experimental groups (Jackson, 2012).
What are confounds? Give an example of a design that has three confounds.
Describe three ways to alter the design to address these confounds and explain the
advantages and disadvantages of each. Confounding occurs when findings are ambiguous
regarding which variable caused an effect to happen. Confounding variables are those that may
WattsSEDU7006-8-1
6
not have been measured or figured in the design of the experiment. A confound is a situation
that was not controlled in the conducting of the experiment. Confounds have the effect of
lowering the ability of the researcher to determine cause-and-effect due to the extraneous
variables that were not controlled for in the experiments design.
A research design that has at least three confounds is the one-shot experimental case
study (Leedy & Ormrod, 2010). In this design the treatment is introduced and the measurement
is taken to determine the effects of the treatment. Numerous extraneous variables could have
influenced the subjects’ measure as well as the independent variable, including; (a) history
effect, (b) maturation effect, (c) diffusion of treatment, (d) experimenter effect, or (e) subject
effect.
By altering the experiment into a two-group, posttest-only, randomized experiment it is
possible to eliminate some possible confounds from the earlier design. In the earlier design it
was impossible to know if the treatment was responsible for the outcome measured. In this
design it is possible to determine that there is a difference because of application of the
treatment; randomization allows the researcher to assume probabilistic equivalency between the
groups. This design enhances internal validity because it is stronger than any single group
design, and is proof against multiple-group threats as well except mortality. This design does not
protect again social threats to internal validity (Trochim & Donnelly, 2008)
Another possibility for modifying the design is with a within-subjects design. By
utilizing the same subjects to determine both the control and experiment, the confounds
associated with possible non-equivalency of the groups are eliminated and the number of
required subjects is halved. Using this design, however, increases the possibility of demand
characteristics and testing effect.
WattsSEDU7006-8-1
7
Another design consideration is a matched-subjects design. The matched subjects design
requires that participants in the control and experimental groups be matched on variables that are
important to the results of the study. By matching participants in this way, groups can be made
not only probabilistically equivalent, but specifically equivalent in relevant areas, increasing the
power of any statistical analyses performed. Having two groups minimizes demand
characteristics, testing effect, or non-equivalent group confounds. To conduct this kind of study,
however, requires many more subjects and may be more difficult to match relevant subjects,
making it more susceptible to mortality confounds as well.
What does “cause” mean and why is it an important concept in research? How are
correlation and causation related? In research, cause identifies that a certain event, treatment,
or variable is the exclusive reason for a specific outcome. Determining the cause of a behavior
or what brings about a specific outcome is one of the primary reasons for conducting research
(Crosby & Bates, 2012). Causation in research can only be determined when three factors are
present, (a) the cause always precedes the effect, (b) the effect only occurs when preceded by the
cause, and does not occur when the cause is not present, and (c) there can be no other
explanation for the effect other than the cause (Crosby & Bates, 2012).
The relationship between correlation and causation is unidirectional. If X causes Y, X is
also perfectly correlated with Y. When X happens, so does Y. When X does not happen, neither
does Y. The inverse of this relationship is not necessarily true. Just because there is a
correlation between two variables does not mean that one causes the other. While there can be
no causation without correlation, correlation does not imply causation.
You are a researcher interested in addressing the question: does smiling cause mood
to rise (i.e., become more positive)? Sketch between-participants, within-participants, and
WattsSEDU7006-8-1
8
matched-participants designs that address this question and discuss the advantages and
disadvantages of each to yielding data that help you answer the question. Describe and
discuss each design in 4-5 sentences.
Between-subjects design. A between-subjects design involves using two randomly
selected groups of subjects from the population; in this case, human beings who can smile. Half
of the subjects are randomly selected for the control group and will use an evaluation of mood
instrument at the culmination of the experiment, but would otherwise receive no other
instructions regarding the relevant purpose of the study. The other half would be assigned to the
experimental group and instructed to smile throughout the experiment. The same mood
evaluation instrument would be administered at the end of the experiment.
Due to the random assignment of subjects it is assumed that the two groups are
probabilistically equivalent, although there is the understanding that some degree of error is
associated with that assumption. The two groups allows for comparison of results, so that the
researcher can determine if there is a positive effect on affect between the experimental group
and the control group. Two groups with no pretest minimizes the likelihood that the confounds
of testing effects, or demand characteristics will effect the results. The primary disadvantages of
the between-subjects design is the need to have sufficient participants to fully populate two (or
more) groups, making it more time consuming.
Within-subjects design. A within-subjects design involves using a single group of
subjects selected from the population. In this design all subjects would be instructed that for the
first half of the experiment they are to either smile or not, and will be selected for which to do
first by random assignment, and then they would take the mood assessment. After the first
WattsSEDU7006-8-1
9
assessment subjects would receive the opposite instructions as before the first assessment, and at
the end of the experiment would again take the mood assessment.
Because the same subjects are measured in the two conditions the possibility of
differences between groups is completely eliminated; making statistical analyses much more
powerful. The within-subjects design also maximizes the number of usable subjects in the study
since there is no need for a control group, minimizing the time for conducting the experiment.
The primary disadvantage of this design is that the assessment must be administered twice, so it
is more likely that the result could be confounded due to testing effects, or demand
characteristics.
Matched-subjects design. A matched-subjects design involves using two randomly
selected groups of subjects from the population, but the participants are also matched on control
variables that are deemed relevant to the subject; in this case, subjects may be matched based on
gender, cultural-identity, and age range. Other than matching subjects in these relevant
variables, subjects will be assigned to groups and the experiment conducted as in the betweensubjects design. The matched-subjects design has all of the advantages of the between-subjects
design, while further strengthening that design because of the matching. The matching allows
for a stronger assumption of equity between the control and experimental groups. The primary
disadvantage of this design is that because subjects must be matched in each group according to
relevant variables, more subjects are required and some volunteers may not be able to
participate??, and the loss of one subject in either group eliminates the matched subject as well,
so this design is more susceptible to the mortality confound.
WattsSEDU7006-8-1
10
References
Cozby, P. C., & Bates, S. C. (2012). Methods in behavioral research (11th ed.). Boston, MA:
McGraw Hill Higher Education.
Creswell, J. W. (2009). Research design: Qualitative, quantitative, and mixed methods
approaches (3rd ed.). Thousand Oaks, CA: SAGE.
Jackson, S. L. (2012). Research methods and statistics: A critical thinking approach (4th ed.).
Belmont, CA: Wadsworth Cengage Learning.
Leedy, P. D., & Ormrod, J. E. (2010). Practical research: Planning and design. Upper Saddle
River, NJ: Merrill.
Trochim, W. M. K., & Donnelly, J. P. (2008). The research methods knowledge base (3rd ed.).
Mason, OH: Cengage Learning.