Variables and Control
Basic Principles of Research Design
Overview of Tonight’s Lecture

Variables and Control


Research designs are rules about controls and
variables


2 variables we can control (subject + situation)
evaluated in terms of Validity (answers the question set
to address)
 Internal Validity (relevant variables are controlled to
answer question); 8 internal validity factors
 External Validity (findings can be generalized to
other settings and populations); 4 external validity
factors
Observations are evaluated in terms of Reliability
Basic Principles of Research Design
One of the most important aspects of
research in CDIS is the careful selection of
the variables that are to be controlled or
systematically varied.
 After a problem has been identified and the
purpose specified, there must be careful
consideration of which variables are most
relevant to the problem and the purpose.

2 Purposes of Research Designs

Answer the research question

Reduce or eliminate contamination of
extraneous variables
In experimental designs, this
means ….

Manipulate the IV in order to answer the
question

Arrange the experiment so that extraneous
variables are controlled
In descriptive designs, this means ….

Select the variables for observation in order
to answer the question

make the observations in a systematic and
unobtrusive manner so that the criterion
variables are not confounded by the
extraneous variables
RESEARCH DESIGNS are the
rules for observation
- research designs are rules
about controls and variables and
controlled variation
Research design is described in
terms of:
systematic variation of the IV
 observation of the effects of the IV on the
DV
 systematic control of other variables that
might produce variation in the DV

Researchers systematically vary the IV by
controlling its variation
 a detailed explanation of controls and
variables is essential to understanding the
basic principles or rules of research design

Control and Variability
Control is a restriction of natural variation
 represent opposite extremes of ORDER and
DISORDER
 both are essential to research
 IV is controlled variation

2 Controlled Variables in Human
Communication Research
Subject (participant) variables
 characteristics of subjects
 Situation variables
 characteristics of the situation in which
the observations are made

In carefully controlled studies, subject
and situation variables are controlled.
- only the phenomena of interest are
allowed to vary.
- Control of the subject and
situation variables permits researchers to
study the effects of the variables in
which they are interested.
Controls are essential for achieving the
purposes of research.
- if the variables under study are not
isolated by controlling the relevant
variables, alternative explanations of the
findings cannot be ruled out.
- the extent to which each kind of
variation is controlled depends on the
research design.
Control of Subject Variables
The type of communication disorder to be
studied is controlled by the selection of
subjects who have that type of disorder
 Subject variables (e.g., age, gender, history
of tx, severity) are controlled by holding
them constant
 If these variables were not controlled, the
effects of the disorder might be confounded
by the effects of the uncontrolled variables

Control of Situation Variables
The research design also controls the
situation variables
 These include variables such as time of day
and instructions given to subjects
 The need for situational controls varies with
the type of design


EX: If you’re interested in determining how
sounds are detected in real-life situations, you
will not control background noise.
Criteria for Evaluating Research
Designs
A research design is evaluated in its ability
to achieve its 2 purposes (answer the
question AND reduce/eliminate
contamination of extraneous variables) in
terms of INTERNAL and EXTERNAL
VALIDITY
 Therefore, research designs are evaluated in
terms of VALIDITY (internal and external)

Internal Validity
Ability to isolate effects of the IV
 means that we can believe that the
manipulation of the IV did cause the
observed change in the DV
 a study is internally valid when relevant
variables have been controlled and the only
variables that affect the DV’s are the IV’s

Internal Validity (con’t)
When uncontrolled variables can affect the
DV, the effects of the IV are confounded
with the effects of the uncontrolled
variables
 internal validity indicates that the research
design accomplished what it was suppose to
accomplish, i.e., answer the question.

External Validity




Means that the findings from the study can be
generalized to other populations and settings
external validity is the extent to which the
effects of the IVs on the DVs in the research
situation apply to the natural settings.
Indicates the degree to which the findings from
the study can be generalized
if findings from a study do not external
validity, they may not be of any practical use
or value.
The primary thing in evaluating
experimental research is whether you
can believe with enough confidence that
the change in the DV was caused by
manipulation of the IV
- the fewer the alternative
explanations for the observed change in
the DV, the greater the internal validity
of the study.
8 Factors that Affect Internal
Validity
history
 maturation
 test-practice effects
 instrumentation
 statistical regression
 differential selection of subjects
 mortality
 Hawthorne effect

History
Events that occur between the first and
subsequent measurements in addition to the
experimental variable (“in the gap”)
 these are the events that occur outside the
experimental setting and therefore are
outside the control of the experimenter
 EX: prior treatment received in another
setting; history of ear infections

History (con’t)

Descriptive studies: longitudinal studies
more susceptible to history effects than
cross-sectional studies

Experimental studies: one group pretestposttest design is susceptible to history
effects
Maturation
Changes in the subjects that cannot be
controlled by the experimenter (“time-tied”)
 these changes may cause effects that are
attributed incorrectly to the experimental
variable
 EX: age changes, changes in biological
or psychological processes that take place
over time; even short-term changes, such
as changes in motivation, interest,
attention

Solution to maturational effects
in studies:
Keep experimental sessions relatively short
 randomize or counterbalance conditions
 randomly assign subjects to experimental
and control groups

Test-Practice Effects

The effect of taking a test may influence the
subjects’ performance when they retake the
test
 practice with the test, familiarity with the
test items, reduction of test anxiety, etc.
 pretest-posttest designs are especially
vulnerable to this problem
 must be aware of the effect of testing on
behavior when evaluating findings from
research
Instrumentation

Changes in the calibration of an instrument
or changes in the observers or scorers may
produce changes in the observed
measurements.

EX: (a) faulty or changing calibration of
equipment; (b) rating scales, questionnaires,
attitude inventories that have not been standardized
or are not reliable; (c) biases of judges or observers
used to collect data for the study; (d) poor test
environment which may contaminate observations
due to distractions, noise, interruptions, etc.
Statistical Regression



The migration of extreme scores toward the mean
atypical scores become more typical (elastic band)
 did change in performance occur because of tx
or because of regression to the mean?
occurs because subjects elected on basis of their
extreme scores.
 Solution: random selection of subjects and
random assignment of subjects to groups
Differential Selection of Subjects

Inherent differences between subjects assigned to
groups may account for the treatment effects
rather than the treatment itself.
 EX: SES, IQ
 Solution: subjects should be equal on
important dimensions prior to the study and
systematic differences between comparison
groups must be minimized with the exception
of the criterion variable under study. Further,
there should be no overlap between groups on
the variable that presumably distinguishes the
Mortality

Differential loss of subjects between
experimental and control groups or between
comparison groups
 EX: subjects who drop out may be
different in important respects from those
subjects who continue to participate in
the study.
Hawthorne Effect

Changes in a subject’s behavior that occur
because the subject knows he/she is
participating in a study (“on stage”)
 Solution: apply Hawthorne effect to
control group; begin sessions with a brief
period of acclimation to establish rapport.
Finally, you can have an
interaction of any one or
combination of the above factors
that affect internal validity.
External Validity (Generalizability)




External validity is qualitatively different from
internal validity.
If a study is internally invalid, then the results are
meaningless.
If, however, the study is externally invalid, the
results are only limited to the degree to which the
internally valid results can be generalized
A study can be internally valid, but externally
invalid. However, a study cannot be externally
valid, but internally invalid
4 Factors that Affect External
Validity
Subject selection
 reactive or interactive effects of pretesting
 reactive arrangements
 multiple treatment interference

Subject Selection

The representativeness of subjects selected
from the population to which the researcher
hopes to generalize the findings.
 The relevant characteristics of the
subjects that determine their degree of
representativeness should be specified in
the article to allow readers to evaluate the
generality of the results to other subjects
Subject Selection (con’t)
Investigators should report what the
relevant characteristics were in selecting the
subjects AND what the characteristics were
of the subjects on those relevant
characteristics (e.g., age, gender, severity,
etc.)
 this factor affects the generalizability of
findings to other subjects

Reactive or Interactive Effects of
Pretesting

The degree to which a reactive pretest may
interact with an IV in determining the subjects’
performance on the DV
 that is, subjects who take a reactive pretest may
react to an experimental treatment in a different
way from people who have not been exposed to
the pretest.
 The test itself may change the subjects’
responses.
Reactive or Interactive Effects of
Pretesting (con’t)
Therefore, the effect of pretesting may
limit the generalization that can be made
to people who have not been pretested
 this factor affects the generalizability of
findings to other measures and subjects

Reactive Arrangements

The degree to which the setting of the
research is reactive or interacts with the IV
in determining the subjects’ performance on
the DV.
 For example, the child may respond in a
certain way with the clinician in the
therapy room, but not in their classroom
 this factor affects the generalizability of
findings to other settings.
Multiple Treatment Interference


The degree to which various parts of multiple
treatment interact with each other in determining
the subjects’ performance on the DV.
This is likely to occur when more than one
treatment is administered to the same subjects
 this factor affects the generalizability of
findings to other treatments.
 The results of a multiple treatment study can
only be generalized to people who would
receive the same sequence and number of
treatments.
The 4 factors that affect external validity
can affect generalization in 1 of 4 ways:
Generalizability to other subjects
 generalizability to other settings
 generalizability to other measurements
 generalizability to other treatments

In sum….

Research designs are evaluated in terms of
validity (internal and external)

Observations are evaluated in terms of
reliability
 a measure is valid to the extent that it
measures what it claims to measure, and
reliable to the extent that it is consistent
and repeatable.
Reliability
Refers to the precision of measurement and
can be assessed by examining the
consistency or stability of a test or measure.
 Measurements of DVs are reliable to the
extent that the same measurements would
be obtained if the study were repeated.

Reliability of measurements can
be estimated in several ways
Test-retest reliability
 split-half reliability
 alternate form reliability
 observer reliability

Test-Retest Reliability

Assessed by obtaining the same
measurement twice and comparing the two
measurements
Split-Half Reliability
Assessed when measurements consist of a
series of items
 the items are split in half and the resulting
measurements are compared.

Alternate Form Reliability

Assessed when alternate forms of
measurement are compared
Observer reliability

Assessed when the judgement of two or
more observers are compared.
The design is evaluated in terms
of validity and the observations
are evaluated in terms of
reliability.