Issues in Research
Reliability: The consistency and stability of a
measure or score.
Validity: the extent to which a measure actually
measures what it is intended to measure
• The truthfulness of a measures.
• A test can be reliable and not be valid.
• Internal validity: the extent to which changes in
the dependent variable can be attributed to the
influence of the independent variable rather than
to confounding variables.
– Degree to which researchers can draw accurate
conclusions about the effects of the
independent variable.
Issues in Research
• Construct validity: The extend to which a
measure measures the conceptual variable it is
designed to measure.
• Convergent validity: the extent to which scores
on a measure correlate with scores on a different
measure of the same construct.
• Discriminant validity: the extent to which scores
on a measure do NOT correlate with scores on a
measure of a different construct.
Developmental Research Methods
(Schmidt & Teti, 2006)
Parameters of Developmental Change:
• Age: chronological
• Cohort: group of individuals born at the same time
period experiencing the same events
• Time of Measurement: calendar time or period


Interindividual differences: differences b/w individuals
Intraindividual change: differences within an individual
over time
Cross-Sectional Designs
• At least two samples of difference ages and
cohorts measured at the same time
– Compare memory strategies of 4 and 6 year olds.
• Age group and interindividual differences
Problems/Issues:
• Does NOT assess age changes, the stability of a
characteristics over time, or intraindividual change
• Compares group means (looses individual info)
• Age by Cohort confound: results may be affected
by historical or cultural differences b/w cohorts.
• Limits external validity (generalizability)
– Historical or cultural difference between cohort
Longitudinal Designs
• Individuals of a given age and cohort are followed
over time.
– Compare how memory strategies change between 4
and 6 years of age.
• Intraindividual change and age changes
• Developmental sequences and co-occurring
social and environmental change
• Early-later relations
• Shape of developmental function
Longitudinal Designs
Issues/Problems:
• Expensive, time consuming
• Sampling biases: those who participate in study
may be different than those who don’t participate
• Populations changes over time (representative at
time 1 but not at time 2)
• Attrition (drop out, moves, death), those who drop
out typically differ from those who stay.
– Siegler and Botwinich (1979) study
• Testing and practice effects: familiar with
measures, do better over time
Longitudinal Designs
Issues/Problems:
• Instrumentation: different tests at different ages
– validity: are you measuring the same construct
• Same tests may become dated, obsolete
• Changes in study personnel
• Changes in how researchers define and measure
the IV and DV.
• Theories and hypothesizes may change, new
findings from other sources.
• Cohort Effect: unique experience of the cohort
• Age by time of measurement confound
– Difficult to separate
Longitudinal Designs
Regression to the Mean:
• Tendency for scores to regress (move towards)
the mean on a subsequent test.
• Less error variance over time
• More common when measuring extreme scores



If participants are selected because they have extreme scores
on the pretest (e.g. select a school with very poor reading
ability) there may be other factors due to measurement error
that resulted in such low scores at the pretest (tired, bad day
etc.) that may have slightly deflated their scores.
Measurement error causes extreme scores to be biased in the
extreme direction (away from the mean)
So when you test them a second time it is unlikely that you will
have those same factors that may have deflated their scores
and their scores will increase a bit and make it look like the
program has an effect.
Time-Lag Designs (Miller 1998)
• Study individuals of the same age at different
points in time
– Study 6 year olds in 2006, 2008, and 2010
• Does not give information on age changes or
differences
– validity: are you measuring the same construct
• Provides information on factors that may
confound age comparisons in other designs.
– Generational factors (cohort effects) and time of
measurement confounds.
Time of Measurement
Year of Birth (Cohort)
2006 2008 2010 2012
1998 8
10
12
14
2000 6
8
10
12
2002 4
6
8
10
2004 2
4
6
8
Cross-Sectional
Longitudinal
Time-Lag
Complex Designs
Sequential Design: combination of cross-sectional,
longitudinal, and/or time-lag designs.
• Allows researcher to separate effects of age,
cohort, and time of measurement
• Cohort-Sequential Design: follows different
cohorts over time
– Two overlapping longitudinal studies.
– E.g., 6 yr olds followed from 2005-2015, and 6 yr olds
from 2010-2020
• Age not confounded with cohort (because
different times of measurement)
• Not limited to one cohort
• Cross-sectional, longitudinal, and time-lag
dimensions
Complex Designs
Time- Sequential Design: Two or more crosssectional studies conducted at different times of
measurement.
– 6, 8 and 10 yr olds compared in 2005, 2010, and 2015.
• Age and time of measurement are not
confounded
• Complex designs are very costly, and time
consuming
Quasi-Experimental Designs
(Leary, 2004)
• When researchers can not manipulate the
independent variable, rather it is a grouping
variable (gender, age, disability) and equivalence
between the groups can not be ensured
• Researchers can not randomly assign
participants to groups, thus they lack control over
extraneous variables
• Quasi-independent variable: is not a true
independent variable but usually occurs naturally
or can not be manipulated.
• Researchers still look for effect of the quasiindependent variable.
• Quasi-experimental designs usually have lower
internal validity than true experiments.
Pretest-Postest designs
• Test participants before and after the quasiindependent variable
One group: measure participants before and after
the quasi-independent variable. Only have one
group of participants (those that experienced the
quasi-independent variable)
• Test reading before children at school A start
reading program and then test their reading after
they finish the reading program.
• O1 X O2
• Threats to internal validity
• Maturation: students may have matured over the
reading program. They may have got better at
reading just because of time and not due to the
program.
• History Effects: something other than the
independent variable may have occurred between
the pretest and posttest.
• Pretest sensitization: taking the pretest may
change the participants reaction to the posttest.
• Regression to the mean: Tendency for extreme
scores on pretest to regress (move towards) the
mean on a subsequent test (posttest).
Nonequivalent Control Group Design:
• We cannot randomly assign participants to control
and study groups, so we select a control group
that is similar to the group that gets the quasiindependent variable.
Posttest-only: measure both groups after one group
has received the treatment.
• Measure reading in School A and School B after
School A has participated in the reading program.
Quasi-experimental group
X O
Nonequivalent control group
-- O
• Selection bias: we do not know whether the two
groups were similar before the intervention
Pretest-Posttest design: Test both groups before
one groups gets the intervention, then test both
groups again after one group gets the
intervention (quasi-independent variable)
Quasi-experimental group
O1 X O2
Nonequivalent control group O1 -- O2
• Allows researchers to see if the two groups
scored similarly on the dependent variable
before the introduction of the treatment.
• To determine if the quasi-independent variable
had an effect you want scores to change
between pretest and posttest ONLY for the
Quasi-experimental group and NOT for the
Nonequivalent control group
Time Series Designs
• Measure the dependent variable many times
before and after the quasi-independent variable is
introduced
Simple interrupted time series design
• Researchers make a series of observations of the
dependent variable before and after the treatment
is introduced
O1 O2 O3 O4 X O5 O6 O7 O8
• Evidence for a treatment effect occurs when there
are abrupt changes in the time-series data at the
time the treatment was implemented
Percentage of cavities after the introduction of
fluoride into toothpaste in 1970
30
25
20
15
10
5
0
1970
1971
Percentage of cavities after the introduction of
fluoride into toothpaste in 1970
35
30
25
20
15
10
5
0
1967
1968
1969
1970
1971
1972
1973
Percentage of cavities after the introduction of
fluoride into toothpaste in 1970
35
30
25
20
15
10
5
0
1967
1968
1969
1970
1971
1972
1973
• This design helps to distinguish changes due to
maturation from the quasi-independent variable
• Contemporary History: Observed effect could still
be due to another event that occurred at the
same time as the quasi-independent variable
• Perhaps the electric toothbrush was introduced in
1970, or there was a major TV add campaign that
promoted brushing teeth.
Interrupted time series with a reversal
• Researchers measure the DV before and after the
treatment is introduced and then again after the
treatment is removed
O1 O2 O3 O4 X O5 O6 O7 O8 -X O9 O10 O11
• We can see what happens to the DV after the
quasi independent variable is introduced and then
again after it is removed.
• If the quasi-independent variable was really having
an effect we would expect performance to change
back to normal after it is removed
Percentage of cavities after the introduction of
fluoride into toothpaste in 1970
35
30
25
20
15
10
5
0
1968
1969
1970
1971
1972
1973
Limitations:
• Researchers may not have the ability to remove
the quasi-independent variable
• remove fluoride from toothpaste, remove a seatbelt
law
• Some effects of the quasi-independent variable
may remain even after it is removed
• if you did a time series study before and after
introduction of reading program, and then removed the
program, reading may not decrease, the children may
not regress because they did learn to read.
• Removal of the quasi-independent variable may
produce effects that are not due to the quasiindependent variable
Control Group Interrupted time series
• Measure more than one group on several
occasions, but only one group receives the quasiindependent variable.
O1 O2 O3 O4 X O5 O6 O7 O8
O1 O2 O3 O4 -- O5 O6 O7 O8
• Helps to rule out history effects, and we can be
more certain the a change was due to X rather
than an outside influence.
• Could still have a local history effect.
Program Evaluation
• Used to assess effectiveness of interventions (or
programs) and provide feedback to the
administrators
• Assess needs, process, outcome, and efficiency
of social services.
Descriptive Research
(Leary, 2004)
• Describe patterns of behavior, thoughts, and
emotions among a group of individuals.
• Provide information about characteristics about
the sample rather than to test hypotheses.
1) Survey: most common type of descriptive
research.
• Select a sample of the population using
predetermined questions
• Surveys are usually questionnaires or interviews
2) Demographic Research:
• describes patterns of life events and experiences
like birth, marriage, employment etc.
3) Epidemiological Research:
• study the occurrence of disease in groups of
people
• Psychologists may study prevalence of
psychological disorders.
Correlational Research
(Leary, 2004)
• Examine whether variables are related to one
another (whether they vary together).
Correlation coefficient: statistic indicating how well
two variables are related to one another (how well
they vary together) in a linear fashion.
• Must obtain a score on each variable for each
participant.
• Pearson correlation coefficient (r): most
common. Values range from -1.00 to +1.00
• The direction of the relationship is indicated by
the sign of the correlation coefficient.
• Positive correlation: indicates a direct, linear,
positive relationship (as one variable increases
the other variable also increases).
• Negative correlation: indicates a direct, linear,
negative relationship (as one variable increases
the other variable decreases)
• Magnitude of the correlation: the numerical value
(ignoring the sign) which expresses the strength
of the relation
• Correlation of .33, indicates that the variables are
not as strongly related as variables with a
correlation of .65
• The stronger the correlation the more tightly the
data cluster around the mean
• Two variables may be related in a curvilinear
fashion.
• The correlation will be 0 but the variables may still
be related in a non-linear way.
Basic Issues in Experimental
Research (Leary, 2004)
• Experimental designs allow researchers to make
cause and effect conclusions.
Three characteristics of a true experiment:
• Researcher must vary at least one independent
variable and assess its effects on a dependent
variable.
• Researcher must assign participants to
experimental conditions in a way that ensures
initial equivalence.
• Researcher must control extraneous variables
that may influence the participants’ behavior.
Assigning participants to conditions:
• Want to ensure that the participants are the same
before they are assigned to conditions, so effects
are due to the manipulation of the IV and not due
to pre-existing participant characteristics.
Between subject designs:
• Simple random assignment: Each participant has
an equal probability of being placed in each
condition.
• Matched random assignment: Test the participants
on a measure related to the dependent variable
and then assign to conditions by matching to
ensure you have the same number of people who
are high and low on the measure in each condition
Within-subjects design
• Repeated measures design: each participant
completes all conditions
• No need for random assignment
• Participants may participate in the experimental
and control group or in all the different levels of the
independent variable
• More powerful than between subjects design
• Because the participants serve as their own
controls
• Requires less participants (can have 30 who
participate in all three conditions, instead of 30
per condition making 90).
• Order effects: the order in which the levels of the
independent variable are received may affect the
participant’s behavior
• If studying memory for words under different
lighting conditions (each condition has more light)
participants may be tired by the last condition
which may reduce performance.
• Participants may show a practice effect in that they
get better at the task in subsequent conditions.
• Counterbalancing: A procedure in which the
order of conditions in a repeated-measures
design is arranged so that each condition
occurs equally often in each order.
• Carryover effects: occurs when the effects at
one level of the independent variable are still
present at another level (condition).
• Must ensure drug of one dosage wears off
before the next conditions started
Internal validity threats:
• Biased assignment of participants to conditions:
participants in each condition differ at the
beginning, so differences in the dependent
variable may reflect pre-existing differences
among the participants rather than differences
due to the independent variable
Random Assignment
ABBC
BCABCB
CAAB B
CBABB
AA B
BCB
BC
AA B
BCB
BC
Biased Assignment
ABBC
BCABCB
CAAB B
CBABB
AAA
BBB
BB
ABB
BCC
CC
• Differential attrition: participants who do not
continue in the study (drop out). Attrition can
occur at a different rate in the different conditions
• Problematic when more participants drop out of
one condition as compared to the other condition
• People who drop out may be different than those
who stay (more scared of experiment, less
motivated).
• Pretest sensitization: taking a pretest may affect
how participants behave in the experiment, so it is
hard to determine whether effect is due to the
pretest or the independent variable.
• History: history effects can effect the DV
• Testing anxiety in participants, perhaps a
participant in one group had just gone through a
very anxious situation and may be more anxious
already due to other factors than in the experiment.
• Maturation: Participants may change overtime,
may be difficult to distinguish effect of the
independent variable from maturation changes
over time.
• More problematic in research with children.
• Miscellaneous design confounds: due to
participants being treated differently in different
conditions, which results in confounding.
• Experimenter expectancy effects: researchers
may observe behavior in a biased way that
reflects what they expect to happen.
• Their expectations can distort the results
• Demand characteristics: participants may behave
differently because of noticeable aspects of the
experiment
• They may be able to guess what the researchers
are researching and act accordingly.
• Double-blind procedure: neither the participant
nor the researcher knows which condition a
participant is in.
• Helps to eliminate experimenter expectancy effects
and demand characteristics
• Placebo Effects: an artifact that occurs when
participant's expectations about what effect an
experimental manipulation is supposed to have
influence the dependent variable
• If participants think they are in a drug group they
may be more likely to say the drug produced an
effect.
• Placebo control group: receive a pill but with no
drug, so participants do not know if they are truly
receiving the drug