Quantitative Research Methods
(II)
Dr Chen Wenli
Learning Sciences and Technologies AG
Learning Sciences Lab
National Institute of Education
Outline
 Logic of quantitative research
 Constructing hypothesis
 Types of quantitative research methods
 Survey research
 Experimental research
 Single-subject research
 Casual-comparative research
 Quantitative content analysis
 Validity and reliability in quantitative research
Experimental Research
 Characteristics of experimental research
 Experimental research design
 Experimental design
 Quasi-experimental design
 Factorial design
 Validity of experimental research
 Control of extraneous variables
Experimental Research
 Researcher applies some treatments to subjects for an
appropriate length of time and then observes the effect of the
treatments on the subjects by measuring response variables
 IV (experimental
82
or treatment variable)
80
 a condition or set of conditions
78
applied to subjects
76
 DV (response, criterion academic 74
performance 72
or outcome Variable)
70
68
 results or outcome on
66
the subjects
64
online
face to face
Condition
blended
Examples
 Quality of learning with an active versus passive
motivational set (Benware & Deci, 1984)
 Comparison of computer-assisted cooperative,
competitive, and individualistic learning (Johnson,
Johnson, & Stanne, 1986)
 The effect of a computer simulation activity versus a
hands-on activity on product creativity in technology
education (Kurt, 2001)
 The effect of language course taught with online
supplement material (Shimazu, 2005)
Characteristics
 The only type of research that directly attempts to
influence a particular variable
 The only type that, when used properly, can really test
hypotheses about cause-and-effect relationships.
 Enable researchers to go beyond description and the
identification of relationships, to at least a partial
determination of what causes them
 3 characteristics of experimental research
Manipulation of IV
 Researcher manipulate the IV
 Decide the nature of treatment/intervention (what is
going to happen to the subjects of the study)
 To whom it is to be applied
 To what extent
 When, where and how
Comparison of Groups
 At least 2 conditions are compared to assess the
effect(s) of particular conditions or “treatments” (IV)
 Experimental group (receive a treatment of some sort)
 Control group (no treatment) or comparison group (receive
different treatment)
 IV may be established in several ways:
 Presence VS absence of a particular form
 One form of variable VS another
 Varying degrees of the same form
Randomization
 Random assignment of subjects to groups
 an important ingredient in the best kinds of
experiments
 every individual who is participating in the experiment
has an equal chance of being assigned to any of the
experimental or control conditions being compared


It takes place before the experiment begins
Allows the researcher to form groups that are equivalent

Eliminate the threat of extraneous, or additional variables that
might affect the outcome of the study
Commonly Used Notation

X1 =
treatment group

X2 =
control/comparison group

O =
observation (pretest, posttest, etc.)

R
random assignment
=
Weak Experimental Designs
 One-shot case study design
 a single group is exposed to a treatment or event, and its
effects assessed.
X
Technology
O
Attitude scale to
measure interest
 One-group pretest-posttest design
 a single group is measured or observed both before and after
exposure to a treatment.
O
Pretest
X
Treatment
O
Post test
True Experimental Designs
 Randomized posttest-only control group design
 involves two groups formed by random assignment and
receiving different treatments
Treatment group
Control group
R
R
X1
X2
O
O
 Randomized pretest-posttest control group design
 differs from the randomized posttest-only control group
only in the use of a pretest
Treatment group
Control group
R
R
O
O
X1
X2
O
O
True Experimental Designs
 Randomized Solomon four-group design
 involves random assignment of subjects to four groups,
with two being pretested and two not.
Treatment group
R
O
X1
O
Control group
R
O
X2
O
Treatment group
R
X1
O
Control group
R
X2
O
» Better control the threat to internal validity
» Drawback—requires twice as many participants
Quasi-Experimental Designs
 Used in place of experimental research when random
assignment to groups is not feasible
• Posttest-only design with
nonequivalent groups
Treatment group
X1
O
Control group
X2
O
 Pretest-posttest design with
nonequivalent groups:
Treatment group O
Control group
O
X1
X2
O
O
Quasi-Experimental Designs
 Counterbalanced design: all groups are exposed to all
treatments, but in a different order
 the order in which the groups receive the treatments
should be determined randomly
 the number of groups and treatments must be equal
 Comparing the average scores fro all groups on the
posttest for each treatment
Group I
X1 O X2 O X3 O
Group II
X3 O X1 O X2 O
Group III
X2 O X3 O X1 O
Quasi-Experimental Designs
 Time-series design: involves repeated measurements
or observations over time (until scores are stable ),
both before and after treatment.
O
O
O
O
X
O
O
O
O
» Uses a single group of participants
Study B
» Examines possible changes
over time
Study A
X
Factorial Designs
 Factorial designs extend the number of relationships
that may be examined in an experimental study.
Treatment
R
O
X1
g1
O
Control
R
O
X2
g1
O
Treatment
R
O
X1
g2
O
Control
R
O
X2
g2
O
» Incorporates two or more factors
» The additional factor could be treatment variable or
subject characteristics
» Enables researcher to detect differential differences (effects
apparent only on certain combinations of levels of IVs)
A 2 X 2 factorial design…
Boy
Girl
Traditional
Group 1
Group 2
Gamebased
learning
Group 3
Group 4
A 2 X 2 factorial design
No interaction
between factors
Attitudes
toward
learning
Game
-based
Interacting
factors
Attitudes
toward
learning
Game
-based
Traditional
Traditional
Girl
Boy
Girl
Boy
Validity
• Validity: the experiment tests the variable(s) that it
purports to test
• If threats are not controlled for, they may introduce error
into the study, which will lead to misleading conclusions
• Threats to validity…
• Internal: factors other than the IV that affect the DV
• External: factors that affect the generalizability of the
study to groups and settings beyond those of the
experiment
Threats to Internal Validity
 History
 Uncontrolled event that occur during the study that may have an
influence on the observed effect other than the IV
 Maturation
 Factors that influence a participant's performance because of time
passing rather than specific incidents (e.g., the physical,
intellectual, and emotional changes that occur naturally)
 Test practice
 The effects of participants taking a test that influence how they
score on a subsequent test
 Instrumentation
 Influences on scores due to calibration changes in any instrument
that is used to measure participant performance
 Statistical regression
 Problem that occurs when participants have been assigned to
particular group on the basis of atypical or incorrect scores.
Threats to Internal Validity
 Bias in group composition
 Systematic differences between the composition of groups in addition
to the treatment under study.
 Experimental mortality
 A differential loss of participants
 Hawthorne effect
 Change in the sensitivity or performance by the participants that may
occur merely as a function of being a part of the study
 Novelty effect
 Participant interest, motivation, or engagement increases simply
because they are doing something different
 Placebo effect
 The participants receive no treatment but believe they are
Threats to External Validity
 Population-sample differences
 The degree to which the participants in a study are representative of
the population to which generalization is desired
 Artificial research arrangements
 The degree that a research setting deviates from the participant's
usual routine
 Multiple-treatment interference
 More than one treatment is administered to the same participants
and results in cumulative effects that may not be similar to the
outside world and may threaten generalization of the results
 Treatment diffusion
 The situation when different treatment groups communicate with
and learn from each other
Validity of Different Experimental Designs
Pre-Test/
Post Test
Control
Group
History
X
Maturation
X
Randomization
Pre-Testing
X
Measuring Instrument
X
Statistical Regression
X
Differential Selection
X
Experimental Mortality
X
Interaction of Factors
X
X
X
X
Pre-Testing
Differential Selection
Procedures
Multiple Treatment
Additional
Groups
X
X
X
X
Control of Extraneous Variables
 Confounding:
 the fact that the effects of the IV may intertwine with
extraneous variables, such that it is difficult to
determine the unique effects of each variable
 Common ways to control for extraneous variables
 Randomization
 Holding certain variables constant
 Matching
 Comparing homogeneous groups or subgroups
 Analysis of covariance (ANCOVA)
Single-Subject Research
 Most commonly used to study the changes in behavior an




individual exhibits after exposure to a treatment or
intervention of some sort.
Can be applied in settings where group designs are difficult
to put into play.
Involves extensive collection of data on one subject at a
time.
Primarily use line graphs to present their data and to
illustrate the effects of a particular intervention or
treatment.
Adaptations of the basic time-series design
Single-Subject Research
 A-B design
 …baseline measurements (O) are repeatedly made until
stability is established, then the treatment (X) is introduced
and an appropriate number of measurements (O) are made
during treatment implementation
O O O X O X O X O
baseline
phase
A
treatment
phase
|
B
Single-Subject Research
 Reversal (A-B-A) design
 …baseline measurements (O) are repeatedly made until stability is
established, then the treatment (X) is introduced and an appropriate
number of measurements (O) are made during treatment
implementation, followed by an appropriate number of baseline
measurements (O) to determine stability of treatment (X)
O O O X O X O X O O O
baseline
treatment
baseline
phase
phase
phase
A
|
B
|
A
Other Single-Subject Research Designs
 A-B-A-B design

2wo baseline periods are combined with two treatment periods
 B-A-B design

Used when an individual's behavior is so severe or disturbing
that a researcher cannot wait for a baseline to be established
 A-B-C-B design:

"C" condition refers to a variation of the intervention in the "B"
condition. The intervention is changed during the "C" phase
typically to control for any extra attention the subject may have
received during the "B" phase.
Threats to Validity in Single Subject
Research
 Internal Validity
 length of the baseline and intervention conditions
 the number of variables changed when moving from one condition




to another
the degree and speed of any change that occurs
whether or not the behavior returns to baseline levels
the independence of behaviors
the number of baselines
 External Validity
 weak when it comes to generalizability
 It is important to replicate single-subject studies to determine
whether they are worthy of generalization.
Controlling Threats in Single-Subject
Studies
 Single-subject designs are most effective in controlling
for subject characteristics, mortality testing, and
history threats.
 They are less effective with location, data collector
characteristics, maturation, and regression threats.
 They are especially weak when it comes to instrument
decay, data-collector bias, attitude, and
implementation threats.
Causal-Comparative Research
• Explores the possibility of cause-and-effect relationships
when experimental and quasi-experimental approaches are
not feasible
• Differs from experimental and quasi-experimental research
» IV is not manipulated (not ethical or not possible)
» Focuses first on the effect, then tries to determine
possible
 Relationships can be identified in causal-comparative study,
but causation cannot be fully established.
Steps in Causal-Comparative Research
 Formulating a problem
 Identify and define the particular phenomena of interest, and
then to consider possible causes for, or consequences of, these
phenomena.
 Selecting a sample
 Define carefully the characteristic to be studied and then to
select groups that differ in this characteristic.
 Instrumentation
 No limits to the kinds of instruments that can be used
 Design
 Select two groups that differ on a particular variable of
interest and then comparing them on another variable or
variables.
Threats to Internal Validity in
Causal-Comparative Research
 Weaknesses :
 lack of randomization
 Inability to manipulate an IV
 A major threat: the possibility of a subject selection bias.
 The procedures used to reduce this threat
 matching subjects on a related variable
 creating homogeneous subgroups
 the technique of statistical matching.
 Other threats to internal validity
 Location
 Instrumentation
 Loss of subjects.
Data Analysis in CausalComparative Studies
 The first step: construct frequency polygons.
 Means and SD are usually calculated if the variables
involved are quantitative.
 The most commonly used test is a t-test for differences
between means.
 ANCOVA is particularly useful in causal-comparative
studies.
 The results of causal-comparative studies should
always be interpreted with caution, because they do
not prove cause and effect.
Common quantitative measure in
learning and education
 Learning gain
 Post-pre
 (post-pre)/(1-pre) (Hake’s gain)
 Adjusted post score (through ANCOVA)
 Learning efficacy
 Does it help reduce time spent for problem solving?
 User’s attitude
 Teachbacks
 How well learner can teach back?
Quantitative Content Analysis
 Content analysis is a quantitative research instrument for a
systematical and intersubjective description of content
 A form of textual analysis *usually*
 Categorizes chunks of text according to Code
 Based on the principles of social science of “measuring and
counting”
 Reduces the complexity of content as it brings out the
central patterns of the coverage
 One objective is to examine large amounts of content with
statistic methods
Rough History
 Classical Content Analysis
 Used as early as the 30’s in military intelligence
 Analyzed items such as communist propaganda, and military
speeches for themes
 Created matrices searching for the number of occurrences of
particular words/phrases
 (New) Content Analysis
 Moved into Social Science Research
 Study trends in Media, Politics, and provides method for analyzing
open ended questions
 Can include visual documents as well as texts
 More of a focus on phrasal/categorical entities than simple word
counting
Procedure
The Sample
 The sample
 Which types of content?
 Which period?
 Which characteristics?
 Elements of the research instrument
 Sampling units
 Units of analysis: unit of the content on which our
“measurements” are based. The categories describe the
properties of the media content which is relevant to our
research question
Validity in Quantitative Research
 Definition: the extent to which any measuring instrument
measures what it is intended to measure
 Types of validity
 Construct Validity: examines the fit between the conceptual
definitions & operational definitions of the variables
 Content Validity : verifies that the method of measurement
actually measures the expected outcomes.
 Predictive Validity : determines the effectiveness of the
instrument as a predictor of a future event
 Statistical Conclusion Validity: concerned with whether the
conclusions about relationships and/or differences drawn
from statistical analysis are an accurate reflection of the real
world
Reliability in Quantitative Research
 Definition: refers to the accuracy and consistency of
information obtained in a study; important in interpreting
the results of statistical analyses; and refers to the
probability that the same results would be obtained with
different samples (generalizability)
 3 common methods to check reliability
 test-retest method

administering the same instrument twice to the same group of
individuals after a certain time interval has elapsed.
 equivalent-forms method
 administering two different, but equivalent, forms of an
instrument to the same group of individuals at the same time.
 internal-consistency method
 comparing responses to different sets of items that are part of
an instrument.
Summary
 Logic of quantitative research
 Constructing hypothesis
 Types of quantitative research methods
 Survey research
 Experimental research
 Single-subject research
 Casual-comparative research
 Others
 Validity and reliability in quantitative research