Quantitative Methods
Elke Johanna de Buhr, PhD
Tulane University
Textbook Chapters
• Creswell, Chapter 8
• (Salkind, Chapters 9, 11, 12)
Your Research Proposal
I. Introduction
•
•
•
•
A. Problem statement
B. Research question(s)
C. Hypothesis
D. Definitions of terms
II. Review of the relevant literature (the more complete, the better)
• A. Importance of the question being asked
• B. Current status of the topic
• C. Relationship between the literature and the problem statement
III. Method
•
•
•
•
A. Target population
B. Research design and sampling
C. Data collection plans
D. Proposed analysis of the data
IV. Implications and limitations
Research Proposal: Part I
I. Introduction
• A. Problem statement
• B. Research question(s)
• C. Hypothesis
• D. Definitions of terms
Research Proposal: Part II
II. Review of the relevant
literature (the more complete,
the better)
• A. Importance of the question being
asked
• B. Current status of the topic
• C. Relationship between the
literature and the problem
statement
Research Proposal: Part III
III. Method
• A. Target population
• B. Research design and sampling
• C. Data collection plans
• D. Proposed analysis of the data
Research Proposal: Part IV
IV. Implications and limitations
(Section discussing generalizability,
reliability and validity of the
collected data.)
Chapter Eight:
Quantitative Methods
Chapter Outline
• Defining Surveys and Experiments
• Components of a Survey Method Plan
• The Survey Design
• The Population and Sample
• Instrumentation
• Variables in the Study
• Data Analysis and Interpretation
• Components of an Experimental Method Plan
• Participants
• Variables
• Instrumentation and Materials
• Experimental Procedures
• Threats to Validity
• The Procedure
• Data Analysis
• Interpreting Results
Defining Surveys and Experiments
• Survey Design
• To provide a quantitative or numeric description of
trends, attitudes, or opinions of a population by
studying a sample of that population
• Experimental Design
• This approach tests the impact of a treatment or
intervention on an outcome, controlling for all
other factors that may influence that outcome
• The sample is identified and generalizations are
made to the population
Components of a Survey
Method Plan
• The Survey Design
• The Population and Sample
• Instrumentation
• Variables in the Study
• Data Analysis and Interpretation
Checklist of Questions for Designing a
Survey Method
1. Is the purpose of the survey stated?
2. Are the reasons for choosing the design mentioned?
3. Is the nature of the survey (cross-sectional vs.
longitudinal) identified?
4. Is the population and its size mentioned?
5. Will the population be stratified? If so how?
6. How many people will be in the sample? On what basis
was this size chosen?
7. What will be the procedure for sampling these
individuals (e.g. random, nonrandom)?
Checklist of Questions for Designing a
Survey Method
8. What instrument will be used in the survey? Who
developed the instrument?
9. What are the content areas addressed in the
survey? The scales?
10. What procedure will be used to pilot or field-test
the survey?
11. What is the timeline for administering the survey?
12. What are the variables in the study?
13. How do these variables cross-reference with the
research questions and items on the survey?
Checklist of Questions for Designing a
Survey Method
14. What specific steps will be taken in data analysis to
do the following:
a)
b)
c)
d)
e)
f)
Analyze returns?
Check for response bias?
Conduct a descriptive analysis?
Collapse items into scales?
Check for reliability of scales?
Run inferential statistics to answer research questions
or assess practical implications of the results?
g) How will the results be interpreted?
A Survey Method Plan
The Survey Design
• Provide a purpose and rationale for
using a survey for the proposed study
• Indicate why a survey is the preferred
type of data collection procedure for
the study, examine the advantages of
this design for the study
A Survey Method Plan
The Survey Design (cont.)
• Indicate the type of survey design:
• Cross-sectional (data collected at one
point in time)
• Longitudinal (data collected over time)
• Specify the form of data collection (telephone,
mail, personal/group interviews, internet
surveys)
• Examine the strengths and weaknesses for the
data collection method(s) chosen
A Survey Method Plan
The Population and Sample
• Identify the population in the study, state the size
of the population, means of identifying individuals
in the population and availability of sample frames
• Specify if the sampling design will be:
• Single-stage (there is access to the names of the
people in the population and they may be sampled
directly)
• Multi-stage (will identify groups or organizations
who will assist in the gathering the names of
individuals who will be sampled)
A Survey Method Plan
The Population and Sample (cont.)
• Identify the selection process for the
sample
• Probability: Randomized sampling e.g.
random sampling
• Nonprobability: Nonrandomized sampling
e.g. convenience sampling
• Indicate if the population and subsequent
sample will be stratified based on specific
population characteristics e.g. age, gender,
income levels or education
A Survey Method Plan
The Population and Sample (cont.)
• Discuss how sample will be selected from
available lists
• Indicate the number of people in the sample
and the procedures used to compute this
number
• Here attention must be given to the
percentage/fraction of the population that
will be included in the sample, margin of
error, confidence interval, confidence level,
and response rate
A Survey Method Plan
Instrumentation
• Name the survey instrument used to collect data
(designed or modified for this research)
• When using an existing instrument, describe the
established validity scores from previous use. Validity
scores in a survey identifies whether the instrument
is a good one to be used:
a) Content validity
b) Predictive or concurrent validity
c) Construct validity
A Survey Method Plan
Instrumentation (cont.)
• Mention whether scores resulting from past use
of the instrument demonstrate reliability
a) Test-retest correlations
b) Consistency in test administration and scoring
• When one modifies or combines an
instrument the original validity and reliability
may not hold for the new instrument,
reestablish validity and reliability in data
analysis
• Include sample items from the instrument so
that the readers can see the actual items used
A Survey Method Plan
Instrumentation (cont.)
• Label the major content sections in the instrument:
a) Cover letter
b) Items: demographics, attitude items, behavior items,
factual items
c) Closing instructions
d) Type of scales used to measure the items: strongly
agree to strongly disagree
• Discuss the pilot testing and field-test the survey
• For mailed survey, identify steps for administering
the survey and for following up
A Survey Method Plan
Variables in the Study
• Useful in the methods section to relate the
variables to research questions and items on
the instrument
• Allows the reader to easily determine how the
data collection connects to the variables and
question or hypotheses
• Allows for cross-referencing the variables, the
questions or hypotheses, and specific survey
items
A Survey Method Plan
A Survey Method Plan
Data Analysis and Interpretation
• Present the steps for analyzing the data
• Step 1. Report response rate
• Step 2. Determine response bias: the effect of
nonresponses on survey estimates
• Step 3. Discuss plan to provide descriptive analyses
• Step 4. Check instrument's scales
• Step 5. Statistics and statistical computer program for
inferential statistical analyses (see Table 8.3)
• Step 6. Present and interpret results
A Survey Method Plan
Data Analysis and Interpretation
(cont.)
• Report how the results answered the
research question or hypotheses
• Discuss the implications for the results for
practice or future research on the topic,
draw inferences and conclusions from
results
Criteria for Choosing Select Statistical Tests
An Experimental Method Plan
• Experimental Design
• To test the impact of a treatment on an outcome,
controlling for other factors that might influence
that outcome
• Components of an Experimental Method
Plan
• Participants
• Materials
• Procedures
• Measures
Checklist of Questions for Designing an
Experimental Method Plan
1.Who are the participants in the study?
2.What is the population to which the results of the
participants will be generalized?
3.How are the participants selected? Was a random
selection method used?
4.How will the participants be randomly assigned? Will
they be matched? How? How many participants will
be in the experimental and control group(s)?
Checklist of Questions for Designing an
Experimental Method Plan
5. What is the dependent variable or variables
(i.e., outcome variable) in the study? How
will it be measured? Will it be measured
before and after the experiment?
6. What is the treatment condition(s)? How
was it operationalized?
7. How will variables be measured?
Checklist of Questions for Designing an
Experimental Method Plan
8. What experimental research design will be used?
What would a visual model of this design look like?
9. What instrument(s) will be used to measure the
outcomes of the study? Why was it chosen? Who
developed it? Does it have established validity and
reliability? Has permission been sought to use it?
10.What are the steps in the procedure (e.g. random
assignment of the participants to groups, collection
of demographic information, administration of
pretest, administration of treatment(s),
administration of posttest)?
Checklist of Questions for Designing an
Experimental Method Plan
11.What are potential treats to the internal and
external validity for the experimental design
and procedure? How will they be addressed?
12.Will a pilot test of the experiment be
conducted?
13.What statistics will be used to analyze the data
(e.g. descriptive and inferential)?
14.How will the results be interpreted?
An Experimental Method Plan
• Participants
• Describe the selection of participants as
either:
• Random
• Nonrandom (convenience)
• Indicate if it is a true experiment or not
• Identify other features in the experimental
design that will influence the outcome
An Experimental Method Plan
• Participants (cont.)
• Describe the assignment of participants to groups
and the procedure for determining group size
• Level of statistical significance
• The amount of power desired
• The effect size
• The experiment is planned so that the size of each
treatment group provides the greatest sensitivity
that the effect on the outcome actually is due to
the experimental manipulation in the study
An Experimental Method Plan
Variables
• Specifying the variables in an experiment
identifies the group receiving the
experimental treatment and the outcomes
being measured
• Clarify the groups
• Identify the independent variable(s), including the
treatment variable
• Identify the dependent variable(s), the outcomes
An Experimental Method Plan
• Instrumentation and Materials
• Describe the instrument(s) participants
complete in the experiment
• development, items, and scales
• reliability and validity reports of past uses
• Thoroughly discuss materials used for the
treatment
An Experimental Method Plan
• Experimental Procedures
• Identify the type of experiment
• Pre-experimental, true experiment, quasi-experiment,
and single-subject designs
• Identify the type of comparisons: within-group or
between-subject
• Provide a visual model to illustrate the research
design used
• X = treatment
• O = observation
See Examples: 8.2, 8.3, 8.4 and 8.5
An Experimental Method Plan
• The Procedure
• Describe the step-by-step procedure for the
experiment
1.
2.
3.
4.
5.
6.
Administer measures of the dependent variable or a variable closely
correlated with the dependent variable to the research participants
Assign participants to match pairs on the basis of their scores on the
measures described in in step 1
Randomly assign one member of each pair to the control and
experimental group
Expose experimental group to the treatment
Measure dependent variables to experimental and control groups
Compare performance of the experimental and control groups on
the posttest(s) using statistical significance
An Experimental Method Plan
• Data Analysis
• Report descriptive statistics (e.g., means, standard
deviations, ranges)
• Conduct inferential statistical tests (e.g., t test, ANOVA,
ANCOVA, or MANOVA)
• Use line graphs for single subject designs
• Report confidence intervals and effect sizes in addition
to statistical tests
• Interpreting Results
• Discuss results, limitations, and implications