Non-Experimental Quantitative
Research Designs
Chapter 8
This multimedia product and its contents are protected under
copyright law. The following are prohibited by law:
any public performance or display, including transmission of any
image over a network;
preparation of any derivative work, including the extraction, in whole
or part, of any images;
any rental, lease, or lending of the program.
Copyright © Allyn & Bacon 2008
Discussion Topics

Non-experimental research designs
–
–
–

Descriptive designs
Relationship designs
Causal-comparative and Ex Post Facto designs
Using surveys in quantitative research
Copyright © Allyn & Bacon 2008
Non-Experimental Designs


Research design - the plan and structure of
research to provide a credible answer to a
research question
Purpose of non-experimental designs
–
Describe current existing characteristics such as
achievement, attitudes, relationships, etc.
Copyright © Allyn & Bacon 2008
Non-Experimental Designs

Four types of designs
–
–
Descriptive
Relationships


–
–
–
Comparative
Correlational
Causal-comparative
Ex Post Facto
Survey
Copyright © Allyn & Bacon 2008
Descriptive Designs

Studies that describe a phenomena
–
Statistical nature of the description





Frequency
Percentages
Averages
Graphs
Importance of these designs in the early
stages of the investigation of an area
Copyright © Allyn & Bacon 2008
Descriptive Designs

Criteria for evaluating descriptive studies
–
–
Conclusions about relationships should not be drawn
Participants and instruments should be described
completely
Copyright © Allyn & Bacon 2008
Relationship Designs

Two types of designs
–
–
Comparative
Correlational
Copyright © Allyn & Bacon 2008
Comparative Designs

These designs investigate the relationship of one
variable to another by examining differences on the
dependent variable between two groups of
participants
–
–
If math scores for males are significantly higher than those
for females, a relationship exists between gender and math
achievement
If the academic self-concept scores for ninth graders are
significantly different than those for twelfth graders, a
relationship exists between grade level and academic selfconcept
Copyright © Allyn & Bacon 2008
Comparative Designs

Criteria for evaluating these designs
–
–
–
–
Participants and instruments are described
completely
Criteria for identifying the different groups is clearly
stated
No inferences are made about causation
Graphs and images depict the results accurately
Copyright © Allyn & Bacon 2008
Correlational Designs

Simple correlation designs
–
–
Designs that examine the relationship between two variables
Two variables


–
Predictor and criterion
Use caution describing the variables as independent and
dependent
Examples


Math achievement and math attitudes
Teacher effectiveness and teacher efficacy
Copyright © Allyn & Bacon 2008
Correlational Designs

Simple correlation designs
–
Cautions in interpreting correlations


A relationship between two variables (e.g., achievement and
attitude) does not mean one causes the other (i.e., positive
attitudes do not cause high levels of achievement)
Possibility of low reliability of the instruments makes it
difficult to identify relationships
Copyright © Allyn & Bacon 2008
Correlational Designs

Simple correlation designs
–
Cautions in interpreting correlations


Lack of variability in scores (e.g., everyone scoring very,
very low; everyone scoring very, very high; etc.) makes
it difficult to identify relationships
Large sample sizes and/or using many variables can
identify significant relationships for statistical reasons
and not because the relationships really exist
Copyright © Allyn & Bacon 2008
Correlational Designs

Prediction designs
–
–
Designs that examine the predictive nature of the
relationships between variables
Two types of designs


Simple prediction
Multiple regression
Copyright © Allyn & Bacon 2008
Correlational Designs

Prediction designs
–
Simple predictive studies


Performance on one variable (i.e., the predictor) is used to
predict performance on a second variable (i.e., the outcome or
criterion)
Examples
–
Scholastic Aptitude Test (SAT) scores are used to predict college
freshmen grade point averages
– Scores from a mathematical attitude scale are used to predict
math achievement scores

Importance of the time interval between collecting the predictor
and criterion variable data
Copyright © Allyn & Bacon 2008
Correlational Designs

Prediction designs
–
Simple predictive studies

Factors influencing correlations
–
Possibility of low reliability of the instruments measuring
the predictor and criterion variables makes it difficult to
identify relationships
– Length of time between the predictor and criterion variable
data collection
– Existence of many factors, not only the one being
examined, that influence the criterion variable
Copyright © Allyn & Bacon 2008
Correlational Designs

Multiple regression
–
–
Studies that examine performance on several variables (i.e.,
predictor variables) to predict performance on a single
outcome variable (i.e., criterion)
Examples


Scholastic Aptitude Test (SAT) scores, high school grade point
average, and high school rank in class are used to predict
college freshmen grade point average
Math attitude scale scores, academic self-esteem scale
scores, and prior math grades are used to predict math
achievement scores
Copyright © Allyn & Bacon 2008
Correlational Designs

Logistic Regression
–
–
Another type of multiple regression analysis used
to examine the relationship between the predictor
variables and dependent variable.
The result is a prediction of whether the
participant is a “case” or “non-case”

Example: Overall GPA, gender, and special education
classification used to determine if a student will pass or
fail a standardized test.
Copyright © Allyn & Bacon 2008
Correlational Designs

Multiple regression
–
Issues of concern


Sample size of at least 10 subjects for each predictor
variable
Relationships among the predictor variables (i.e.,
colinearity)
Copyright © Allyn & Bacon 2008
Correlational Designs

Significance of correlation coefficients
–
Statistical significance


–
Does a statistical relationship exist?
Is the observed correlation significantly different from
zero?
Practical significance


Does a relationship of practical importance exist?
Coefficient of determination (r2) - the percentage of the
criterion variable variation that can be explained by the
variation in the predictor variable
Copyright © Allyn & Bacon 2008
Correlational Designs

Guidelines for interpreting the size of correlation
coefficients
–
Much larger correlations are needed for predictions with
individual than with groups


–
Crude group predictions can be made with correlations as low as
.40 to .60
Predictions for individuals require correlations above .75
Exploratory studies


Correlations of .25 to .40 indicate the need for further research
Much higher correlations are needed to confirm or test hypotheses
Copyright © Allyn & Bacon 2008
Correlational Designs

Guidelines for interpreting the size of correlation
coefficients
–
Multiple correlation coefficients of .20 - .40 are
common and usually indicate practical significance
Copyright © Allyn & Bacon 2008
Correlational Designs

Criteria for evaluating correlational studies
–
–
–
Causation should not be inferred from correlational
studies
The reported correlation should not be higher or
lower than the actual correlation
Practical significance should not be confused with
statistical significance
Copyright © Allyn & Bacon 2008
Correlational Designs

Criteria for evaluating correlational studies
–
–
–
The size of the correlation should be sufficient for the
use of the results
Prediction studies should report the accuracy of
predictions for new subjects
Procedures for collecting data should be clearly
indicated
Copyright © Allyn & Bacon 2008
Causal Comparative Designs

Ex-post-facto designs
–
Studies that investigate the relationships between
independent and dependent variables in situations where it
is impossible or unethical to manipulate the independent
variable

Example - what is the effect of pre-kindergarten (Pre-K)
attendance on first grade achievement
–
Cannot mandate Pre-K attendance for children
– Characteristics and resources of families who do and do not send
their children to Pre-K may influence first grade achievement
–
Similarities with correlational and experimental research
designs
Copyright © Allyn & Bacon 2008
Causal Comparative Designs

Ex-post-facto designs
–
Issues of concern


Selecting participants who are as similar as possible on all
characteristics except the independent variable
Generalizing beyond the participants studied
Copyright © Allyn & Bacon 2008
Causal Comparative Designs

Correlational causal-comparative studies
–
–
Use of correlational models to investigate possible
cause and effect relationships
Sophisticated statistical models


–
Path analysis
Structural equation modeling
Fundamental limitations of all correlational research
designs apply
Copyright © Allyn & Bacon 2008
Causal Comparative Designs

Criteria for evaluating causal-comparative studies
–
–
–
–
–
Primary purpose is to investigate causal relationships when
experimental designs are not possible
Presumed causal condition has already occurred
Potential extraneous variables are considered
Existing differences between groups being compared are
controlled
Causal conclusions are made with caution
Copyright © Allyn & Bacon 2008
Using Surveys

A data collection method that is very useful in
descriptive and correlational studies
–
–
–

Versatile
Efficient
Generalizable
Two types of survey designs
–
–
Cross sectional designs
Longitudinal designs
Copyright © Allyn & Bacon 2008
Using Surveys

Cross sectional designs
–
–
Information is collected from one or more groups at the same
time
Examples


–
Student’s, teacher’s, administrator’s, and parent’s opinions
regarding an extended school year
Elementary, middle, and secondary teachers’ feelings toward a
new school board policy
Issue of concern - comparisons across groups can be the result
of differences between participants within the groups

Fifth and seventh graders opinions can be affected by a changes
in the attendance zones of a school
Copyright © Allyn & Bacon 2008
Using Surveys

Longitudinal designs - information is collected
from the same participants over time
–
–
Example - changes in the academic self-concept of
students from the sixth to the twelfth grade
Issues of concern


Loss of subjects over time
Difficulty tracking participants over time
Copyright © Allyn & Bacon 2008
Using Surveys

Steps in designing a survey
–
–
Define a purpose and objectives
Identify the resources needed and the target population



–
Cost of preparation, printing, mailing, and analyzing results
Time needed to administer the survey
Sample size
Choose the method




Paper
Electronic
Telephone
Interview
Copyright © Allyn & Bacon 2008
Using Surveys

Steps in designing a survey
–
Develop the items – guidelines






–
Use clear, unbiased, non-ambiguous language
Keep it short and simple
Use grammatically correct language
Do not write leading items
Use the same response scale for all items
Be consistent with wording
Design the format of the survey


White space
Font size
Copyright © Allyn & Bacon 2008
Using Surveys

Steps in designing a survey
–
Develop directions




–
Make it clear with no ambiguity
Indicate clearly how participants are to respond
Indicate where responses are recorded
Indicate what participants should do when finished
Develop a letter of transmittal



Keep it brief
Include a statement of the purpose of the research
Include a statement of the benefits of the research
Copyright © Allyn & Bacon 2008
Using Surveys

Steps in designing a survey
–
Pilot test


15-20 representative participants
Identify concerns
–
–
–
–
–
Clarity
Format
Responding
Directions
Time to complete
Copyright © Allyn & Bacon 2008
Using Surveys

Response rates
–
–
Low response rates are the major limitation of survey use
Suggestions for increasing response rates






Design the survey well
Contact the subjects several times especially following-up on nonrespondents
Include a self-addressed return envelope
Use a good transmittal letter
Use a telephone for follow-up
Use incentives for completing the survey
Copyright © Allyn & Bacon 2008
Using Surveys

Using electronic surveys
–
Internet based surveys


–
–
E-mail attachments
Web pages
Evaluations of online university survey research
centers
Criteria for evaluating survey research centers
Copyright © Allyn & Bacon 2008
Using Surveys

Using electronic surveys
–
Advantages




–
Reduced time and cost
Easy access
Quick responses
Ease of creating data sets
Disadvantages


Limited to those with access to the technology
Confidentiality and privacy issue
Copyright © Allyn & Bacon 2008