CHAPTER ELEVEN
NON-EXPERIMENTAL RESEARCH of GROUP DIFFERENCES
Chapter Objectives:
•
Understand how non-experimental studies of group differences
compare to experimental studies of groups differences.
•
Understand the nature of the independent variable in nonexperimental studies of group differences
•
Understand methods of data collection in non-experimental studies
of group differences
•
Understand the procedures for conducting a non-experimental
studies of group differences
•
Understand inferential tests of significance used for continuous and
categorical dependent variables
•
Understand how non-experimental studies of group differences are
evaluated for validity
1 _________________________________________________________
Purpose and Comparison to Experiments
•
Non-Experimental Studies of Group Differences provide a new angle
from which to view group differences; they investigate existing
differences between and among groups that differ on a categorical
variable like gender or ethnicity, grade level, etc.
Experiments and non-experiments of group differences are similar in
the following ways: (1) they investigate theory-based differences between
groups in order to determine whether the differences exist and whether they
are statistically significant, (2) they select the sample for the study and seek
to make groups as equivalent as possible, (3) they operationalize the
dependent variable, (4) they select a valid and reliable measure for the
dependent variable and use the measure to collect data for both groups, (5)
they sort the data into two groups and use inferential tests to determine
statistical significance, and (6) they make inferences about results.
2 Non-experiments of group differences are different from experiments in
these ways: (1) Experiments demonstrate a cause and effect relationship
between groups variables by introducing an intervention or treatment and
controlling conditions of the study by assigning subjects to groups and
manipulating the treatment so that one group receives it and another does not.
Non-experiments of group differences do not demonstrate a cause and effect
relationship, nor do they introduce an intervention or treatment, or assign
subjects to groups. As a result, while they can explore the connections between
groups and may suggest an explanation for differences, they cannot attribute a
cause. This is due to the nature of the subjects, who are divided into groups
before the study begins and that are based on a shared, immutable
characteristic like their gender, ethnicity, grade level, etc. (2) Experiments
almost always use continuous variables for the dependent variable. Nonexperiments of group differences may use either continuous or categorical
variables as the dependent variable. In the case of a categorical dependent
variable, different tests of significance are used.
By way of Review:
•
A continuous variable can be measured and quantified. Continuous
variables may include scaled scores on assessments of
achievement or aptitude or responses on a Likert scale survey or
questionnaire -- all of which can be quantified and described
statistically as means and standard deviations.
3 •
A categorical variable is an attribute that cannot be measured or
quantified. Categorical variables may be on the order of innate
characteristics like gender or ethnicity or on the order of extrinsic
characteristics such as, highest level of education attained, grade
level taught, or political party affiliation.
Numerous labels describe this form of research, including ex-post
facto, causal-comparative, and pre-experimental studies. We prefer the
term non-experiments of group differences to indicate how they are similar
to and different from the way that experiments and quasi-experiments
investigate differences in groups.
Data Collection and Analysis
4 Surveys and large-scale tests are commonly used as measures for
collecting data in non-experimental group comparisons. A survey usually
has a section that asks questions about the attributes of subject
demographics like their gender, race, grade level, last degree earned, etc.
Having this information allows the researcher to divide responses into
different categorical groups for analysis. As an example, researcher may
want to investigate whether there are gender differences in job satisfaction
within a sample of teachers in a school district.
IV- gender
DV= job satisfaction
The researcher would collect data through a survey that asks for
demographic information like gender and that uses a Likert scale to assess
job satisfaction as a continuous variable. After administering the survey to
all teachers in the sample, the researcher (1) divides the sample into two
groups, male and female, (2) calculates means and standard deviations for
each group, and (3) selects an inferential test of significance to determine
whether differences do exist and whether they are significant. Because the
dependent variable is a continuous variable, the researcher would use a ttest, ANOVA/MANOVA, or ANCOVA/MANCOVA as the test of significance.
Now consider another example. Here the researcher also wants to
use the survey to investigate whether there are significant gender
differences in the highest level of education attained by teachers in the
same sample.
5 IV= gender
DV= highest level of education
In this instance, the dependent variable is a categorical variable. After
administering the survey to all teachers in the sample, the researcher would
divide the sample into two groups: male and female. Since the dependent
variable is categorical, it cannot be quantified in terms of means and
standard deviations and therefore cannot depend on t-tests,
ANOVA/MANOVA, or ANCOVA/MANCOVA as test of significance. Instead
it uses the Chi-Square Statistic that relies on frequencies and proportions/
percentages, to determine statistical significance. The process for
determining statistical significance is similar to that used in other tests of
significance. The researcher calculates a Chi-Square value and then
consults a table of critical values to establish a p-value.
Summary of Tests of Significance Used in Non-Experiments
Type of Dependent
Significance Test
Variable
t-test
Continuous
ANOVA/MANOVA
ANCOVA/MANCOVA
Categorical
Chi-Square
6 Examples of Non Experimental Studies of Group Differences
Continuous DV Study
Al-Yagon and Cinamom (2008) investigated the “work-family
relations” of two groups of mothers: those with and without children with
disabilities. The researchers used a survey that measured six continuous
dependent variables: attachment, affect, sense of coherence, work-tofamily facilitation, family-to-work conflict, and family cohesion.
IV = Mothers of disabled and typical children
DV = attachment, affect, sense of coherence, work to family
facilitation, family to work conflict, family cohesion
The first level of analysis was to calculate the means and standard
deviations for each group of mothers on each of the six continuous
dependent variables. The second step was to conduct the analysis of
variance (ANOVA) for each dependent variable. Altogether there were
fourteen comparisons, as represented in the table below, which looks like a
table you might find in an experimental or quasi-experimental study.
7 Two of the variables showed significant differences: work-to-family
facilitation (F=4.20, p< .05) and family-to-work conflict (F=4.70, p<.05). Both
are starred (*) to indicate p= < .05, which appears at the bottom of the
table. According to the authors “As a group, mothers of children with LD
reported a higher level of Family-to-Work conflict and a higher level of
Work-to-Family facilitation compared to mothers of non-LD children” (2009,
p. 99). The inference might be that having a child with learning disabilities
influences mothers’ engagement in facilitation and their experience of
conflict when it comes to work-family issues.
8 Categorical DV Study
Li (2006) investigated whether there were gender differences in
occurrences of bullying, which she divided into four categories: bully, bully
victim, cyberbully, and cyberbully victim
IV= gender
DV= bullying (bully, bully victim, cyberbully, cyberbully victim)
Li administered a survey to a sample of students that asked for
demographic data like gender and that also asked students to report on
their status in the bullying culture of the school, using a yes/no response
format. The answers to questions like those below allowed the researcher
to create the four categorical variables for comparison. Examples of
questions are:
1. I have been bullied during school: Yes No
2. I have bullied others: Yes No
3. I have been cyber-bullied (e.g. via email, chat room cell
phone): Yes No
4. I have cyber-bullied others: Yes No
The researcher began the analysis of data by reporting the percentages of
students that fell into each category by gender, as indicated below (p. 163).
9 By looking at these percentages, it appears there is a difference in
boys and girls. However, as we know from previous chapters, descriptive
data like percentages are not sufficient to make inferences about the
degree of difference or whether the difference is statistically significant.
Because the variables against which the two groups are being compared
are categorical, Li used Chi-Square as the statistical tool to determine
whether gender makes a significant difference in bullying. The Chi-Square
(X2) values and p-values are reported in the table below:
IV
Male vs.
DV’s
0.101
0.91
4.83
0.028*
4.82
0.021*
Being a cyberbully
female
0.17
Being a traditional bully
female
Male vs.
3.50
Cyberbully victim
female
Male vs.
p-value
Traditional Bully victim
female
Male vs.
X2
10 * p ≤ .05
The results showed significant differences between girls and boys in
terms of being a bully and being a cyber-bully, but showed no significant
differences between girls and boys in terms of being a bully or victim. The
inference would be that gender has a significant influence on whether one
becomes a bully and a cyber bully.
Evaluating the Validity of Non-Experimental Group Differences
Non-experiments cannot be evaluated for internal validity because they
do not show causal relationships; they are evaluated on the basis of
statistical conclusion validity and external population validity, as represented
in the table below.
11 Criterion
Statistical Conclusion
Validity
Theory
Evidence of “fishing?”
External (Population) Validity
• A sound theoretical
construct generated through an extensive
research review,
Clear identification of independent and
•
dependent variables
Sample
• Adequate sample
Random selection of the sample or sample
•
matching
size
Detailed description of who is in the sample
•
and the characteristics of subjects
Measure
Demonstrated construct validity of the
• Reliability of measure •
measures in relation to the theoretical
= > .7
construct
• Appropriate
Evidence of the reliability of the measure.
•
inferential statistics
• Appropriate alpha
level for significance
testing
Rating
H
M
L
H
12 M
L
Though based on weaker causal inferences, non-experimental group
difference studies can explore the complexity of differences between existing
intrinsic and membership groups on different outcomes.
13 Chapter Summary
•
Non-experimental studies of group differences are similar to group
experiments because they investigate the statistical significance of
group differences on a dependent variable.
•
Non-experimental studies of group differences are different from group
experiments because they are not causal, do not involve an intervention
or treatment, do not assign subjects to groups, and use both continuous
or categorical variables for the dependent variable
14 •
Data are commonly collected through surveys, questionnaires, and
achievement and aptitude measures that ask for demographic
information as well as for responses.
•
Data analysis for differences and statistical significance are different for
continuous and categorical variables; continuous variables are tested on
the same statistics as experiments; categorical variables are tested on
the Chi Square Statistic.
•
Studies are evaluated for statistical conclusion validity and external
validity.
Terms and Concepts
categorical variable
continuous variable
independent variable
dependent variable
Chi Square statistic
statistical conclusion validity
external validity
Review, Consolidation, and Extension of Knowledge
1. In your own words,
a. describe how non- experiments of group differences are similar to
and different from experiments
b. identify inferential tests of significance for studies that have a
continuous variable as the DV.
15 c. Identify the inferential test of significance for studies that have a
categorical variable as the DV.
2. Using an electronic database, search for non-experimental group difference
study (look for ex-post facto and casual comparative in your search. on a topic of
interest. Read the article and answer the questions in the Guide below.
3. Using the guide as a template write a critique of about 750 words of the
non-experimental group differences study you selected. See the Appendix
for an Exemplar.
.
Guide to Reading and Critiquing a Non- Experimental Group
Study
Research Review and Theory:
What is the purpose of the research review?
Does it establish an underlying theory (big ideas) for the research?
Purpose and Design:
What is the purpose of the study?
16 Is there a hypothesis or a research question? If so, what is it? If not,
can you infer the question from the text of the article?
What is the basic research design and type?
What are the groups being studied for differences?
Is the dependent variable continuous or categorical?
Sampling:
How is the sample selected: random or non-random?
Are the groups matched on characteristics other than their defining
characteristic?s If so, how ?
Who is in the sample? What are the characteristics of the sample?
What is the sample size?
Data Collection:
How are data collected: survey, questionnaire, test reports?
What information does the instrument collect?
For continuous dependent variable studies,
What is the response format of the questions?
Are there indications of validity and reliability of the measure?
What are they (r-values)?
Data Analysis and Results:
What statistical tests are used to analyze the data?
Were the results (p-values) significant or non-significant?
17 What does the researcher conclude about the findings?
Evaluation of Validity:
What is the quality of statistical conclusion validity? What is the rationale
for this judgment in terms of theory, sample, and measurement?
What is the quality of external validity? What is the rationale for this
judgment in terms of theory, sample, and measurement?
18 19