Educational Research Methods
Dr. K. A. Korb
University of Jos
Introduction
•
•
There is no point to analyzing data from a study
that was not properly designed to answer the
research question under investigation. In fact,
there’s a real point in refusing to analyze such data
less faulty results be responsible for implementing a
program or policy contrary to what’s really needed.
Two of the valuable things a researcher can possess
are:
1. Knowledge of the principle of good study design
2. The courage to refuse to cut corners
(Dallal, 1998)
Dr. K. A. Korb
University of Jos
Scientific Method
1: Ask
Question
6: Share
Findings
2: Design
Study
5: Reach
Conclusions
3: Collect
Data
4: Analyze
Results
Dr. K. A. Korb
University of Jos
1. Ask a Question
• What is your general research question?
• What is the independent variable in the
research question?
• What is the dependent variable in the
research question?
Dr. K. A. Korb
University of Jos
2. Design the Study
• Research Design: Structure the research to show
how the major parts of the research project work
together to address the central research questions
– Participants
• What is the population of people I am interested in?
– Instruments
• How will I accurately measure my independent and dependent
variables?
– Procedure
• How will I collect data in an appropriate manner?
Dr. K. A. Korb
University of Jos
Outline
• Types of Research Methodologies
– Descriptive
– Causal Comparative
– Correlation
– Experimental
– Quasi-Experiment
• Threats to valid Research Design
Dr. K. A. Korb
University of Jos
Descriptive
• Purpose: Carefully describe a naturally
occurring educational phenomenon through
systematic observation
• Key characteristic: Describes one (sometimes
more) variable within a particular population
• Descriptive research designs are sometimes
also called Survey Research Designs.
Dr. K. A. Korb
University of Jos
Descriptive
• Research hypotheses are impossible for descriptive
designs. Only research questions should be posed.
• There are no independent or dependent variables, simply the key
psychological variables that the researchers is interested in.
– Example questions:
• What are teachers’ attitude towards a new moral educational
program?
• What would counselors recommend be included in an HIV/AIDS
curriculum?
• How many students engage in exam malpractice?
Dr. K. A. Korb
University of Jos
Descriptive
1. Select participants: Define the group that
possess the variables you want to study
2. Data collection: Administer valid measures
of the variables of interest
3. Data analysis: Compute descriptive statistics
Dr. K. A. Korb
University of Jos
Descriptive
Dr. K. A. Korb
University of Jos
Causal Comparative
• Purpose: Determine a cause/effect
relationship where the independent variable
cannot be manipulated
• Key characteristic: Compare two or more
naturally-occurring groups on the dependent
variable
• Causal Comparative designs can also be called
ex post facto designs
Dr. K. A. Korb
University of Jos
Causal Comparative
• Null hypothesis: There is no significance difference
between GROUPS on DEPENDENT VARIABLE.
• Independent variable: Group category
• Dependent variable: What the groups are hypothesized to differ
on.
– Examples:
• There is no significant differences between boys and girls on
interest in mathematics.
• There is no significant differences between children from low and
high socioeconomic status in the number of books read in a
month.
• There is no significance differences between Nigeria and British
children in academic self efficacy.
Dr. K. A. Korb
University of Jos
Causal Comparative
1. Select participants: Obtain a critical number of
participants in each group
2. Data collection: Administered valid measures of
your variables to all groups of participants
3. Data analysis: Calculate the mean score for each
group on the dependent variable.
–
–
However, any difference in the mean score between
groups might be due to error.
Therefore, inferential statistics are necessary to
determine if the difference are significant.
Dr. K. A. Korb
University of Jos
Inferential Statistics
– The inferential statistics to be depends on how many
groups are being compared
• Two groups: t-test
• Three or more groups: ANOVA
– If the value is greater than.05, retain the null hypothesis.
• There is no significant difference between boys and girls on
interest in math.
– If the p-value is less than .05, reject the null hypothesis.
• There is a significant difference between boys and girls on interest
in math.
– Look at the mean score to determine which group has the stronger
interest.
Dr. K. A. Korb
University of Jos
Causal Comparative
Dr. K. A. Korb
University of Jos
Correlational
• Purpose: Quantity the extent to which two variables
are associated
• Key characteristic: The same group of participants
are given measures of both key variables in order to
calculate the correlation coefficient
• Because of the directionality and third variable
problems, there are no independent or dependent
variables in correlational designs
Dr. K. A. Korb
University of Jos
Correlational
• Null hypothesis: There is no significant relationship
between VARIABLE 1 and VARIABLE 2.
• Examples
– There is no significant relationship between number of
books a child reads at home and reading ability.
– There is no significant relationship between WAEC scores
and university GPA.
– There is no significant relationship between the amount of
time spent in lecture and frequency of exam malpractice.
Dr. K. A. Korb
University of Jos
Correlational
1. Select participants: A homogeneous sample
2. Data collection: Administer valid measures
of the variables
3. Data Analysis: Calculate the correlation
coefficient and the significance test for
correlation
Dr. K. A. Korb
University of Jos
Dr. K. A. Korb
University of Jos
Interpreting Correlations
Nature:
Strength:
-1
Negative
Positive
0
+1
• Nature
– Positive: Two variables increase or decrease together
– Negative: As one variable increases, the other decreases
• Strength
– Closer to -1 or +1 is stronger relationship
– 0 is no relationship
Dr. K. A. Korb
University of Jos
Correlation = .78
100
90
.
70
Exam Score
80
60
50
40
30
20
10
0
2
4
6
8
10
Hours Studied
Dr. K. A. Korb
University of Jos
Correlation = -.86
4.00
3.50
.
2.50
GPA
3.00
2.00
1.50
1.00
0.50
0.00
0
2
4
6
8
10
Hours per day watching TV
Dr. K. A. Korb
University of Jos
Glasses of Milk Consumed per Day
.
Correlation: .04
4.50
4.00
3.50
3.00
2.50
2.00
1.50
1.00
0.50
0.00
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
Number of Stamps on Passport
Dr. K. A. Korb
University of Jos
Research Example
• The effect of regular leisure reading on reading
achievement in primary school.
• Students in a primary school will be randomly
assigned to either a treatment or a control group.
– The treatment group will spend 10 minutes everyday reading
a book with an adult.
– In an attempt to keep the treatment and control groups as
similar as possible, the control group will also spend 10
minutes with an adult everyday, but will instead do math
problems.
Dr. K. A. Korb
University of Jos
Correlational
Correlation does not prove causation.
Interest
Amount
of time
spent
studying
Academic
Achievement
Interest
Interest
Academic
Achievement
Academic
Achievement
Directionality Problem
Third Variable Problem
Dr. K. A. Korb
University of Jos
Experimental
• Purpose: Establish cause and effect
• Key characteristic: Statement about how one
variables affects (causes) another
• Three essential components of an experiment
– Treatment
– Control
– Random assignment to treatment and control groups
Dr. K. A. Korb
University of Jos
Experimental
• Null hypothesis: There is no significant effect of
TREATMENT on DEPENDENT VARIABLE.
– Independent variable: Treatment
– Dependent variable: What the treatment should change
• Examples
– There is no significant effect of a counseling intervention on
homework completion.
• Treatment: Counseling
– There is no significance effect of reading a book a day on
reading achievement.
• Treatment: Reading a book a day
– There is no significant effective of an anti-exam malpractice
campaign on the frequency of exam malpractice.
• Treatment: Anti-exam malpractice campaign
Dr. K. A. Korb
University of Jos
Experimental
• Goal: Keep the
experiences of the
treatment and control
groups as identical as
possible except for the
treatment in order to
conclude that the
change in the
dependent variable is
the result of the
treatment.
Experimental Group
Read a
book
every
day
Reading
Achievement
Reading
Class
Control Group
Reading
Class
Reading
Achievement
Dr. K. A. Korb
University of Jos
Experimental
• Random Assignment:
Each subject has an
equal chance of being
assigned to either the
treatment or control
group
– Minimize individual
differences in extra
variables that might
influence the dependent
variable
Interest
Experimental Group
Read a
book
every
day
Reading
Achievement
Reading
Class
Control Group
Reading
Class
Reading
Achievement
Dr. K. A. Korb
University of Jos
Random Assignment
• Draw names out of a
hat
• Random number table
S/No
Random
Number
Group
1
1
Treatment
2
1
Treatment
3
0
Control
4
1
Treatment
5
1
Treatment
6
0
Control
7
1
Treatment
8
0
Control
9
0
Control
10
0
Control
Dr. K. A. Korb
University of Jos
Experiment
•
Pretest, Post-test Procedure
1. Random assignment of participants to experimental and
control groups
2. Administered valid measure of DV as pretest to all
groups
3. Administer treatment to experimental groups(s)
4. Administer identical (or conceptually identical) measure
as posttest
5. Statistical Analysis
–
–
Descriptive statistics of pretest and post-test score for
experiment and control groups separately
Significance tests to determine significance between differences
(Analysis of Covariance: ANCOVA)
Dr. K. A. Korb
University of Jos
Experimental
Dr. K. A. Korb
University of Jos
Experimental
•
Post-test Only Procedure
1.
2.
3.
4.
Random assignment of participants to experimental and
control groups
Administered treatment to experimental group(s)
Administer valid measure of DV post-test
Statistical Analysis
–
–
–
Descriptive statistics of post-test score for experimental and control
groups separately
Significance tests to determine significance between differences
(Analysis of Variance – ANOVA or t-test)
The post-test only procedure is typically recommended
because of the possibility that the pretest may have an
effect on the experiment treatment
Dr. K. A. Korb
University of Jos
Experimental
Dr. K. A. Korb
University of Jos
Quasi-Experiment
• Purpose: Conduct an experiment when
random assignment is not possible
• Key characteristic: Treatment and control
groups, but no random assignment
• The null hypotheses are exactly the same as
the Experimental method
Dr. K. A. Korb
University of Jos
Quasi-Experiment
•
Pretest-Posttest Procedure
1. Administer valid measure of DV as pretest
2. Administer treatment to experimental group(s)
3. Administer identical (or conceptually identical) measure
as posttest
4. Statistical Analysis
•
•
Descriptive statistical of pretest and posttest score for
experiment and control groups separately
Significance tests to determine significance between difference
(ANCOVA)
Dr. K. A. Korb
University of Jos
Review: Research Designs
•
•
•
•
•
Descriptive: Carefully describe a naturally occurring
educational phenomenon through systematic
observation
Causal Comparative: Compare two or more groups on a
dependent variable
Correlational: Quantify the extent to which two
variables are related
Experiment: Establish cause and effect
–
Three requirement:
1. Treatment Groups(s)
2. Control Group
3. Random Assignment
Quasi-Experiment: Experiment where random
assignment is not possible
Dr. K. A. Korb
University of Jos
Research Designs Conclusion
• Testing the effectiveness of a counseling or
teaching intervention MUST either use an
experimental or quasi-experimental design
• Simply administering surveys CANNOT
establish the quality of a counseling or
teaching intervention
Dr. K. A. Korb
University of Jos
Research Example
• The effect of regular leisure reading on
reading achievement in primary school
• Students in a primary school will be randomly
assigned to either a treatment or a control
group.
– The treatment group will spend 10 minutes
everyday reading a book with an adult.
– In an attempt to keep the treatment and control
groups as similar as possible, the control group
will also spend 10 minutes with an adult everyday,
Dr. K. A. Korb
University of Jos
but will instead do math problems.
Research Example
IV:
Regular
Leisure
Reading
DV:
Reading
Achievement
Experimental Group
Reading
Achievement
Read Book
Every Day
with Adult
Reading
Achievement
Control Group
Reading
Achievement
Do Maths
Every Day
with Adult
Reading
Achievement
Dr. K. A. Korb
University of Jos
Construct Validity
• Construct Validity: Establishing valid operational
measures for the concepts being studied
– This relates to:
• Independent Variable
• Dependent Variable
• Definitions of Variable:
– Construct Definition: General explanation of the
construct
– Operational Definition: Statement of specifically how
the construct will be measured or implemented in the
study
Dr. K. A. Korb
University of Jos
Construct vs. Operational Definitions
Construct
Operational Definition
WAEC Scores
Academic
Achievement
Final Course Exam
NECO Scores
State Exams
Economic
Status
Self-Report of Parents’
Yearly Income
Public/Private School
Dr. K. A. Korb
University of Jos
Research Example
Construct validity focuses on:
IV:
Regular
Reading
DV:
Reading
Achievement
Experimental Group
Reading
Achievement
Read a Book
Every Day
Reading
Achievement
Control Group
Reading
Achievement
Reading
Achievement
Dr. K. A. Korb
University of Jos
Construct Validity
• Research studies begin with hypotheses about
psychological construct.
– There is no significant effect of regular leisure reading on
reading achievement scores.
• Independent variable: Regular leisure reading
• Dependent variable: Reading achievement score
– The psychological constructs are then operationalized into
manipulatable, measurable terms.
• Regular leisure reading: Reading a book with an adult everyday for
10 minutes
• Reading achievement score: Classroom exam scores
Dr. K. A. Korb
University of Jos
Construct Validity
• When finished analyzing the data, the result will be
translated back into psychological constructs.
– The conclusion should not be limited to Reading a book for
10 minutes everyday with an adult leads to higher score on a
classroom reading exams.
– The conclusion should be Regular leisure reading will lead to
higher reading achievement scores in general.
• However, in order to translate the results back to
psychological construct, the study must have construct
valid measures of the independent and dependent
variables.
Dr. K. A. Korb
University of Jos
Construct Validity
• When you develop your research project, you start with a theory.
– For example, regular leisure reading will lead to increased reading
achievement scores.
• Then you operationalize your theory into your research study.
– You translate “regular leisure reading” into how it will be implemented in your
study – reading a book with an adult everyday for 10 minutes.
– You translate “reading achievement scores” into how it will be measured in
your study – classroom exam scores.
• When you finish collecting data, you hope to be able to move back to
theory.
– You want to be able to say that not only did reading a book for 10 minutes
everyday with an adult lead to higher classroom exam scores on reading
achievement, but also that regular leisure reading will lead to higher reading
achievement scores in general.
– However, in order to do that, you must have construct valid measures of your
independent and dependent variables.
Dr. K. A. Korb
University of Jos
Research Example
Constructs:
Operationalized:
Constructs:
IV:
Regular
Reading
DV:
Reading
Achievement
Read book
every day with
an adult for
10 min.
Scores on
reading section
of classroom
exams
IV:
Regular
Reading
DV:
Reading
Achievement
Dr. K. A. Korb
University of Jos
Construct Validity
•
•
Construct Validity of Research Designs: Allows
generalization from the operationalized treatment
and measures to the general psychological
constructs
Threats to Construct Validity
–
–
Poor construct definitions in the paper
Flawed matching of operationalization to constructs
•
The measures do not align with the constructs they were
designed to measure.
Dr. K. A. Korb
University of Jos
Construct Validity
• Research studies begin with hypotheses about
psychological construct.
– There is no significant effect of regular leisure reading on
reading achievement scores.
• Independent variable: Regular leisure reading
• Dependent variable: Reading achievement score
– The psychological constructs are then operationalized into
manipulatable, measurable terms.
• Regular leisure reading: Reading a book with an adult everyday for
10 minutes
• Reading achievement score: Classroom exam scores
Dr. K. A. Korb
University of Jos
Construct Validity
• When finished analyzing the data, the result will be
translated back into psychological constructs.
– The conclusion should not be limited to Reading a book for
10 minutes everyday with an adult leads to higher score on a
classroom reading exams.
– The conclusion should be Regular leisure reading will lead to
higher reading achievement scores in general.
• However, in order to translate the results back to
psychological construct, the study must have construct
valid measures of the independent and dependent
variables.
Dr. K. A. Korb
University of Jos
Construct Validity
• When you develop your research project, you start with a theory.
– For example, regular leisure reading will lead to increased reading
achievement scores.
• Then you operationalize your theory into your research study.
– You translate “regular leisure reading” into how it will be implemented in your
study – reading a book with an adult everyday for 10 minutes.
– You translate “reading achievement scores” into how it will be measured in
your study – classroom exam scores.
• When you finish collecting data, you hope to be able to move back to
theory.
– You want to be able to say that not only did reading a book for 10 minutes
everyday with an adult lead to higher classroom exam scores on reading
achievement, but also that regular leisure reading will lead to higher reading
achievement scores in general.
– However, in order to do that, you must have construct valid measures of your
independent and dependent variables.
Dr. K. A. Korb
University of Jos
Internal Validity
• Internal validity: Extent to which variables other than the
treatment provide plausible explanations to the experimental
results
– Only relevant for experimental designs
• Primary consideration for interventions
– Extraneous variable: Any variable other than treatment variable that,
if not controlled, can affect the experimental outcome
– Hold constant or eliminate all extraneous variables that might affect
the posttest
– Goal of Research Design: Create set of conditions so any observed
changes in your dependent variable can be attributed to experimental
treatment instead of extraneous variables
• Do variables other than the treatment provide plausible
explanations to the experimental results?
Dr. K. A. Korb
University of Jos
Research Example
IV:
Regular
Reading
DV:
Reading
Interest
Achievement
Maturation
Experimental Group
Reading
Achievement
Read a Book
Every Day
Reading
Achievement
Control Group
Reading
Achievement
Dr. K. A. Korb
University of Jos
Reading Reading
Adult
Achievement
Ability
Attention
Threats to Internal Validity
• History: If treatment lasts over time, other events
may influence dependent variable
– To prevent: Keep the control group equivalent in all
aspects but treatment
• Maturation: Physical or psychological changes may
influence the dependent variable
– To prevent: Use a control group
• Selection Bias: Treatment and control groups are
different on an important extraneous variable
– To prevent: Random assignment OR Use a pre-test
Dr. K. A. Korb
University of Jos
Threats to Internal Validity
• Testing: Giving a pretest may increase performance
on the post-test
– To prevent: Don’t use a pre-test OR Use different pre- and
post-tests
• Instrumentation: Nature of measuring instrument
has changed between pre- and post-test
– To prevent: Do not use a pre-test OR Use the same for preand post-test
• Many of these solutions to prevent threats to
internal validity contradict.
– Consider the threats that are most relevant to a particular
study and then develop a strategy for overcoming the most
important threats.
Dr. K. A. Korb
University of Jos
External Validity
• External Validity: Establishing the group of
people to which the research findings can be
generalized
Dr. K. A. Korb
University of Jos
Research Example
Population
Experimental
Group
Reading
Achievement
Read a Book
Every Day
Reading
Achievement
Sample
Reading
Achievement
Control
Group
Reading
Achievement
Dr. K. A. Korb
University of Jos
Types of External Validity
• Population Validity: Generalizing the results from an
experimental sample to a defined population
– To establish: Give a thorough explanation of the constitution of the
constitution of the sample and how the sample was selected.
• Ecological Validity: Generalizing the results of a study from
the artificial conditions created by researcher to real-life
conditions
– To establish:
• Give an adequate description of treatment
• Give an adequate description of the measure for the dependent
variable
– Threats to Ecological Validity:
• Novelty: The treatment was different than typical treatment
• Experimenter Effect: The particular person delivering the treatment
affects results
Dr. K. A. Korb
University of Jos
Reliability
• Reliability: Demonstrating that the operations
of the study can be repeated with the same
results
– Conduct research as if someone were always
looking over your shoulder
– Report the procedure of your study in explicit
detail
Dr. K. A. Korb
University of Jos
Other Issues in Valid Research Design
• Experimenter Bias: Researcher’s expectations about
the outcome of experiment influences participants’
response
– To prevent: Use research assistants who are blind to the
study (aka do not know the purpose of the study)
• Treatment Fidelity: Extent to which treatment
conditions are implemented according to the
researcher’s specifications
– To support:
• Carefully train research assistants
• Periodically check up on research assistants
unexpectedly to determine if they are following
guidelines
Dr. K. A. Korb
University of Jos
Other Issues in Valid Research Design
• Strong Experimental Treatments: Developing a
treatment that will make a robust effect on the
dependent variable
– To support:
• Develop a thorough understand of the dependent
variable and how it can be influenced.
• Spend much thoughtful time developing your
treatment.
Dr. K. A. Korb
University of Jos
Revision
• Describe the five types of educational research
designs.
• What is construct validity as related to research
designs? How can it be supported?
• What is internal validity? What are threats to
internal validity?
• What is external validity? What are types of
external validity?
• What are some other issues to consider when
conducting educational research?
Dr. K. A. Korb
University of Jos