FOUNDATIONS OF QUALITY RESEARCH
DESIGN: RELIABILITY & VALIDITY
WINNIE MUCHERAH
BALL STATE UNIVERSITY
Literature review
•
Systematic identification, location, and analysis of
documents containing information related to the
research problem
•
Reviews are used to guide practice and/or to guide
research
•
Narrative reviews
•
Topic reviews
•
Theoretical reviews
•
Meta-analyses
(Mills, Airasian, & Gay, 2012)
Types of reviews
• Narrative/Traditional Reviews
• Most often conducted when writing
dissertations and theses in the social
sciences
• Also used in introductory paragraphs of
a typical research article
• Provides a brief narrative about
previous research on a subject to set
the context for the current research topic
Topic reviews
• Introductory and investigatory reviews
• Conducted when working on a topic for
the first time
• Often includes introductory works, e.g.,
encyclopedia entries and textbooks
• Criteria for good topic reviews:
• Recency (based on up-to-date sources)
• Importance (built on important sources,
quality of the journal, impact factor)
• Breath (sources discuss topic broadly)
Theoretical reviews
• Not usually featured in lists of types of
reviews, but are important subtypes
• It’s a version of a traditional/narrative
review
• It’s specific purpose is to synthesize
established theories by focusing on
points of agreement and/or to generate
new theories by focusing on gaps
• To either synthesize previous theories
or to generate new ones.
Meta-analyses reviews
• Systematic reviews/ Research synthesis
• Systematic- used frequently to refer to
evidence-based practical applications
• Research synthesis-often refers to
research that is not necessarily tied to
practical applications
• Similar: researcher states in advance
the procedures for findings, selecting,
coding and analyzing the data
• Data enables you to calculate effect size
Effect size
• Effect size is aptly named
• It’s a measure of the size of an effect.
• Specifically, it’s a standardized measure
• Standardized measures are often stated
in standard deviation units
• Therefore, they can be used to compare
and combine results across studies
• Comparing and combining results
across studies is the whole point of
meta-analysis.
quantitative v. qualitative
•
Quantitative research
•
Numerical data
•
•
•
Ex - surveys and tests
Research plan includes an introduction, method
section, data analysis description, and results
Qualitative research
•
Comprehensive, narrative, and visual data
•
Ex - interviews and naturalistic observations
•
Research plan must be responsive to the context
and setting under study
•
Mixed-method design is ideal
(Mills, Airasian, & Gay, 2012)
correlational v. Experimental
• Correlational research
• Collecting data to determine whether a
relation exists between two or more
quantifiable variables
• Measured by a correlation coefficient (r)
• Strength of relationship ranges from 0 to
1
• Relationship can be positive or negative
(inverse)
• Correlation is not causation
Experimental research
• Random assignment to groups
• Involves IV and DV
• At least one independent variable is
manipulated
• Effect of one or more dependent
variable(s) observed
• Quantitative measure of the degree of
correspondence between two or more
respondents
Reliability
•
It’s the consistency or agreement among measures
•
Consistency of data collection
•
Results are more likely to be repeatable if you
conduct the experiment all over again (because the
sample size is large enough to produce the
necessary precision)
•
Reliability coefficients generally range from 1.0 for a
perfectly reliable measure to 0 for one that is
completely inconsistent from one
rater/test/observation to the next
•
Cronbach’s alpha (α)-estimates internal consistency
(Rumsey, 2005)
•
Measure
of
reliability
Cronbach’s alpha (α)
• It’s used when you want to know
whether the items in your scale or index
are measuring aspects of the same
thing
• The “scale if item deleted” feature helps
identify items that could be removed or
analyzed individually (IRT)
• .70 is usually considered the minimum
acceptable level; higher levels are
needed when results are used for high-
Types of reliability
• Inter-rater reliability-refers to the
consistency of two or more raters
• Test-retest reliability-refers to the
consistency of the same test over time
or consistency of results on repeated
tests
• Internal reliability- refers to the
consistency of multiple questions
probing aspects of the same concept
Validity
•
It’s a central issue at all stages of a research project
•
Chief concern is whether the study is set up so that
you can reach justifiable conclusions about your
topic. This is referred to as Internal Validity
•
It addresses the question: Do my conclusions apply
to my sample?
•
The degree to which differences on a measure
are attributable to the manipulation of the
independent variable
•
This is highest in true experimental studies
(Mills, Airasian, & Gay, 2012)
External validity
•
The degree to which results will be generalizable and
to a certain extent replicable in other settings
•
It addresses the question: Do my conclusions apply
to anyone else?
•
Can you generalize your conclusions beyond the
participants in the experiment?
•
The answer depends on the quality and the
appropriateness of your sample
•
Construct validity: are concepts measured in ways
that enable us to study what we aim to study?
•
Content validity: is the measure thorough or
representative of the thing being measured?
Sampling procedures
•
Population
• collection of all individuals of interests
• Sample
• subset of the population we measure
• Parameter
•
a numeric characterization of the population that
is of interest to us
• Statistic
• a numeric characterization of the sample that is
an estimate of the population
• Since we cannot access population, we don’t have
access to parameter, so we take a sample we can
obtain, then we make a numeric measurement, also
known as a statistic
Coladarci & Cobb, 2014
Contextualizing your
research
•
Refining the substantive question and developing a
plan for collecting relevant data
•
Use of existing/new measures: Use Factor Analysis
•
FA helps you decide about reliability and validity of
your measurements of latent variables and thus how
to analyze and interpret them
•
FA is simply correlations and associations among
items
•
Purpose of FA is to improve the measurement of
latent variables or constructs that cannot be directly
observed
(Coladarci & Cobb, 2014)
Latent variables
• Latent variables can only be studied
indirectly by using indicators of
observed variables, e.g., in a multi-item
measure of traits, the items would be
indicators (or observed variables) and
clusters of questions identified by the FA
would help you identify the factors or
latent variables, which are the
constructs or concepts you seek in your
research. E.g., 15 questions toward a
controversial issue
• Efficacy or social tolerance or attitudes
Types of Factor analysis
• Exploratory FA and Confirmatory FA
• EFA-used when researchers are
looking for interesting patterns among
variables
• CFA-used when researchers have
theories about the patterns they want to
test
• The two are often linked because it is
very common to conduct them in
sequence-first EFA to refine theories,
then CFA to test them.
Conclusion
• Substantive Question ---> Statistical
Question ---> Statistical Conclusion --> Substantive Conclusion
• Substantive Conclusion is a contextbased conclusion
references
•
Coladarci, T., Cobb, C.D., Minium, E.W. & Clarke, R.C. Fundamentals of Statistical
Reasoning in Education.
•
Mills, G.E., Airasian, P. & Gay, L.R. 2012. Educational Research: Competencies for analysis
and applications. 10th Edition.
•
Rumsey, D. 2005. Statistics Workbook for Dummies.