Reliability
Validity
Levels of Measurement
Scales
How we figure out what to measure
• Conceptualization
– Process of taking a construct and refining it by
giving it a conceptual or theoretical definition
– Research focusing on college students
• In Ohio? What region? Age? Major?
• Operationalization
– Links a conceptual definition to a specific set
of measurement techniques
Coming up with a measure
•
•
•
•
•
Remember the conceptual definition
Keep an open mind
Borrow from others
Anticipate difficulties
Don’t forget units of analysis
Empirical Hypothesis
• The degree of
association
• How well
operationalized
variables are
associated (or not)
with the concept
construct determines
the hypothesis
Reliability
• Reliability means dependability or consistency
• Same thing occurs over and over under same
conditions
• A scale, for example
• How dependable is the study?
• Is the study consistent, or does it yield wide
varying results?
• Can the study be replicated?
Reliability
• Measurement directly affects the quality of
conclusions.
• Care is needed to make sure that results
are not corrupted by improper
measurement.
• The operational definition of a concept
should have a precise meaning:
– The terms by which you measure a concept
should be explicit.
Reliability
• Reliability and validity are the biggest
threats to proper measurement.
• Reliability is the extent to which an
experiment, test, or any measuring
procedure yields the same results on
repeated trials.
• Do you get the same result every time?
Reliability
• Three tests of reliability:
– Test-retest method
• Applying the same test to the same observations after
a period of time and then comparing the results of the
different measurements
– Alternative form method
• Two different measures of the same concept
administered to the same respondents at different
times before the scores are compared
– Split-halves method
• Divide a multi-item measure into two measures with
both of the new measures applied at the same time
Improving Reliability
• Clearly conceptualize constructs
• Increase level of measurement
• Use multiple indicators of a variable
– Triangulation
• Use pretests, pilot studies, and replication
Validity
• A valid measure is one that measures
what it is supposed to measure, in other
words, the degree of correspondence
between the measure and the concept it is
thought to measure.
• Four tests of validity
Validity
• Truthfulness
• Refers to the match between a
construct and a measure
• Want it to be valid for a
particular purpose and
definition
• How good is the measure?
• Is the data measured
correctly?
• Is the data analyzed correctly
(statistical)?
Internal Validity
• Are there errors as a result of the internal
design of the study?
• Are there errors as a result of the
controls?
• Internal validity problems can occur from a
flawed survey along with a multitude of
other factors
External Validity
• Can your experiment’s findings be
generalized?
• External Validity questions are evident in
every study; however, methods exist to
keep external validity high and the number
of external flaws low
Types of Validity
• Face validity
– Judgment that the indicator really measures the construct
• Content validity
– Does your measure represent the full content of a defintion?
• Criterion validity
– Use some standard or criteria to indicate a construct accurately
• Concurrent validity
– Indicator must be associated with a preexisting indicator judged
to be valid
• Predictive validity
– Indicator predicts future events that are logically related to a
construct
Types of Validity
•
•
•
•
•
•
•
•
•
1) If I create a new test of mathematical ability for high school students and test it by having high
school math teachers look at it and tell me if it seems appropriate, I am measuring for
____________________ validity.
2) If I am examining an individuals ability to cope with stress and have three attributes I am
particularly interested in and I am checking to see if my construct hits on all three attributes, I am
measuring for _______________________ validity.
3) If I create a new test for cognitive recognition and students that score high on it also score high
on previously existing tests for cognitive recognition, I have demonstrated
____________________ validity.
4) If I compare my measure for testing the potential to suffer from childhood diabetes with a
previously used test, I am looking for _________________________validity.
5) If I create a new test of intelligence and students that score high on it also do better in college
than those who score lowly, I have shown ___________________ validity.
Validity
• Tests of validity are not as good as tests of
reliability.
• Reliability is easy to demonstrate through
some form of repeated trials.
• Validity is more difficult because we can
never be sure about the true value of a
concept:
– Especially true with abstract concepts
Validity
• Whereas a valid measure is reliable
(because if truly valid, it will measure the
concept correctly every time), a reliable
measure is not necessarily valid.
• The measure could be measuring the
concept incorrectly in a consistent way.
Relationship between reliability and
validity
Levels of Measurement
• The level of measurement of a variable
describes
– The amount of precision associated with a
variable
– The mathematical properties of the variable
• Both precision and mathematical
properties increase as you increase the
level of measurement from nominal to
ratio.
Levels of Measurement
• Continuous v. discrete variables
– Continuous
• Have an infinite number of values or attributes that
flow directly along a continuum
• Temperature, age, income, crime rate
– Discrete
• Relatively fixed set of separate values or attributes
• Gender, religion, marital status
Nominal Level
• Only reports a
difference
• Candidate
preference, religious
preference, Yes/No,
etc.
• Discrete Variables
Levels of Measurement
• The level of measurement of a variable
describes
– The amount of precision associated with a
variable
– The mathematical properties of the variable
• Both precision and mathematical
properties increase as you increase the
level of measurement from nominal to
ratio.
Ordinal Level
• Rank ordered
• Grades, opinion
•
•
•
•
Strongly Agree
Agree
Disagree
Strongly
Disagree
Levels of Measurement
• At the ordinal level, categories may be
ranked in order in addition to indicating a
difference between categories.
• Example: Please indicate the highest level of
education you reached (elem., high, college,
more).
• Precision: A little more precision and can be
used with more statistical tools
Interval/Ratio Level
• A specified distance
• Interval does not contain
a true zero point (ratio
does)
• Interval: IQ, SAT
• Ratio: years of school,
income
Levels of Measurement
• The interval level includes all of the
information of the preceding levels and
adds meaningful intervals between values
of the variable but does not use a
meaningful zero.
• Example: What did you score on the SAT?
• Precision: More precision and can be used
with most statistical tools
Levels of Measurement
• The ratio level adds a meaningful zero to
the interval level.
• Example: How many years of education?
• Precision: Most precision and can be used
with most statistical tools
Scales
• Some concepts can be captured with a
single question.
• More complex concepts may require a
multi-item measure consisting of several
questions that capture different
components of the concept and increase
validity.
Scales
• Summation index:
– Combines the scores on multiple questions to create
one single measure of a concept
• Likert scale:
– Uses only select questions from an index that
differentiate between different respondents to create
a single score for each respondent
Scales
• Guttman scale:
– Has answer choices arranged in an ordinal manner;
respondents will agree with each of the lower-ranked
answers if they agree with a higher-ranked answer
• Factor analysis:
– Allows researchers to uncover patterns across related
measures to create summary variables that represent
different dimensions of the same concept
Mutually Exclusive
• “One and only.”
• One may only fit the criteria of one
category
• Ex: Religion:
Christian, non-Christian, Jewish, Buddhist
NOT MUTUALLY EXCLUSIVE
Exhaustive
• All cases fit into one category
• Ex: If the Election were today, would you
support Sherrod Brown, the Democrat, or
Mike DeWine, the Republican or neither?
• NOT EXHAUSTIVE
Missing Data
• No survey is perfect, and certain questions will
be left unanswered or completely skipped
• Remedy?
• A “catch all” category and/or a way to factor out
missing data
• Yet, missing data can still mislead a study
One Last Thing
• Feeling Thermometers
• Likert Scales
• Response set problem
– How do we fix this???