The Theory of Measurement
0378 – Methods I
Quantitative Research Methods
Florian Schausberger
Measurement
§ Measurement is the process of observing and
recording the observations that are collected
as part of a research effort.
§ 3 sections
Levels of
Measurement
Construct Validity
Reliability
SEITE 2
The meaning of the four major levels of
measurement
What constitutes a good measure?
Consistency and Dependability of
Measurement
Levels of Measurement
„The level of measurement refers to the
relationship among the values that are assigned
to the attributes for a variable.“
SEITE 3
Levels of Measurement (2)
Nominal
• Variables that
can be named;
one is not
necessarily
greater or less
than the next
SEITE 4
Ordinal
• These indicate
order or
ranking,
but they do not
tell us the
difference
between
rankings; the
distance is not
meaningful
Interval
• Interval also
has order, but
the distance
between the
scores is
equivalent and
meaningful;
Ratio
• Has order,
equal intervals
and a true zero.
Levels of Measurement (3)
Nominal
Ordinal
Interval
Ratio
Mode, counts, frequency
Yes
Yes
Yes
Yes
Median, Minimum, Maximum,
Range
No
Yes
Yes
Yes
Mean, Variance, Standard
deviation
No
No
Yes
Yes
s
SEITE 5
The hierarchy
of measurement levels
SEITE 6
Construct Validity
§ Construct Validity refers to the degree to
which inferences can legitimately be made from
the operationalizations in your study to the
theoretical constructs on which these
operationlizations are based.
§ That is, does a test measure the construct as it
claims to measure.
§ e.g. does an intelligence test measure intelligence? We
can‘t observe intelligence as such, yet we have an idea
of what intelligence means.
SEITE 7
Measurement Validity Types
Construct Validity
Translation Validity
Criterion-related
Validity
Face Validity
Predictive Validity
Content Validity
Concurrent Validity
Convergent Validity
Discriminant Validity
SEITE 8
Translation Validity
Translation
Validity
A type of validity construct related to how
well you translated the idea of your
measure into its operationalization. That
is, it the operationalization a good
reflection of the construct?
Face Validity
A type of validity that „on its face“ the
operationalization seems like a good
translation of the construct (prima facie)
Content Validity
A check of the operationalization against
the relevant content domain for the
construct.
SEITE 9
Criterion-related Validity
Criterion-related
Validity
The validation of a measure based on its
relationship to another independent
measure as predicted by your theory of
how the measure should behave.
Predictive
Validity
A type of construct validity based on the
idea that your measure is able to predict
what it theoretically should be able to
predict.
Concurrent
Validity
Any operationalization‘s ability to
distinguish between groups that it should
theoretically be able to distinguish
between.
SEITE 10
Criterion-related Validity (2)
SEITE 11
Convergent
Validity
The degree to which the operationalization
is similar to (or converges on) other
operationalizations to which it should be
theoretically similar.
Discriminant
Validity
The degree to which the operationalization
is not similar to (diverges from) other
operationalizations that it theoretically
should not be similar to.
Definitionalist vs. Relationalist
Views
A Definitionalist
Perspective
A Relationalist
Perspective
Either you have operationalized the
construct correctly or you haven‘t.
Recognizes that the meaning of constructs
differs relatively, not absolutely.
A Relationalist View:
SEITE 12
Convergent Validity
SEITE 13
Discriminant Validity
SEITE 14
Putting it all together …
SEITE 15
The Nomological Network
SEITE 16
The MTMM-Matrix
(Multitrait-Multimethod)
SEITE 17
A Modified MTMM
§ … leaving out the methods factor
SEITE 18
The Theory of
Pattern Matching
Pattern
Matching
SEITE 19
Pattern Matching means the degree of
correspondence between two patterns. Do things,
as they operate in reality, correspond to the
theoretical pattern, how the world should work?
Threats to Construct Validity
§ Design Threats
§ Inadequate Preoperational Explication of Constructs
§ Mono-Operation Bias
§ Mono-Method Bias
§ Interaction of Different Treatments
§ Interaction of Testing and Treatments
§ Restricted Generalizability across Constructs
§ Confounding Constructs and Levels of Constructs
SEITE 20
Threats to Construct Validity
(2)
§ The Social Threats to Construct Validity
§ Hypothesis Guessing
§ Evaluation Apprehension
§ Experimenter Expectancies
SEITE 21
Reliability
§ Reliability refers to the degree to which a
measure is consistent and dependable; the
degree to which it would give you the same
result over and over again, assuming the
underlying phenomenon is not changing.
SEITE 22
True Score Theory
SEITE 23
Random and Systematic Error
in Measurement
SEITE 24
Estimating Reliability
= correlation
Standard
Deviation
SEITE 25
SD refers to the spread or variability of
the scores around their average in a single
sample. SD measures dispersion and is
measured in the same units as the
original measure.
Types of Reliability
Test-Retest
Reliability
Inter-rater or
Inter-Observer
Reliability
Internal
Consistency
Reliability
Parallel-Forms
Reliability
Consistency of the
results of two tests
constructed in the
same way from the
same domain
SEITE 26
Consistency of results
across items within a
test
Average Interitem Correlations
SEITE 27
Split-half Reliability
SEITE 28
Cronbach‘s Alpha
SEITE 29
Validity and Reliability
The shooting-target metaphor
SEITE 30
Thank you