CHAPTER 6: VALIDITY
➢ Validity
○ as applied to a test, is a
judgement or estimate of how
well a test measures what it
purports to measure in a
particular context.
➢ One way measurement specialists
have traditionally conceptualized
validity is according to three
categories: (trinitarian view)
1. content validity
○ scrutinizing the test’s content
2. criterion-related validity
○ relating scores obtained on the
test to other test scores or
other measures
3. construct validity
○ executing a comprehensive
analysis of
○ a. how scores on the test relate
to other test scores and
measures
○ b. how scores on the test can
be understood within some
theoretical framework for
understanding the construct
that the test was designed to
measure.
➢ FACE VALIDITY
○ relates more to what a test
appears to measure to the
person being tested than to
what the test actually
measures
○ a judgement concerning how
relevant the test items appear
to be
○ Face validity may be more a
matter of public relations than
psychometric soundness,but it
seems important nonetheless.
○ Does not contribute to
construct validity
➢ CONTENT VALIDITY
○ a judgement of how
adequately a test samples
behavior representative of the
universe of behavior that the
test was designed to sample
○ most common n objective
test,achievement test, aptitude
test
○ whether the test is a fair
sample of total potential
content.
➢ If more than half the panelists indicate
that an item is essential, that item has
at least some content validity. Greater
levels of content validity exist as
larger numbers of panelists agree that
a particular item is essential. Using
these assumptions, Lawshedeveloped
a formula termed the Content
Validity Ratio (CVR):
CVR = ne- (N/2) / (N/2)
Negative CVR: fewer than half the panelists
indicate “essential,” the CVR is negative.
Zero CVR: exactly half the panelists indicate
“essential,” the CVR is zero.
Positive CVR: more than half but not all the
panelists indicate “essential,” the CVRranges
between .00 and .99.
➢ CRITERION RELATED
VALIDITY
○ a judgement of how
adequately a test score can be
used to infer an individual’s
most probable standing on
some measure of interest—the
measure of interest being the
criterion.
➢ CRITERION
○ a standard on which a
judgement or decision may be
based.
Characteristics of Criterion:
Relevant - it is pertinent or applicable to the
matter at hand.
Valid - valid for the purpose for which it is
being used.
Uncontaminated - Criterion contamination is
the term applied to a criterion measure that has
been based, at least in part, on predictor
measures.
Two types of validity evidence are
subsumed under the heading
criterion-related validity:
➢ Concurrent validity
○ an index of the degree to
which a test score is related to
some criterion measure
obtained at the same time
(concurrently).
○ test scores are obtained at
about the same time that the
criterion measures are
obtained, measures of the
relationship between the test
scores and the criterion
provide evidence of
concurrent validity.
○ Candidates ability
➢ Predictive validity
○ an index of the degree to
which a test score predicts
some criterion measure.
○ test scores may be obtained at
one time and the criterion
measures obtained at a future
time, usually after some
intervening event has taken
place
○ the intervening event may take
varied forms, such as training,
experience, therapy,
medication, or simply the
passage of time
Judgments of criterion-related validity,
whether concurrent or predictive, are based on
two types of statistical evidence: the validity
coefficient and expectancy data.
➔ Validity Coefficient
○ a correlation coefficient that
provides a measure of the
relationship between test
scores and scores on the
criterion measure. The
validitycoefficient should be
high enough to result in the
identification and
differentiation of testtakers
with respect to target attribute.
➔ Incremental Validity
○ the degree to which an
additional predictor explains
something about the criterion
measure that is not explained
by predictors already in use.
➔ Expectancy Data
○ provide information that can
be used in evaluating the
criterion-related validity of a
test. Using a score obtained on
some test(s) or measure(s),
expectancy tables illustrate the
likelihood that the testtaker
will score within some
interval of scores on a
criterion measure—an interval
that may be seen as “passing,”
“acceptable,” and so on.
➢ CONSTRUCT VALIDITY
○ a judgement about the
appropriateness of inferences
drawn from test scores
regarding individual standings
on a variable called a
construct.
○ Skills and abilities are trying
to measure
Construct - an informed, scientific
idea developed or hypothesized to
describe or explain behavior.
Constructs are unobservable,
presupposed (underlying) traits that a
test developer may invoke to describe
test behavior or criterion performance.
EVIDENCE OF CONSTRUCT VALIDITY:
➢ Convergent Evidence
○ high correlation/similar
construct
○ evidence for the construct
validity of a particular test
may converge from a number
of sources, such as other tests
or measures designed to assess
the same (or a
similar)construct. Thus, if
scores on the test undergoing
construct validation tend to
correlate highly in the
predicted direction with scores
on older, more established,
and already validated tests
designed to measure the same
(or a similar) construct, this
would be an example of
convergent evidence.
➢ Divergent Evidence
○ A validity coefficient showing
little (that is, a statistically
insignificant) relationship
between test scores and/or
other variables with which
scores on the test being
construct-validation should
not theoretically be correlated
provides discriminant
evidence of construct validity.
➢ Factor Analysis
○ both convergent and
discriminant evidence of
construct validity can be
obtained by the use of factor
analysis
○ factor analysis is a shorthand
term for a class of
mathematical procedures
designed to identify factors or
specific variables that are
typically attributes,
characteristics, dimensions on
which people may differ
Exploratory factor analysis typically entails
“estimating, or extracting factors;deciding how
many factors to retain; and rotating factors to
an interpretable orientation”
Confirmatory factor analysis, “a factor
structure is explicitly hypothesized and is
tested for its fi t with the observed covariance
structure of the measured variables”
➢ Evidence of homogeneity
○ homogeneity refers to how
uniform a test is in measuring
a single concept
➢ Evidence of changes with age
○ subjects from different groups
➢ Evidence of pretest–posttest changes
○ evidence that test scores
change as a result of some
experience between a pretest
and a posttest can be evidence
of construct validity.
➢ Evidence from distinct groups
○ referred to as the method of
contrasted groups, one way of
providing evidence for the
validity of a test is to
demonstrate that scores on the
test vary in a predictable way
as a function of membership
in some group
○ use a group of subjects to
validate the test construct
VALIDITY,BIAS AND FAIRNESS
➢ TEST BIAS
○ Term bias as applied to
psychological and educational
tests may conjure up many
meanings having to do with
prejudice and preferential
treatment (Brown et al.,
1999).
➢ Rating error
○ a judgment resulting from the
intentional or unintentional
misuse of a rating scale
➢ Lenient Error - tendency to give
highly favorable ratings
➢ Severity Error - tendency to give low
ratings
➢ Central tendency error - tendency to
give ratings that are neither low or
high
➢ Halo Effect - rating judgement basing
on some associated characteristic of
theratee
➢ TEST FAIRNESS
○ fairness in a psychometric
context as the extent to which
a test is used in an impartial,
just, and equitable way.
➢ SCORING VALIDITY
○ Test score in accordance with
what the test is measuring.
○ Scoring validity is most
closely related to construct
validity
○ Scoring criteria should match
the skill you are trying to test