

Degree to which inferences made using
data are justified or supported by evidence
Some types of validity
◦ Criterion-related
◦ Content
◦ Construct


All part of unitarian view of validity
Constructs - theoretical abstractions
aimed at organizing and making sense of
our environment; they are LATENT





A criterion is any variable you wish to
explain and/or predict
They are the key to well-developed theory,
good measurement, and strong research
design
Ultimate criterion
Multidimensional nature of criteria
Intermediate criteria




Process of establishing a relationship
between variables
Predictive, concurrent, postdictive
Usually based on correlation or regression
equation
Low reliability will attenuate or mask
relationships




Selection Ratio – proportion of the individuals
in the sample who are selected of the total
number
Base rate – percent of successful individuals
under random selection
Range Restriction
Differential Prediction for different subgroups
False
Negatives
VP
FN
Yc
Unsuccessful
FP
VN
Reject
False
Positives
Xc
Successful
Accept
FN+VP=BR
VN+FP=1-BR
VP+FP=SR
FN+VN=1-SR
FN+VP=BR
VN+FP=1-BR
VP+FP=SR
FN+VN=1-SR
False
Negatives
VP
Yc
Successful
FN
VN
Reject
Unsuccessful
FP
False
Positives
Xc
Accept
False
Negatives
VN
Successful
VP
FN
Yc
FP
False
Positives
Unsuccessful FN+VP=BR
VN+FP=1-BR
VP+FP=SR
FN+VN=1-SR
Reject
Xc
Accept
VP
FN
FN+VP=BR
VN+FP=1-BR
Yc
FP
VN
VP+FP=SR
FN+VN=1-SR
Xc







Even low correlations can lead to large increases
in selection efficiency
SR and BR have strong influences
When SR is small (choose few), fewer FP and
more FN
When SR is large, fewer FN and more FP
When BR is large (many can be successful), SR
and validity have little effect on selection
efficiency
Most gains in success ratio when BR = .50 and
SR is small (e.g., .10)
The tradeoffs depend on purpose of selection
- Direct
- Indirect
- Ambiguous
Y
X
Y
X
Same prediction for each group
Y
X
Different prediction for each group

Extent to which items or measures cover the
content area the test purports to measure
◦ Expert judges determine if a measure came from a
particular content domain
◦ Scoring and content is based upon theory
◦ If measures are from same content domain, should
demonstrate high reliability
◦ If low internal consistency reliability, low content
validity



Validity of inferences about latent
unobserved variables on the basis of
observed variables
Does a measure assess what it is
intended to assess? Do the variables
relate in theoretically meaningful ways?
Low reliability will make it difficult to
assess the nature of a particular
construct and attenuate relationships
with other constructs
Construct Validity
Theory
Cause
Construct
Measure or
Manipulation
What you think
True
Relationship
Observed
Relationship
Effect
Construct
Observed
Outcomes
What you see
Can we generalize to the constructs from the
measures?
3
Ability
to Learn
Anxiety
2
Measure
of Anxiety
(X)
4
1
Test Score
(Y)
Vegetarianism
5
Salads
Eaten (Z)



Internal Structure Analysis
Cross Structure Analysis
Nomological network (Cronbach & Meehl)

Factor Analysis

Exploratory - Useful When:

Confirmatory - Useful When:

◦ Used to identify factors or dimensions that
underlie relations among observed variables
◦ No info on internal structure available
◦ Factor structures may look different than original
scale
◦ You have reservations about previous factor
analyses
◦ You have some idea of the internal structure
◦ Confirming factor structures from previous studies
Necessary but not sufficient to establish construct
validity
Anxiety
X1
e1
Ability to Learn
X2
X3
e2
e3
X4
Z1
Z2
e4
e5
e6
Z3
e7


Embedded in nomological network
(nomological validity)
Test of hypotheses by examining
relationships between different indicators
of underlying constructs
◦ e.g., leadership style based on reports from
subordinates and leadership self-report
inventory

Relies on multiple methods of
measurement



A representation of constructs of interest in a
study, their observable manifestations
(measures), and the interrelationships among
and between them
Cronbach & Meehl said this is necessary to
establish construct validity
Elements include:
◦ Specify linkage between constructs (hypotheses)
◦ Operationalize constructs (specify measurement)



Convergent validity - Convergence among
different methods designed to measure
the same construct
Discriminant validity - Distinctiveness of
constructs, demonstrated by divergence of
methods designed to measure different
constructs
Multi-Trait Multi-Method

Heterotrait-Monomethod
◦ Different traits, same method

Heterotrait-Heteromethod
◦ Different traits, different methods

Monotrait-Heteromethod
◦ Same trait, different methods
◦ Validity diagonals

Monotrait-Monomethod
◦ Same trait, same method
◦ Reliability diagonals
Method1
A1 B1 C1
A1 (.89)
B1 .51 (.89)
C1 .38 .37 (.76)
Method2
A2 B2 C2
M2
A2 .57 .22 .09
B2 .22 .57 .10
C2 .11 .11 .46
(.93)
.68 (.94)
.59 .58 (.84)
M3
A3 .56 .22 .11
B3 .23 .58 .12
C3 .11 .11 .45
.67 .42 .33 (.94)
.43 .66 .34 .67 (.92)
.34 .32 .58 .58 .60 (.85)
M1
Method3
A3 B3 C3






Specify the nomological net (expected + and
- relationships) of expected relations
Establish reliability
Check convergence with other preexisting
measures of the construct (convergent
validity)
Factor analysis
Empirical studies of relatedness
Empirical studies of discriminability

Take the hypotheses you developed in
assignment 2 and the variables that were
included in them.
◦ Draw a picture of what you believe the nomological
network of these variables would look like
◦ What alternative measures of each variable might you
use (different than those specified in Assignment 3)
to establish convergent validity?
◦ Draw what an MTMM construct validity chart would
look like that includes each variable in your study and
the original and alternative measures you identified
for each construct. Specify whether each correlation
would be expected to be Hi, Low or Moderate.