Schmidt & Hunter
Approach to r
Artifact Corrections
Statistical Artifacts
Extraneous factors that influence observed effect
Sampling error*
Reliability*
Range restriction*
Computational error
Dichotomization of variables
*Addressed in the analysis
Sampling error (in theory) has no systematic effect on ES; just
noise. Less than perfect reliability and typical range restriction
serve to reduce the observed ES. Differences between studies in
reliability and range restriction increase the between studies
variance (REVC) in theoretically barren ways.
ryy =1, M=100, SD=10
80
100
120
Effect of a single reliability
60
ryy =.6, M=77, SD=7.7
sam1
sam2
If we multiply a
distribution by a
constant (e.g., .77
(sqrt(.6)), the new mean
is the old mean times
the constant (100*.77)
and the new standard
deviation is the old SD
times the constant
(10*.77=7.7).
Unreliability reduces the
mean and variability. If
we correct, increase
both.
Effect of multiple reliabilities
Study 1 rxtyt = .8, r yy = .9, rxy = .8*sqrt(.9) = .76
Study 2 rxtyt = .8, r yy = .7, rxy = .8*sqrt(.7) = .67
Differences in reliability across studies will increase the
variance of observed correlations
Range Restriction/Enhancement
These are examples
of direct RR.
Psychometric Meta-Analysis
Disattenuation for reliability
rC 
rxy
Correction for both
rxx ryy
rC 
rxy
rC 
rxy
Correction for IV
rxx
Correction for DV
ryy
Suppose rxy = .30, rxx = ryy = .80. Then:
rC 
rxy
rxx ryy

.30
.80 2
 .375
rC 
rxy
rxx

.3
 .33541
.8
Direct Range
Restriction/enhancement
rC 
U X rxy
UX 
(U X2  1)rxy2  1
SDunrestri cted
SDrestrict ed
Suppose rxy = .33, SD1=12, SD2 = 20. Then:
UX 
rC 
SDunrestri cted 20

 1.67
SDrestrict ed
12
U X rxy
(U  1)r  1
2
X
2
xy

1.67(.33)
(1.67  1)(.33 )  1
2
Can also invert by uX = 1/UX
2
 .50
Indirect RR
rxxi Reliability of IV in restricted sample (job
incumbents in I/O validation study).
rxxa Reliability of IV in unrestricted sample (job
applicants in I/O validation study).
sT
uT 
Ratio of SD of true scores; analogous to uX.
ST
You will need rxxa for DIRECT
rxxa  1  u X2 (1  rxxi )
range restriction correction.
2
rxxi  1  U X (1  rxxa )
You will need uT AND rxxi for
1/ 2
2
 u X  (1  rxxa ) 
INDIRECT range restriction
uT  

r
correction.
xxa


Meta-Analysis of corrected r
If information is available can correct r for each study
Compute M-A on the corrected values
Can also be done with assumed distributions, but I do
not recommend it.
Steps (1)
Record data (N, r, artifact values rxx, etc.)
Compute the corrected correlation for each study:
If there is only 1 kind of artifact, disattenuation is simple:
rC 
ro
a
Where a is the disattenuation factor.
Note ro is observed and rC is corrected.
If there is range restriction, things are tricky. If
INDIRECT range restriction, then use Ut instead of Ux
and disattenuate for reliability before adjusting for range
restriction. Use reliabilities from the restricted group.
If DIRECT range restriction, adjust for ryy, then range
restriction, then rxx, but rxxa, the reliability in the
unrestricted group.
Steps (1b)
For each study, compute compound attenuation factor:
ro
A
rC
Compute sampling variance of uncorrected r:
Var (eo i )  [1  ro2 ]2 /( N i  1)
Note this is sampling variance for one study.
Steps (2)
Compute sampling variance of disattenuated r:
ve  Var (eo ) / A2
If there is range restriction, then do the following 2 steps.
Compute adjustment for range restriction:
arr  1 /[(U X2  1)ro2  1]
Adjust sampling variance of disattenuated r:
ve  arr2 ve
Compute weights:
wi N i A
2
i
Note A is the compound attenuation
factor.
Steps (3)
Compute the weighted mean:
rC
wr


w
i Ci
 ̂
i
Compute the weighted variance:
Var (rC ) 
2
w
[
r

r
]
 i Ci C
w
i
Compute average corrected r sampling error:
w ve

Ave(ve) 
w
i
i
i
Compute variance of rho: Var(  )  ˆ 2  Var(rC )  Ave(ve)
Psychometric M-A data
Study Ni
r
rxxi
ryy
Ux
1
200
.20
.90
.80
1.5
2
100
.20
.80
.82
1.5
3
150
.40
.85
.88
1.0
4
80
.40
.85
.90
1.2
Mean 132.5 .30
.85
.85
1.3
We did bare bones already. Now we will analyze 3 ways: (1) just
criterion reliability, (2) all artifacts with INDIRECT RR, (3) all
artifacts DIRECT rr.
Correct ryy only (1)
Suppose we only wish to correct for criterion unreliability.
Study 1 r = .20, rxx = .90, ryy = .80, Ux = 1.5
Disattenuation ryy : rC = .2/sqrt(.8) = .223607.
Compound attenuation factor A = .20/.223607
= .894.
Correct ryy only (2)
Study r
A
rC
N
V1
V2
1
.2
.894
.224
200
.0042
.0053
2
.2
.906
.221
100
.0085
.0104
3
.4
.938
.426
150
.0057
.0064
4
.4
.949
.422
80
.0107
.0118
ro  .2868
V1  Var (eo )  [1  .2868 ] /( N i  1)
2 2
V2  Var (eo ) / A2
Correct ryy only (3)
Study
rC
A
Ni
wi
wirC
1
.224
.894
200
160
35.78
2
.221
.906
100
82
18.11
3
.426
.938
150
132
56.28
4
.422
.949
80
72
30.36
446
140.53
Sum
wi N i A
2
i
rC
wr


w
i Ci
i
 ˆ  140.53 / 446  .315
Correct ryy only (4)
Study rC
wi
Wi[rC-rbarC]2
V2
1
.224
160
1.339
.0053 .8465
2
.221
82
.728
.0104 .8508
3
.426
132
1.635
.0064 .8479
4
.422
72
.817
.0118 .8529
446
4.52
Sum
rC  .315
2
w
[
r

r
]
 i Ci C
wV2
3.3981
Var (rC ) 
 4.52 / 446  .0101
wi

w
V
 i 2i  3.3981 / 446  .0076
Ave(ve) 
 wi
Correct ryy only (5)
Var (  )  Var (rC )  Ave(ve)  .0101  .0076  .0025
SD  Var (  )  .0502
BareBones
M
.2868
V(r)
.0098
SDrho
.0585
95CRlow .17
ryy
corrected
.315
.0101
.0502
.22
95CRup
.41
.40
All corrections, Indirect RR
Study Ni
r
rxxi
ryy
Ux
1
200
.20
.90
.80
1.5
2
100
.20
.80
.82
1.5
3
150
.40
.85
.88
1.0
4
80
.40
.85
.90
1.2
Mean 132.5 .30
.85
.85
1.3
ro  .2868
Already know bare-bones mean.
Indirect RR (2)
Study
ro
rxxi
ryy
Ux
ux
rxxa
UT
1
.20
.90
.80
1.5
.67
.96
1.55
2
.20
.80
.82
1.5
.67
.91
1.60
3
.40
.85
.88
1.0
1
.85
1
4
.40
.85
.90
1.2
.83
.90
1.23
rxxa  1  u (1  rxxi )
2
X
 u  (1  rxxa ) 
uT  

r
xxa


2
X
1/ 2
Indirect RR (3)
Study
ro
rxxi
ryy
UT
1
.20
.90
.80
1.55 .236 .351 .570
2
.20
.80
.82
1.60 .247 .378 .530
3
.40
.85
.88
1
4
.40
.85
.90
1.23 .457 .535 .747
rc1  ro / rxxi ryy
rc 
rc1
rc
A
.462 .462 .865
U T rc1
(U  1)r  1
2
T
2
c1
A  ro / rc
Indirect RR (4)
Study
Ni
1
200 .351 .570 64.93
22.79
.480
2
100 .378 .530 28.04
10.59
.099
3
150 .462 .865 112.2
51.89
.073
4
80
23.92
.431
rc
A
wi
.535 .747 44.69
Sum
wrc
wi (rC  rC ) 2
249.86 109.19 1.082
wi N i A
2
i
109.19
1.082
rC 
 .437 Var (rC ) 
 .0043
249.86
249.86
Indirect RR (5)
Study
Ni
rc
A
wi
V1
1
200 .351 .570 64.93
.0042 .013 .95
.012 .76
2
100 .378 .530 28.04
.0085 .030 .94
.027 .75
3
150 .462 .865 112.2
.0057 .008 1
.008 .85
4
80
.0107 .019 .92
.016 .73
.535 .747 44.69
Sum
249.86
V1  Var (eo )  [1  .28682 ]2 /( N i  1)
arr  1 /[(U T2  1)ro2  1]
V2
arr
V3
wV3
3.09
V2  Var (eo ) / A2
3.09
2
Ave(ve) 
 .0124
V3  arrV2
249.86
Indirect RR (6)
Var (  )  Var (rC )  Ave(ve)  .0043  .0124  .008
SD  Var (  )  0
BareBones
ryy
corrected
Full
indirect
correction
M
.2868
.315
.437
V(r)
.0098
.0101
.0043
SDrho
.0585
.0502
0
95CRlow .17
.22
.44
95CRup
.41
.44
.40
Direct Range Restriction (1)
Study
ro
rxxi ryy
Ux
rxxa
rc1
rc2
rc
1
.20 .90 .80
1.5
.96
.22
.33
.33
2
.20 .80 .82
1.5
.91
.22
.32
.34
3
.40 .85 .88
1.0
.85
.43
.43
.46
4
.40 .85 .90
1.2
.90
.42
.49
.51
rxxa  1  u X2 (1  rxxi )
rC1  ro / ryy
rC 2 
U X rC1
(U  1)r  1
2
X
2
C1
rC  rC 2 / rxxa
Direct RR (2)
Study
ro
Ni
w
wrc
1
.20 .33 .60
200
72.20
24.03
2
.20 .34 .59
100
35.23
11.87
3
.40 .46 .86 150
112.2
51.89
4
.40 .51 .78
48.30
24.86
rc
A
80
267.92 112.66
Sum
wi N i A
2
i
rC
wr


w
i Ci
i
 ˆ  112.66 / 267.92  .42
Direct RR (3)
Study
rc
1
.33 72.20
.55
2
.34 35.23
.25
3
.46 112.2
.20
4
.51 48.30
.43
Sum
rC  .42
w
wi (rC  rC ) 2
267.92 1.43
1.43
Var (rC ) 
 .0053
267.92
Direct RR (4)
Study
Ni
1
rc
A
wi
V1
200 .33
.60
72.20
.0042 .012 .95
.011 .77
2
100 .34
.59
35.23
.0085 .024 .95
.022 .77
3
150 .46
.86 112.2
.0057 .008 1
.008 .85
4
80
.78
.0107 .018 .93
.015 .74
.51
Sum
48.30
267.92
V1  Var (eo )  [1  .28682 ]2 /( N i  1)
arr  1 /[(U X2  1)ro2  1]
V2
arr
V3
wV3
3.13
V2  Var (eo ) / A2
3.13
2
Ave(ve) 
 .012
V3  arrV2
267.92
Direct RR (5)
Var (  )  Var (rC )  Ave(ve)  .0053  .0117  .006
SD  Var (  )  0
BareBones
ryy
corrected
Full
Full direct
indirect
correction
correction
M
.2868
.315
.437
.42
V(r)
.0098
.0101
.0043
.005
SDrho
.0585
.0502
0
0
95CRlow .17
.22
.44
.42
95CRup
.41
.44
.42
.40
Philosophical Issues
Psychometric M-A
(Schmidt & Hunter)
Other methods (Hedges,
Rosenthal)
Want to know relations
among constructs, not the
measured variables
Avoid extrapolating to
data that you have not
observed
Sometimes unrealistic
estimates at the end
Parameter estimates may
be incorporated into
weights leading to bias
Problematic prediction
intervals