Meta-Analysis using HLM 6.0
Yaacov Petscher
Florida Center for Reading Research
Why use HLM?
• Nested structure
• Necessity of special models
– Variation at both subject and study levels
“Special Case” of HLM
• If ES are based on n ≥ 30, we assume
approximate normal distribution with
sampling variance assumed to be known
• V-known models run in Interactive Mode
– Time to brush up on your DOS command
code!
Standardized Mean Difference
• No raw data for us
– Must rely on stats to be converted to single
metric
• Many types of statistics that may be used
Z
χ²
r
t
F
r²
M/SD
p-value
Level-1 (Within-Studies) Model
d j   j  ej
dj
= any standardized effect measure from study j
j
= the corresponding population parameter
ej
= sample error associated with d
Level-2 (Between-Studies) Model
 j   0   sWsj  u j
0
= grand mean effect size
s
= regression coefficients
Wsj = study characteristics (moderators)
uj
= level 2 random error
Combined Model
d j   0    sWs  u j  e j
s
Estimation
Empirical Bayes Estimator
   j d j  (1   j )(ˆ0   ˆsWsj )
*
j
s
where
 j   /(  V j )
EB Estimates
• Level 1
– May be used as a shrinkage estimator to
identify potential outliers
• Shrinkage in the direction of the grand mean
• Level 2
– Supplying the grand mean provides an
estimate of the conditional shrinkage
• Shrinkage towards a value that is conditional on
the amount of prior contacts (WEEKS)
The following example will be run using data from Raudenbush & Bryk
Chapter 7 data (pg 211)
The Effect of Teacher Expectancy on Pupil IQ
Study
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Week s
2
3
3
0
0
3
3
3
0
1
0
0
1
2
3
3
1
2
3
ES
0.03
0.12
-0.14
1.18
0.26
-0.06
-0.02
-0.32
0.27
0.8
0.54
0.18
-0.02
0.23
-0.18
-0.06
0.3
0.07
-0.07
Std Error
0.125
0.147
0.167
0.373
0.369
0.103
0.103
0.22
0.164
0.251
0.302
0.223
0.289
0.29
0.159
0.167
0.139
0.094
0.174
Empirical Bayes Estimates
Unconditional Model Conditional Model
0.05
0.09
0.10
-0.06
0.00
-0.06
0.22
0.41
0.11
0.41
-0.01
-0.06
0.02
-0.06
-0.03
-0.06
0.16
0.41
0.25
0.25
0.16
0.41
0.11
0.41
0.06
0.25
0.11
0.09
-0.03
-0.06
0.03
-0.06
0.19
0.25
0.07
0.09
0.02
-0.06
Some Calculations
• Need the Conditional Variances
– Since d in this model is Fisher’s r to Z transformation,
the formula is
1
vi 
(n  3)
Since we’re not given n we need to calculate
another way…..ideas?
YAY!
• Simply square your standard error
Study
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
ES
0.03
0.12
-0.14
1.18
0.26
-0.06
-0.02
-0.32
0.27
0.8
0.54
0.18
-0.02
0.23
-0.18
-0.06
0.3
0.07
-0.07
Std Error
0.125
0.147
0.167
0.373
0.369
0.103
0.103
0.220
0.164
0.251
0.302
0.223
0.289
0.290
0.159
0.167
0.139
0.094
0.174
Study
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
ES
0.03
0.12
-0.14
1.18
0.26
-0.06
-0.02
-0.32
0.27
0.8
0.54
0.18
-0.02
0.23
-0.18
-0.06
0.3
0.07
-0.07
Std Error
0.125
0.147
0.167
0.373
0.369
0.103
0.103
0.220
0.164
0.251
0.302
0.223
0.289
0.290
0.159
0.167
0.139
0.094
0.174
Vj
0.0156
0.0216
0.0279
0.1391
0.1362
0.0106
0.0106
0.0484
0.0269
0.0630
0.0912
0.0497
0.0835
0.0841
0.0253
0.0279
0.0193
0.0088
0.0303
Data File Prep Considerations
• Since meta-analysis in HLM is a V-known model,
only one data file is used
Study
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
ES
0.03
0.12
-0.14
1.18
0.26
-0.06
-0.02
-0.32
0.27
0.8
0.54
0.18
-0.02
0.23
-0.18
-0.06
0.3
0.07
-0.07
Vj
0.0156
0.0216
0.0279
0.1391
0.1362
0.0106
0.0106
0.0484
0.0269
0.0630
0.0912
0.0497
0.0835
0.0841
0.0253
0.0279
0.0193
0.0088
0.0303
Week s
2.000
3.000
3.000
0.000
0.000
3.000
3.000
3.000
0.000
1.000
0.000
0.000
1.000
2.000
3.000
3.000
1.000
2.000
3.000
Data File Prep Considerations, cont
• Four key features to data prep (assume
using SPSS)
– Column 1 = ID in character format
– Column 2 = ES estimates
– Column 3 = Variance estimates
– Column 4-n = Potential level-2 predictors
Formatting
• Variable View
– Column 1
• String, width = 2, decimal = 0
– Columns 2-n
• Numeric, width = 12, decimal = 3
– Save as a Fixed ASCII (.dat) file
– Hold onto your output, you’re gonna need it!
• You should now have a list of the variables and
associated Format statements
HLM in Batch Mode
• Bring up your computer’s Command Prompt
– Typically found in “Accessories”
• By default, you should see C:\>
– If not, type c: and hit enter
• At this point you want to locate HLM
– Type dir, hit enter
• cd program files, enter
• cd HLM6, enter
• hlm2, enter
– We’re now ready to begin!
HLM2- MDM File Creation
C:\Program Files\HLM6\hlm2
Will you be starting with raw data? y
Is the input file a v-known file? y
How many level-1 statistics are there? 1
How many level-2 predictors are there? 1
Enter 8 character name for level-1 variable number 1: Zes
Enter 8 character name for level-2 variable number 1: weeks
Input format of raw data file (the first field must be the character ID)
format: (a2, 3f12.3)
What file contains the data: e:\test.dat
Enter name of MDM file: e:\test.mdm
19 groups have been processed
C:\Program Files\HLM6>
Specifying UC HLM Model
C:\HLM6> hlm2 e:\test.mdm
SPECIFYING AN HLM MODEL
Level-1 predictor variable specification
Which level-1 predictors do you wish to use?
The choices are:
For ZES enter 1
Level-1 predictors? (Enter 0 to end) 1
Level-2 predictor variable specification
Which level-2 variables do you wish to use?
The choices are:
For WEEKS enter 1
Which level-2 predictors to model ZES?
Level-2 predictor? (Enter 0 to end) 0
ADDITIONAL PROGRAM FEATURES
Select the level-2 variables that you might consider for
Inclusion as predictors in subsequent models.
The choices are:
For WEEKS enter 1
Which level-2 variables to model ZES?
Level-2 variable? (Enter 0 to end) 0
Do you wish to use any of the optional hypothesis testing procedures? n
OUTPUT SPECIFICATION
Do you want a level-2 residual file? n
How many iterations do you want to do? 10000
Do you want to see OLS estimates for all of the level-2 units? n
Enter a problem title: lvl1
Enter name of output file: e:\lvl1.lis
Results for UC Model
----------------------------------------------------------------------------------------Standard
Approx.
Fixed Effect
Coefficient
Error
T-ratio
d.f.
----------------------------------------------------------------------------------------For
ZES, B1
INTRCPT2, G10
0.084376 0.052039 1.621
18
-----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------Random Effect
Standard Variance df Chi-square P-value
Deviation Component
-----------------------------------------------------------------------------------------------ZES,
U1
0.13896
0.01931
18
36.25115 0.007
------------------------------------------------------------------------------------------------
Significant variability exists in true-effect sizes
EB Estimation Level 1
   j d j  (1   j )(ˆ0   ˆsWsj )
*
j
s
Since there are no predictors at Level 1, the last term is omitted, leaving us with
   j d j  (1   j )ˆ0
*
j
Using Excel to Calculate EB
 j   /(  V j )
ES
Variance
0.03
0.0156
0.12
0.0216
-0.14
0.0279
1.18
0.1391
0.26
0.1362
-0.06
0.0106
-0.02
0.0106
-0.32
0.0484
0.27
0.0269
0.80
0.0630
0.54
0.0912
0.18
0.0497
-0.02
0.0835
0.23
0.0841
-0.18
0.0253
-0.06
0.0279
0.30
0.0193
0.07
0.0088
-0.07
0.0303
Using the variance component from
lvl 1 Model, create Lambda using
the formula function
ES
Variance
0.03
0.0156
0.12
0.0216
-0.14
0.0279
1.18
0.1391
0.26
0.1362
-0.06
0.0106
-0.02
0.0106
-0.32
0.0484
0.27
0.0269
0.80
0.0630
0.54
0.0912
0.18
0.0497
-0.02
0.0835
0.23
0.0841
-0.18
0.0253
-0.06
0.0279
0.30
0.0193
0.07
0.0088
-0.07
0.0303
lambda
0.548736
0.467877
0.405212
0.120155
0.122453
0.641697
0.641697
0.281899
0.413979
0.231704
0.172408
0.276448
0.185328
0.184287
0.429078
0.405212
0.495812
0.682569
0.385583
   j d j  (1   j )ˆ0
*
j
Supplying the grand mean ES
into the Excel formula function
allows us get the EB estimates
See the difference in results?
ES
Variance
0.03
0.0156
0.12
0.0216
-0.14
0.0279
1.18
0.1391
0.26
0.1362
-0.06
0.0106
-0.02
0.0106
-0.32
0.0484
0.27
0.0269
0.80
0.0630
0.54
0.0912
0.18
0.0497
-0.02
0.0835
0.23
0.0841
-0.18
0.0253
-0.06
0.0279
0.30
0.0193
0.07
0.0088
-0.07
0.0303
lambda
0.548736
0.467877
0.405212
0.120155
0.122453
0.641697
0.641697
0.281899
0.413979
0.231704
0.172408
0.276448
0.185328
0.184287
0.429078
0.405212
0.495812
0.682569
0.385583
Ebdelta
0.05
0.10
-0.01
0.21
0.10
-0.01
0.02
-0.03
0.16
0.25
0.16
0.11
0.06
0.11
-0.03
0.03
0.19
0.07
0.02
1.40
1.20
1.00
0.80
0.60
0.40
0.20
0.00
ES
-0.20
-0.40
Ebdelta
Specifying CL2 HLM Model
C:\HLM6> hlm2 e:\test.mdm
SPECIFYING AN HLM MODEL
Level-1 predictor variable specification
Which level-1 predictors do you wish to use?
The choices are:
For ZES enter 1
Level-1 predictors? (Enter 0 to end) 1
Level-2 predictor variable specification
Which level-2 variables do you wish to use?
The choices are:
For WEEKS enter 1
Which level-2 predictors to model ZES?
Level-2 predictor? (Enter 0 to end) 1
ADDITIONAL PROGRAM FEATURES
Select the level-2 variables that you might consider for
Inclusion as predictors in subsequent models.
The choices are:
For WEEKS enter 1
Which level-2 variables to model ZES?
Level-2 variable? (Enter 0 to end) 0
Do you wish to use any of the optional hypothesis testing procedures? n
OUTPUT SPECIFICATION
Do you want a level-2 residual file? n
How many iterations do you want to do? 10000
Do you want to see OLS estimates for all of the level-2 units? n
Enter a problem title: lvl2
Enter name of output file: e:\lvl2.lis
Results for CL2 Model
------------------------------------------------------------------------------------------------Standard
Approx.
Fixed Effect
Coefficient
Error
T-ratio
d.f. P-value
------------------------------------------------------------------------------------------------For
ZES, B1
INTRCPT2, G10
0.408572 0.087146 4.688
17 0.000
WEEKS, G11
-0.157963 0.035943 -4.395
17 0.000
-------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------Random Effect
Standard
Variance
df
Chi-square P-value
Deviation Component
-----------------------------------------------------------------------------------------------ZES,
U1
0.00283
0.00001
17
16.53614 >.500
------------------------------------------------------------------------------------------------
EB Estimation Level 2
Since
ˆ  0
then  j
  /(  V j ) = 0
 *j  ˆ0  ˆ1 (WEEKS) j
Using G10 and G11, we can calculate
the EB estimates for Level 2
and we’re left with
ES
Weeks
EBLvl2
0.03
2
0.09
0.12
3
-0.06
-0.14
3
-0.07
1.18
0
0.41
0.26
0
0.41
-0.06
3
-0.07
-0.02
3
-0.07
-0.32
3
-0.07
0.27
0
0.41
0.80
1
0.25
0.54
0
0.41
0.18
0
0.41
-0.02
1
0.25
0.23
2
0.09
-0.18
3
-0.07
-0.06
3
-0.07
0.30
1
0.25
0.07
2
0.09
-0.07
3
-0.07
1.40
1.20
1.00
0.80
0.60
0.40
0.20
0.00
ES
-0.20
-0.40
EBLvl2
EB Estimation Level 2
• Since our Level-2 predictor takes on one
of four different values, the shrinkage is
towards one of the four points.
End
(for now)