Funded through the ESRC’s Researcher
Development Initiative
Session 3.3 & 3.4: Teacher Expectancy
Example
Prof. Herb Marsh
Ms. Alison O’Mara
Dr. Lars-Erik Malmberg
Department of Education,
University of Oxford
Session 3.3 & 3.4: Teacher Expectancy Example
Establish
research
question
Define
relevant
studies
Develop code
materials
Data entry
and effect size
calculation
Pilot coding;
coding
Locate and
collate studies
Main analyses
Supplementary
analyses
(Meta-analysis data from Raudenbush & Bryk,
2002)
 Do teacher expectations influence student IQs?
 Teachers led to have high expectations of
experimental (through bogus feedback) but not
control students.
 The focus is on the effect of how long teachers
knew students prior to the experimental
intervention.
Teacher Expectancy Effects on IQ (Metaanalysis data from Radudenbush & Bryk, 2002
Do teacher expectations influence student IQs? Teachers led to
have high expectations of experimental (through bogus feedback)
but not control students. Focus here is on the effect of how long
teachers knew students prior to the experimental intervention.
Teacher Expectancy Effects on IQ (Metaanalysis data from Radudenbush & Bryk, 2002
6
Potential
Outliers
90th %tile
75th %tile
Median
25th %tile
10th %tile
Mean effect size and homogeneity analyses
SPSS Commands
File: TCH-EXPT meta-analysis 19cases.sav
compute w = 1/SE ** 2.
Include "N:\MetaAnalysis\SPSS\METAES.SPS"
MeanES ES=d /W=w.
MeanES MACRO
2523
2524
2525
2526
2527
2529
2530
2532
2534
2535
2536
2537
2539
2540
2541
2542
2543
2545
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
*-------------------------------------------------------------*' Macro for SPSS/Win Version 6.1 or Higher
*' Written by David B. Wilson (dwilson@crim.umd.edu)
*' Meta-Analyzes Any Type of Effect Size
*' To use, initialize macro with the include statement:
*' INCLUDE "[drive][path]MeanES.SPS" .
*' Syntax for macro:
*' MeanES ES=varname /W=varname /PRINT=option .
*' E.g., MeanES ES = D /W = IVWEIGHT .
*' In this example, D is the name of the effect size variable
*' and IVWEIGHT is the name of the inverse variance weight
*' variable. Replace D and INVWEIGHT with the appropriate
*' variable names for your data set.
*' /PRINT has the options "EXP" and "IVZR". The former
*' prints the exponent of the results (odds-ratios) and
*' the latter prints the inverse Zr transform of the
*' results. If the /PRINT statement is ommitted, the
*' results are printed in their raw form.
MeanES MACRO
MeanES ES=d /W=w.
------- Distribution Description --------------------------------N
Min ES
Max ES
Wghtd SD
19.000
-.320
1.180
.218
------- Fixed & Random Effects Model ----------------------------Mean ES
-95%CI
+95%CI
SE
Z
P
Fixed
.0603
-.0112
.1319
.0365
1.6539
.0981
Random
.0893
-.0201
.1987
.0558
1.6005
.1095
------- Random Effects Variance Component -----------------------v
=
.025920
------- Homogeneity Analysis ------------------------------------Q
df
p
35.8254
18.0000
.0074
Random effects v estimated via noniterative method of moments.
MeanES MACRO
MeanES ES=d /W=w.
Conclusions: Small (NS) effect size based on both Fixed &
Random models. Significant unexplained variance
suggesting non-generalisability of effects across studies
and the need for a random effects model.
------- Distribution Description --------------------------------N
Min ES
Max ES
Wghtd SD
19.000
-.320
1.180
.218
------- Fixed & Random Effects Model ----------------------------Mean ES
-95%CI
+95%CI
SE
Z
P
Fixed
.0603
-.0112
.1319
.0365
1.6539
.0981
Random
.0893
-.0201
.1987
.0558
1.6005
.1095
------- Random Effects Variance Component -----------------------v
=
.025920
------- Homogeneity Analysis ------------------------------------Q
df
p
35.8254
18.0000
.0074
Random effects v estimated via noniterative method of moments.
Analogue to the ANOVA analyses
MetaF MACRO (ANOVA)
based on categorical weeks
METAF AF=d /W=w /group = wkcat/MODEL=ML .
Conclusions: Large effect
Q) ------of weeks (20.4/35.8=
p
.0001 57% var expl);
.4198
.0074 Total Residual, variance
component & residual by
group all NS;
ES significant for 1st two
groups, NS last two
groups
------- Analog ANOVA table (Homogeneity
Q
df
Between
20.3788
3.0000
Within
15.4466
15.0000
Total
35.8254
18.0000
------- Q by Group ------Group
Qw
df
p
.0000
6.2140
4.0000
.1837
1.0000
4.9825
2.0000
.0828
2.0000
.4045
2.0000
.8169
3.0000
3.8456
7.0000
.7974
------- Effect Size Results Total
------Mean ES
SE
-95%CI
+95%CI
Z
P
k
Total
.0603
.0365
-.0112
.1319
1.6539
.0981 19.0000
------- Effect Size Results by Group ------Group Mean ES
SE
-95%CI
+95%CI
Z
P
k
.0000
.3620
.1099
.1466
.5774
3.2939
.0010
5.0000
1.0000
.3516
.1121
.1319
.5712
3.1367
.0017
3.0000
2.0000
.0665
.0727
-.0760
.2091
.9147
.3604
3.0000
3.0000
-.0630
.0500
-.1611
.0350 -1.2605
.2075
8.0000
------- Maximum Likelihood Random Effects Variance Component ------v
=
.00000
se(v) =
.00648
Regression analyses
MetaReg MACRO (Regression)
based of categorical weeks
METAREG ES=d /IVS =/W=w wkcat /MODEL=ML .
Conclusions: Large
------- Descriptives ------effect of weeks
Mean ES
R-Square
k
(54% var expl);
.0603
.5375
19.0000
Constant term
------- Homogeneity Analysis ------highly significant
Q
df
p (at intercept = 0);
Model
19.2573
1.0000
.0000 Residual variance
Residual
16.5680
17.0000
.4840 NS (variance
Total
35.8254
18.0000
.0074 component = 0)
------- Regression Coefficients ------B
SE -95% CI +95% CI
Z
P
Beta
Constant
.4072
.0871
.2366
.5779 4.6775
.0000
.0000
-.1573
.0358 -.2275 -.0870 -4.3883
.0000 -.7332
------- Maximum Likelihood Random Effects Variance Component ------v
= .00000
se(v) = .00648
MetaReg MACRO (Regression)
based on uncategorical weeks
METAREG ES=d /W=w/IVS=weeks /MODEL=ML .
Conclusions: Large
effect of weeks (21%
var expl);
------- Descriptives ------Mean ES
R-Square
k
.0686
.2107
19.0000
Constant term highly
------- Homogeneity Analysis ------significant (at
Q
df
p intercept = 0);
Model
6.7540
1.0000
.0094 Residual var NS;
Residual
25.3035
17.0000
.0881 Does not do as well as
Total
32.0575
18.0000
.0216 categorised weeks
------- Regression Coefficients ------B
SE -95% CI +95% CI
Z
P
Beta
Constant
.1622
.0550
.0544
.2699 2.9498
.0032
.0000
weeks
-.0132
.0051 -.0232 -.0032 -2.5988
.0094 -.4590
------- Maximum Likelihood Random Effects Variance Component ------v
= .00501
se(v) = .00895
Variations of the previous analyses
MetaF MACRO (ANOVA)
based of Blind vs. Aware Test Administrators
METAF ES=d /W=w /group = BLDvAWR/MODEL=ML .
------- Analog ANOVA table (Homogeneity Q) ------Conclusions: Small, NS
Q
df
p
Between
.8381
1.0000
.3599 effect;
Within
26.8855
17.0000
.0598
Resid var marginally
Total
27.7236
18.0000
.0664
significant for “Aware”
------- Q by Group ------not “blind”
Group
Qw
df
p
1.0000
9.9550
8.0000
.2682
2.0000 16.9305
9.0000
.0498
------- Effect Size Results Total
------Mean ES
SE
-95%CI
+95%CI
Z
P
k
Total
.0785
.0480
-.0156
.1727
1.6344
.1022 19.0000
------- Effect Size Results by Group ------Group Mean ES
SE
-95%CI
+95%CI
Z
P
k
1.0000
.1314
.0752
-.0159
.2788
1.7487
.0803
9.0000
2.0000
.0420
.0625
-.0805
.1644
.6720
.5016 10.0000
------- Maximum Likelihood Random Effects Variance Component ------v
=
.01333
se(v) =
.01263
MetaF MACRO (ANOVA)
based of Group vs. Individual IQ Tests
METAF ES=d /W=w /group = GPvIN /MODEL=ML . Conclusions: Small,
marginally significant
------- Analog ANOVA table (Homogeneity Q) ------Q
df
p effect (4.1/34.3=12%
Between
4.1421
1.0000
.0418 var expl); ES NS for
Within
30.1044
17.0000
.0256 “Group” but signi for
Total
34.2464
18.0000
.0117 “individual”
------- Q by Group ------Resid var for group
Group
Qw
df
p
signif but individual NS;
1.0000 25.2512 15.0000
.0467
2.0000 4.8531 2.0000
.0883
------- Effect Size Results Total
------- All effects very small
Mean ES
SE -95%CI +95%CI
Z
P
k
Total
.0638
.0386 -.0118
.1393 1.6536
.0982 19.0000
------- Effect Size Results by Group ------Group Mean ES
SE -95%CI +95%CI
Z
P
k
1.0000
.0444
.0397 -.0335
.1222 1.1169
.2640 16.0000
2.0000
.3812
.1607
.0663
.6960 2.3726
.0177 3.0000
------- Maximum Likelihood Random Effects Variance Component ------v
= .00189
se(v) = .00745
MetaReg MACRO (Regression)
based of Group vs. Individual IQ Tests
METAREG ES=d /W=w/IVS=GPvIN /MODEL=ML .
Conclusions: Small,
marginally
significant effect
(12% var expl);
------- Descriptives ------Mean ES
R-Square
k
.0638
.1209
19.0000
Constant term
------- Homogeneity Analysis ------Q
df
p (Group) NS;
Model
4.1421
1.0000
.0418 Resid var signif (but
Residual
30.1044
17.0000
.0256 variance component
Total
34.2464
18.0000
.0117 NS)
------- Regression Coefficients ------B
SE -95% CI +95% CI
Z
P
Beta
Constant -.2924
.1792 -.6437
.0588 -1.6318
.1027
.0000
GPvIN
.3368
.1655
.0124
.6612 2.0352
.0418
.3478
------- Maximum Likelihood Random Effects Variance Component ------v
= .00189
se(v) = .00745
MetaReg MACRO (Regression)
based of (Group vs. Individual IQ Tests) & (categorised weeks)
Conclusions: Effect of
METAREG ES=d /W=w/IVS=GPvIN wkcat /MODEL=ML . test type no longer
------- Descriptives ------signif when weeks
Mean ES
R-Square
k
included.
.0603
.5770
19.0000
------- Homogeneity Analysis ------Effect of weeks nearly
Q
df
p
unaffected.
Model
20.6711
2.0000
.0000
Note can only look at
Residual
15.1543
16.0000
.5134
multiple variables
Total
35.8254
18.0000
.0074
with regression.
------- Regression Coefficients ------B
SE -95% CI +95% CI
Z
P
Beta
Constant
.1782
.2113 -.2360
.5925
.8434
.3990
.0000
GPvIN
.1983
.1667 -.1286
.5251 1.1890
.2344
.2032
wkcat
-.1481
.0367 -.2199 -.0762 -4.0399
.0001 -.6904
------- Maximum Likelihood Random Effects Variance Component ------v
= .00000
se(v) = .00648
Website Address to get MLwiN
Harvey Goldstein
developed the
MLwiN statistical
package used here
and has made
many contributions
to multilevel
modeling, including
meta-analysis.
Always a bit dangerous to say some one
person invented a new approach.
However, fair to say that Stephen
Raudenbush at least popularised the
multilevel approach to meta-analysis
with the meta-analysis of the teacherexpectancy data considered here.
Raudenbush, S.W. and Bryk, A.S. (2002).Hierarchical Linear Models (Second
Edition).Thousand Oaks: Sage Publications, 482 pp.
Raudenbush, S.W. (1984). Magnitude of teacher expectancy effects on Pupil IQ as
a function of the credibility of expectancy induction: A synthesis of findings from
18 experiments.Journal of Educational Psychology, 76, 1, 85-97.
Getting Data Into MLwiN
In an empty MLwiN file, puts the xx
input variables into first xx columns.
(can also add new data to existing
files). Check to see that data is
correct and click on “paste” button
Getting Data Into MLwiN
Check the MLwiN
“names” file to see
that data looks ok
(e.g., missing
values; min & max
values).
Setting Up Meta-analysis
MLwiN will open an
empty equation that
you have to construct.
Click on the “y” to bring up this screen.
select “d” (the effect size) as the dependent variable
Select “2” for “N of levels”
select “ID” for Level 1
select “d” for Level 2
Setting Up Meta-analysis
3
2
1.Click “Add Term” Button (bottom equations
window)
2.Select “cons” (variable = 1 for all cases)
3.Click the “done” button
1
Setting Up Meta-analysis
1
1.Click “Cons” in the
2
equation
2.Tick “Fixed Parameter” &
“j(id)” but not “i(d)”
3
3.Click the “done” button
Setting Up Meta-analysis
2
3
1
4
1.Now click “add term” button
2. This will bring up the “X-Variable” select SE
(the standard error computed earlier)
3.Tick only the “i(d)” box
4. Click “done”
Setting Up Meta-analysis
1
1
Now we want to constrain the variance at
level 1 to be fixed at 1.0. Under “model”
select “constrain parameters”; will bring
up “parameter constraint” window
Setting Up Meta-analysis
1
In the parameter constraint window:
1.Click the “random” button
2.Change “d: SE/SE” to 1
3.Change “to equal” to 1”
2&3
Setting Up Meta-analysis
3
1
2
1. “store” the constraints in the first empty column (“C19”)
2. Click the “attach random constraints” button.
3. Close the “Parameter Constraint” Window
“null” model with no predictors
->pred c50
->calc c51=(('d'-c50)/'se')**2
->sum c51 to b1 = 36.057
->cprob b1 18 = 0.0069377
Conclusion:
The mean effect
size (.078) is not
significant.
The chi-square is
significant; there is
study-to-study
variation.
Reasonable to
explore moderator
variables
After Closing the “parameter constraint” window (last slide)
Click on “start” button in “equation” window (may have to click
estimates button to get values). Compute chi-square value in
command interface window
Add raw “weeks” variable
>pred c50
->calc c51 = (('d'-c50)/'se')**2
->sum c51 to b1
28.937 (Compared to 35.825 for null model)
->cprob b1 17
0.035115
Conclusion:
The effect of weeks (-.013/.005) is significant
The mean effect size (.162/.055) signif (when weeks = 0).
chi-sq signif; some remaining study-to-study variation.
WKCAT: 4-category weeks
->pred c50; ->calc c51 = (('d'-c50)/'se')**2;->sum c51 to b1 = 16.568
->cprob b1 17 = 0.48398
Conclusion: categorized weeks does best of of (chi-sq = 16.568)
Aware vs. Blind Administration
For a categorical variable, you
choose a reference (“left out”
category. Default is the 1st category
>pred c50; ->calc c51 = (('d' - c50)/'se')**2; ->sum c51 b1 = 35.608
->cprob b1 17 = 0.0051714
Conclusion: Main Effect of Aware vs. Blind is NS
Aware vs. Blind Administration
For a categorical variable, you
choose a reference (“left out”
category. Default is the 1st category
>pred c50; ->calc c51 = (('d' c50)/'se')**2; ->sum c51 b1 = 16.445
->cprob b1 17 = 0.49254
Conclusion: Effect for Blind vs. Aware reduced by controlling for
“wkcat” but was already nonsignificant
Individual vs. Group Tests
>pred c50; ->calc c51 = (('d'-c50)/'se')**2;->sum c51 to b1 = 31.478
->cprob b1 16 = 0.011685
->pred c50; ->calc c51 = (('d' - c50)/'se')**2; ->sum c51 b1 = 15.154
->cprob b1 16 = 0.51337
Conclusion: Effect seems larger for individually administered
tests, but not after control for weeks (wkcat)
Individual vs. Group Tests
Order = 1 to specify a
2-way interaction term
Variables in interaction
->pred c50; ->calc c51 = (('d' - c50)/'se')**2; ->sum c51 b1 = 15.114
Conclusion: No interaction effect (chi-sq little different than
wkcat alone (15.114 vs. 16.568). Notice that the effect of test
type (and its SE) are very large (.2901/.4859=.597)
Individual vs. Group Tests
Grand Mean Centered
Grand Mean Centered
->pred c50; ->calc c51 = (('d' - c50)/'se')**2; ->sum c51 b1 = 15.114
Conclusion: Same results but estimated & SE for individual term
is smaller (reduced multicollinearity by grand mean centering
the wkcat variable).
Note that chi-sq is the same.
“weeks” centered at 2
Weeks is
centered at 2.
>pred c50;->calc c51 = (('d'-c50)/'se')**2; ->sum c51 to b1 = 28.937 ; ->cprob b1 17 = 0.035115
Conclusion:
The effect of weeks (-.013/.005) & chi-sq (28.937) same as
with original weeks.
The mean effect size (.136/.049) signif (when weeks = 2).
“weeks” centered at 6 & 7
Weeks centered at 6
Weeks centered at 7
>pred c50;->calc c51 = (('d'-c50)/'se')**2; ->sum c51 to b1 = 28.937 ; ->cprob b1 17 = 0.035115
Conclusion:
Constant term (intercept weeks = 6) is signif (.083/.042=1.98)
Constant term (intercept weeks = 7) is NS (.070/.042=1.68)
“weeks” polynomial = 2
>pred c50;->calc c51 = (('d'-c50)/'se')**2; ->sum c51 to b1 = 26.237; ->cprob b1 16 = 0.050779
Conclusion:
The linear term is significant but the quad term is not.
“weeks” polynomial = 3
->calc c51 = (('d' - c50)/'se')**2; ->sum c51 b1 = 18.543; ->cprob b1 15 = 0.23521
Conclusion:
All three polynomial terms are significant and the residual
variance component is substantially reduced.
“Log-e weeks+1”
->pred c50; ->calc c51 = (('d' - c50)/'se')**2; ->sum c51 b1 = 24.636
->cprob b1 17 = 0.10317
Conclusion: The linear term based on the log transform explains
more variance than the original (untransformed) weeks (chi-sq
= 24.636 vs. 28.937).
WKCAT: Centered at 2 & 3+
Conclusion: Intercept at wkcat = 2 is significant, but intercept
at wkcat = 3 is not
Graphs: Caterpillar Plots
1
Caterpillar plot based on L1 residuals.
Go to the “model” menu and select “residuals” option. This will bring up
the “settings” window. Set “SD (comparative)” to 1.96; 3. Set “level” to
“1d”; 4. click the “Calc” button; 5. click on the “plot” button to bring up
the next window. In the “plot” window select “residual +/- 1.96SD x
rank. This brings up the original graph. Clicking on the graph bring up
a window to modify the graph (a bit)
 The mean effect size associated with the intervention
was not significant. However, the results did not
generalise across studies (there was study-to-study
variation).
 Consistent with a priori predictions, the effect size was
significantly moderated by the amount of time students
had been in contact with teachers. If students and
teachers knew each other 0 or 1 week prior to the
intervention, there was a significant expectancy effect. If
they knew each other 2 or 3+ weeks the effect was not
significant (although the precise cutoff might
dependend on the scaling of weeks).
 Effects of test type (individual or group) and test
administrator awareness (“blind” or aware) were not
significant and did not interact with length.
Purpose-built
 Comprehensive Meta-analysis (commercial)
 Schwarzer (free, http://userpage.fuberlin.de/~health/meta_e.htm)
Extensions to standard statistics packages
 SPSS, Stata and SAS macros, downloadable from




http://mason.gmu.edu/~dwilsonb/ma.html
Stata add-ons, downloadable from
http://www.stata.com/support/faqs/stat/meta.html
HLM – V-known routine
MLwiN
MPlus
 Cooper, H., & Hedges, L. V. (Eds.) (1994). The
handbook of research synthesis (pp. 521–529).
New York: Russell Sage Foundation.
 Hox, J. (2003). Applied multilevel analysis.
Amsterdam: TT Publishers.
 Hunter, J. E., & Schmidt, F. L. (1990). Methods of
meta-analysis: Correcting error and bias in
research findings. Newbury Park: Sage
Publications.
 Lipsey, M. W., & Wilson, D. B. (2001). Practical
meta-analysis. Thousand Oaks, CA: Sage
Publications.
 Raudenbush, S.W. (1984). Magnitude of teacher
expectancy effects on Pupil IQ as a function of
the credibility of expectancy induction: A
synthesis of findings from 18 experiments.
Journal of Educational Psychology, 76, 85-97.
 Raudenbush, S.W. and Bryk, A.S.
(2002).
nd
Hierarchical Linear Models (2 Ed.).Thousand
Oaks: Sage Publications.
 Download macros for free from
http://mason.gmu.edu/~dwilsonb/ma.html
 Download MLwiN for free from
http://www.cmm.bristol.ac.uk/MLwiN/index.shtml