Meta-Analysis Graphs
Flow Diagram
These authors used PRISMA as
a guide for their chart, but they
are medical. Whatever you do
should communicate to your
audience so that the study can
be replicated.
Bambra CL, Hillier FC, Cairns J-M, Kasim A, Moore HJ, Summerbell CD. How effective are
interventions at reducing socioeconomic inequalities in obesity among children and adults?
Two systematic reviews. Public Health Res, 2015;3(1).
Funnel Plots
Shows ES by precision. Expect a funnel shape if fixed-effects
sampling distribution. Asymmetry suggests pub bias; excess
variance suggests heterogeneity (moderators). This one
shows pretty good symmetry but lots of variability (mortality
rates by hospital).
Funnel Plot
This one suggests publication bias (small studies with large ES).
I think this graph was created using Comprehensive Meta-Analysis (CMA), which is standalone software).
Funnel Plot
This one shows imputed data: ‘trim and fill’ on the basis of symmetry.
Funnel Plot
Epi_Meta_Plots
This one shows a masochist.
Input to R & Check Data
Compute Summary
0.279
0.419
0.558
Standard Error
0.140
0.000
Compute Funnel
-1.00
-0.50
0.00
0.50
Observed Outcome
1.00
1.50
This plot shows a lot of heterogeneity.
It also shows asymmetry consistent with
missing nonsignificant results (perhaps
publication bias). But the mean is too
high to worry about publication bias as
the sole explanation.
LMX &
Affective
Commitment
0.081
0.122
I added the
brown line to
mark the
approximate
place of a
nonsignificant
correlation
0.162
Standard Error
0.041
0.000
R Funnel Plot
0.20
0.40
0.60
Observed Outcome
0.80
1.00
Class Exercise
Open R, metafor
Download GenderMath.xlsx from website and import
Difference between boys and girls in math achievement
(positive d means boys’ mean is higher)
Multiple countries and multiple moderators
Compute a meta-analysis overall effect size (d1 is
standardized mean difference, v is variance of SMD)
Create a funnel plot
Forest Plots
General Types
General - unsorted
Sorted
Moderators - subgroups
Cumumlative
Forest Plots 1 – General
Summary
Note box, wings,
and diamond.
This plot was made with SAS. You can do this with enough patience; I use various programs.
Forest Plot 2 – Sort by ES
I think this graph
was created using
the program
metafor in R.
Steady progression?
Missing middle? Expect
some curl at the ends
for small samples.
Source:
http://stats.stackexchange.com/questions/107557/forest-plotfor-meta-analysis-displaying-the-mean-es-with-and-withoutoutliers
Forest Plot 3 – Subgroups
Subgroup means and CIs
Shiri (2014) Obesity & Sciatica
I think this graph was created using the program
meta in R.
Forest Plot 4 - Cumulative
Pineles (2014).
Miscarriage and
exposure to tobacco
smoke.
Overall OR and CI
plotted by adding
each study over
time.
Shows that
contrary to some
authors, there has
always been good
evidence of risk.
Note on sociology
of science; often
largest ES first;
decline over time.
Forest Plots in R
This is the
default. There
are too many
studies for this
to look good. I
would start by
sorting by the
moderator and
then by the
effect size.
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
0.37 [ 0.25 , 0.48 ]
0.37 [ 0.23 , 0.50 ]
0.50 [ 0.36 , 0.64 ]
0.47 [ 0.39 , 0.55 ]
0.34 [ 0.10 , 0.58 ]
0.45 [ 0.22 , 0.67 ]
0.45 [ 0.39 , 0.51 ]
0.54 [ 0.44 , 0.63 ]
0.47 [ 0.17 , 0.77 ]
0.55 [ 0.39 , 0.71 ]
0.39 [ 0.27 , 0.51 ]
0.47 [ 0.38 , 0.57 ]
0.62 [ 0.52 , 0.72 ]
0.62 [ 0.42 , 0.82 ]
0.59 [ 0.45 , 0.73 ]
0.34 [ 0.22 , 0.47 ]
0.41 [ 0.18 , 0.65 ]
0.34 [ 0.25 , 0.44 ]
0.59 [ 0.45 , 0.74 ]
0.23 [ 0.08 , 0.39 ]
0.48 [ 0.35 , 0.62 ]
0.46 [ 0.26 , 0.65 ]
0.59 [ 0.49 , 0.69 ]
0.83 [ 0.69 , 0.97 ]
0.71 [ 0.60 , 0.82 ]
0.24 [ 0.12 , 0.37 ]
0.59 [ 0.42 , 0.76 ]
0.37 [ 0.21 , 0.52 ]
0.41 [ 0.28 , 0.54 ]
0.60 [ 0.40 , 0.81 ]
0.55 [ 0.36 , 0.74 ]
0.44 [ 0.29 , 0.58 ]
0.52 [ 0.39 , 0.66 ]
0.30 [ 0.07 , 0.53 ]
0.32 [ 0.15 , 0.49 ]
0.30 [ 0.16 , 0.44 ]
0.22 [ 0.09 , 0.35 ]
0.63 [ 0.51 , 0.76 ]
0.38 [ 0.27 , 0.48 ]
0.20 [ 0.11 , 0.30 ]
0.52 [ 0.40 , 0.65 ]
0.34 [ 0.23 , 0.46 ]
0.45 [ 0.36 , 0.53 ]
0.39 [ 0.31 , 0.47 ]
0.50 [ 0.36 , 0.63 ]
0.54 [ 0.38 , 0.69 ]
0.52 [ 0.39 , 0.66 ]
0.37 [ 0.26 , 0.47 ]
0.54 [ 0.40 , 0.67 ]
0.51 [ 0.39 , 0.63 ]
0.50 [ 0.39 , 0.61 ]
0.44 [ 0.24 , 0.63 ]
0.97 [ 0.81 , 1.13 ]
0.46 [ 0.34 , 0.58 ]
0.74 [ 0.62 , 0.87 ]
0.47 [ 0.31 , 0.64 ]
0.38 [ 0.18 , 0.57 ]
0.41 [ 0.24 , 0.58 ]
0.37 [ 0.27 , 0.46 ]
0.42 [ 0.26 , 0.59 ]
0.40 [ 0.08 , 0.72 ]
0.31 [ 0.08 , 0.54 ]
0.37 [ 0.11 , 0.62 ]
0.34 [ 0.21 , 0.47 ]
0.42 [ 0.26 , 0.58 ]
0.44 [ 0.30 , 0.57 ]
0.38 [ 0.25 , 0.50 ]
0.46 [ 0.32 , 0.60 ]
0.41 [ 0.27 , 0.56 ]
0.62 [ 0.48 , 0.76 ]
0.21 [ 0.10 , 0.33 ]
0.56 [ 0.43 , 0.70 ]
0.20 [ 0.08 , 0.32 ]
0.30 [ 0.15 , 0.44 ]
0.63 [ 0.53 , 0.74 ]
0.42 [ 0.31 , 0.54 ]
0.20 [ 0.08 , 0.33 ]
0.59 [ 0.51 , 0.67 ]
0.58 [ 0.49 , 0.66 ]
0.58 [ 0.48 , 0.67 ]
0.71 [ 0.57 , 0.85 ]
0.48 [ 0.37 , 0.60 ]
0.74 [ 0.58 , 0.90 ]
0.65 [ 0.52 , 0.78 ]
0.24 [ 0.14 , 0.35 ]
0.47 [ 0.34 , 0.61 ]
0.46 [ 0.38 , 0.54 ]
0.52 [ 0.37 , 0.67 ]
0.44 [ 0.30 , 0.57 ]
0.66 [ 0.46 , 0.86 ]
0.54 [ 0.46 , 0.61 ]
0.29 [ 0.13 , 0.44 ]
RE Model
0.46 [ 0.44 , 0.49 ]
0.00
0.20
0.40
0.60
Observed Outcome
0.80
1.00
1.20
Sorting in R
You can always sort in Excel or some other program if
you’d rather
The sort command you want in R is ‘order’
Let’s look at an example:
The Devine data shows standardized mean differences
for something… and it’s coded by journal article or
dissertation.
Attach the data
You have to ATTACH the data (object) and have this accepted
without error before you can sort.
Sort the Data by ES
Rerun the Analysis
Compute Sorted Forest
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
-0.23 [ -1.11 , 0.65 ]
-0.21 [ -0.83 , 0.41 ]
-0.20 [ -0.85 , 0.45 ]
-0.15 [ -1.03 , 0.73 ]
-0.08 [ -0.73 , 0.57 ]
-0.02 [ -0.31 , 0.27 ]
0.04 [ -0.42 , 0.50 ]
0.08 [ -0.80 , 0.96 ]
0.12 [ -0.32 , 0.56 ]
0.16 [ -0.68 , 1.00 ]
0.19 [ -0.55 , 0.93 ]
0.20 [ -0.39 , 0.79 ]
0.21 [ -0.24 , 0.66 ]
0.25 [ -0.02 , 0.52 ]
0.27 [ -0.66 , 1.20 ]
0.28 [ -0.23 , 0.79 ]
0.29 [ -0.22 , 0.80 ]
0.30 [ -0.34 , 0.94 ]
0.32 [ -0.61 , 1.25 ]
0.34 [ -0.12 , 0.80 ]
0.35 [ -0.31 , 1.01 ]
0.37 [ -0.29 , 1.03 ]
0.39 [ -0.12 , 0.90 ]
0.40 [ -0.38 , 1.18 ]
0.42 [ -0.26 , 1.10 ]
0.42 [ 0.20 , 0.64 ]
0.46 [ -0.29 , 1.21 ]
0.46 [ -0.22 , 1.14 ]
0.48 [ -0.08 , 1.04 ]
0.52 [ 0.01 , 1.03 ]
0.52 [ 0.01 , 1.03 ]
0.54 [ 0.14 , 0.94 ]
0.56 [ -0.51 , 1.63 ]
0.57 [ 0.12 , 1.02 ]
0.58 [ -0.09 , 1.25 ]
0.59 [ -0.41 , 1.59 ]
0.59 [ -0.30 , 1.48 ]
0.62 [ -0.14 , 1.38 ]
0.63 [ 0.00 , 1.26 ]
0.69 [ -0.13 , 1.51 ]
0.70 [ -0.12 , 1.52 ]
0.74 [ 0.05 , 1.43 ]
0.77 [ 0.43 , 1.11 ]
0.78 [ 0.01 , 1.55 ]
0.80 [ 0.12 , 1.48 ]
0.88 [ -0.21 , 1.97 ]
1.00 [ 0.19 , 1.81 ]
1.08 [ 0.09 , 2.07 ]
1.10 [ 0.05 , 2.15 ]
1.26 [ 0.19 , 2.33 ]
1.27 [ 0.43 , 2.11 ]
1.36 [ 0.39 , 2.33 ]
1.36 [ 0.34 , 2.38 ]
1.38 [ 0.45 , 2.31 ]
RE Model
0.41 [ 0.32 , 0.50 ]
-2.00
-1.00
0.00
1.00
Observed Outcome
2.00
3.00
Sort by Publication and ES
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
-0.23 [ -1.11 , 0.65 ]
-0.21 [ -0.83 , 0.41 ]
-0.20 [ -0.85 , 0.45 ]
-0.15 [ -1.03 , 0.73 ]
-0.08 [ -0.73 , 0.57 ]
0.08 [ -0.80 , 0.96 ]
0.12 [ -0.32 , 0.56 ]
0.21 [ -0.24 , 0.66 ]
0.27 [ -0.66 , 1.20 ]
0.29 [ -0.22 , 0.80 ]
0.34 [ -0.12 , 0.80 ]
0.39 [ -0.12 , 0.90 ]
0.40 [ -0.38 , 1.18 ]
0.52 [ 0.01 , 1.03 ]
0.57 [ 0.12 , 1.02 ]
0.58 [ -0.09 , 1.25 ]
0.74 [ 0.05 , 1.43 ]
0.77 [ 0.43 , 1.11 ]
0.80 [ 0.12 , 1.48 ]
1.08 [ 0.09 , 2.07 ]
1.10 [ 0.05 , 2.15 ]
-0.02 [ -0.31 , 0.27 ]
0.04 [ -0.42 , 0.50 ]
0.16 [ -0.68 , 1.00 ]
0.19 [ -0.55 , 0.93 ]
0.20 [ -0.39 , 0.79 ]
0.25 [ -0.02 , 0.52 ]
0.28 [ -0.23 , 0.79 ]
0.30 [ -0.34 , 0.94 ]
0.32 [ -0.61 , 1.25 ]
0.35 [ -0.31 , 1.01 ]
0.37 [ -0.29 , 1.03 ]
0.42 [ -0.26 , 1.10 ]
0.42 [ 0.20 , 0.64 ]
0.46 [ -0.29 , 1.21 ]
0.46 [ -0.22 , 1.14 ]
0.48 [ -0.08 , 1.04 ]
0.52 [ 0.01 , 1.03 ]
0.54 [ 0.14 , 0.94 ]
0.56 [ -0.51 , 1.63 ]
0.59 [ -0.41 , 1.59 ]
0.59 [ -0.30 , 1.48 ]
0.62 [ -0.14 , 1.38 ]
0.63 [ 0.00 , 1.26 ]
0.69 [ -0.13 , 1.51 ]
0.70 [ -0.12 , 1.52 ]
0.78 [ 0.01 , 1.55 ]
0.88 [ -0.21 , 1.97 ]
1.00 [ 0.19 , 1.81 ]
1.26 [ 0.19 , 2.33 ]
1.27 [ 0.43 , 2.11 ]
1.36 [ 0.39 , 2.33 ]
1.36 [ 0.34 , 2.38 ]
1.38 [ 0.45 , 2.31 ]
RE Model
0.41 [ 0.32 , 0.50 ]
Categorical Moderator
Input File
z
0.42
0.45
0.48
0.69
0.62
0.78
w
197
172
247
247
197
223
V
Sex
0.005076142
0.005813953
0.004048583
0.004048583
0.005076142
0.004484305
1
1
1
2
2
2
SAT & GPA
Overall model, no moderator
Study 1
0.42 [ 0.28 , 0.56 ]
Study 2
0.45 [ 0.30 , 0.60 ]
Study 3
0.48 [ 0.36 , 0.60 ]
Study 4
0.69 [ 0.57 , 0.81 ]
Study 5
0.62 [ 0.48 , 0.76 ]
Study 6
0.78 [ 0.65 , 0.91 ]
RE Model
0.58 [ 0.46 , 0.69 ]
0.20
0.40
0.60
0.80
Observed Outcome
1.00
Add Moderator
Study 1
0.42 [ 0.28 , 0.56 ]
Study 2
0.45 [ 0.30 , 0.60 ]
Study 3
0.48 [ 0.36 , 0.60 ]
Study 4
0.69 [ 0.57 , 0.81 ]
Study 5
0.62 [ 0.48 , 0.76 ]
Study 6
0.78 [ 0.65 , 0.91 ]
0.20
0.40
0.60
0.80
Observed Outcome
1.00
A bit hard to see in
this graph, but the
grey diamonds align
vertically. With more
ES sorted by ES value,
the grey diamonds do
a good job of showing
the overall category
mean.
The program represents the group
means by a grey diamond that is
superimposed on each study.
Add Other Stuff
You can add lots of information into the graph to
enhance the graph’s ability to communicate to your
audience.
Labels
Vertical lines
Diamonds
Change Axes – limits or scale
See the file “forest_plot_modification.R”
Continuous Moderator
Input Data
z
w
v
PctQ
0.42
0.45
0.48
0.69
0.62
0.78
197
172
247
247
197
223
0.005076142 0.1
0.005813953 0.15
0.004048583 0.12
0.004048583 0.2
0.005076142 0.25
0.004484305 0.3
Forest with Continuous
Prediction
This is probably not
the best way to show
the results. A metaanalytic scatterplot
(meta-regression)
with best fit line
would be better. But
it does give a sense of
the data.
Study 1
0.42 [ 0.28 , 0.56 ]
Study 2
0.45 [ 0.30 , 0.60 ]
Study 3
0.48 [ 0.36 , 0.60 ]
Study 4
0.69 [ 0.57 , 0.81 ]
Study 5
0.62 [ 0.48 , 0.76 ]
Study 6
0.78 [ 0.65 , 0.91 ]
0.20
0.40
0.60
0.80
Observed Outcome
1.00
Heterogeneity
Effect sizes often vary far more than what would be expected
by sampling error
Report REVC and CI
Some of this variability can be accounted for by moderators
explicitly incorporated into the model
Report reduction and remaining heterogeneity
The remaining heterogeneity is a problem.
Meaning of the average
Unanswered question of source
Representing the heterogeneous effects
Prediction Interval
Interval contains both sampling variance and randomeffects variance
Intended to contain a percentage (usually 95) of the
infinite-sample effect sizes
If REVC is large, the prediction interval is much larger
than the confidence interval
Sitzmann - CI vs Web - Declarative
Meta-analysis of 71
Samples (studies)
comparing classroom
instruction versus online
instruction on tests of
declarative knowledge
(college classes and
corporate training).
Diamond is the
Confidence Interval, the
red line at bottom is the
Prediction Interval; it
shows expected outcomes.
Average difference near
zero, but SUBSTANTIAL
variability. Begs us to ask
why the study outcomes are
so different.
Favors
classroom
Favors
web
Red is random effects.
Orange is fixed effects.
-3
-2.5
-2
-1.5
-1
-0.5
0
Standardized Mean Difference
0.5
1
1.5
2
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
Study
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
Sitzmann data
Sorted by ES
Classroom
Web
RE Model
-1.31 [ -2.47 , -0.16 ]
-0.97 [ -1.53 , -0.42 ]
-0.85 [ -1.65 , -0.05 ]
-0.84 [ -1.60 , -0.08 ]
-0.68 [ -1.26 , -0.09 ]
-0.60 [ -1.38 , 0.19 ]
-0.55 [ -1.02 , -0.09 ]
-0.52 [ -0.95 , -0.09 ]
-0.48 [ -1.53 , 0.57 ]
-0.47 [ -0.86 , -0.08 ]
-0.46 [ -1.66 , 0.75 ]
-0.43 [ -1.09 , 0.22 ]
-0.43 [ -1.08 , 0.21 ]
-0.40 [ -1.00 , 0.21 ]
-0.38 [ -0.91 , 0.14 ]
-0.31 [ -0.71 , 0.08 ]
-0.30 [ -0.93 , 0.34 ]
-0.28 [ -0.96 , 0.39 ]
-0.27 [ -0.98 , 0.44 ]
-0.23 [ -0.62 , 0.16 ]
-0.22 [ -0.77 , 0.33 ]
-0.19 [ -0.63 , 0.24 ]
-0.14 [ -0.51 , 0.24 ]
-0.12 [ -0.76 , 0.52 ]
-0.08 [ -0.50 , 0.34 ]
-0.07 [ -0.65 , 0.52 ]
-0.06 [ -0.74 , 0.63 ]
-0.03 [ -0.41 , 0.34 ]
-0.03 [ -0.58 , 0.52 ]
-0.03 [ -0.56 , 0.50 ]
-0.02 [ -0.41 , 0.36 ]
-0.01 [ -0.51 , 0.49 ]
0.00 [ -0.67 , 0.67 ]
0.00 [ -0.55 , 0.55 ]
0.00 [ -0.70 , 0.70 ]
0.00 [ -0.43 , 0.44 ]
0.01 [ -0.53 , 0.54 ]
0.01 [ -0.39 , 0.41 ]
0.02 [ -0.13 , 0.17 ]
0.02 [ -0.49 , 0.53 ]
0.04 [ -0.49 , 0.57 ]
0.04 [ -0.21 , 0.29 ]
0.10 [ -0.27 , 0.47 ]
0.12 [ -0.25 , 0.50 ]
0.13 [ -0.46 , 0.73 ]
0.14 [ -0.14 , 0.41 ]
0.15 [ 0.08 , 0.23 ]
0.18 [ -0.34 , 0.71 ]
0.26 [ -0.70 , 1.21 ]
0.26 [ -0.03 , 0.55 ]
0.26 [ -0.04 , 0.56 ]
0.27 [ -0.05 , 0.60 ]
0.27 [ -0.27 , 0.82 ]
0.28 [ -0.12 , 0.68 ]
0.34 [ -0.60 , 1.28 ]
0.37 [ 0.09 , 0.66 ]
0.47 [ 0.21 , 0.74 ]
0.55 [ -0.20 , 1.30 ]
0.55 [ 0.03 , 1.08 ]
0.57 [ -0.31 , 1.45 ]
0.59 [ 0.26 , 0.93 ]
0.60 [ -0.28 , 1.48 ]
0.65 [ 0.34 , 0.95 ]
0.68 [ 0.41 , 0.94 ]
0.69 [ 0.10 , 1.27 ]
0.71 [ 0.01 , 1.42 ]
0.77 [ 0.03 , 1.51 ]
0.81 [ 0.56 , 1.06 ]
1.06 [ 0.50 , 1.62 ]
1.27 [ 0.78 , 1.77 ]
1.30 [ 0.94 , 1.67 ]
0.08 [ -0.01 , 0.18 ]
-3.00
-2.00
-1.00
0.00
Observed Outcome
1.00
2.00
Sitzmann – Web vs CI - Procedural
Meta-analysis of 12 studies
of procedural knowledge;
college students and
corporate training. Note
that many studies are
significantly different from
zero, but on opposite sides.
Expected effects range
approximately from -1.25 to
1.25.
Favors
Classroom
Favors Web
This graph was produced
using the software program
MIX (stand-alone software).
-2
-1.5
-1
-0.5
0
0.5
Standardized Mean Difference
1
1.5
2
Cook – Web vs Other – Declarative
Meta-analysis of 63 interventions
comparing instruction through
the web vs. other (mostly
classroom) instruction on tests of
declarative knowledge in health
professionals.
Favors
Alternate
Favors
Web
Note that again there is a slight
overall mean difference favoring
the internet, but even larger
variability (see the horizontal red
line). The expected true
differences range from about .7
in favor of the classroom to
about 1.0 in favor of the internet.
-2
-1.5
-1
-0.5
0
0.5
Standardized Mean Difference
1
1.5
2
Cook – Web vs. Alternate - Procedural
Results for 12 studies of health
professionals for procedural
knowledge. Note same pattern
of results; enormous
variability.
Favors Alternate
Favors Web
Most studies show results
significantly different from
zero, but they fall on both
sides of the argument.
Cook, Levinson, Garside, Dupras, Erwin, & Montori
(2008). Internet-based learning in the health
professions. Journal of the American Medical Association,
300, 1181-1196.
-2.5
-2
-1.5
-1
-0.5
0
0.5
Stamdardized Mean Difference
1
1.5
2
Shrunken Estimates
Bayesian estimates of local populations (estimates of the
location of infinite-sample effect sizes) aka Empirical Bayes
Estimates
Moves observed effect sizes toward the mean in proportion
to uncertainty
Smaller movement if study sample sizes are large
Smaller movement if tau-squared is large
Moves studies to the mean if sample sizes are small or if tausquared approaches zero
This kind of analysis is more difficult to do correctly; I do
not expect you to do it for this class, but want you to know
that it is available and might be useful.
EB Estimates: Conceptual Diagram
dˆEBi
d
This slide copied from a talk by David Rindskopf
di
Forest Plot – 2 Methods Stacked for Comparison
Bayes
Why is this study different?
Rodriguez
Bayesian Estimate
Rodriguez & Maeda estimate
This graph
was
produced
using MS
Excel and
PowerPoint.
Prediction Interval
Mean and 95 % CI
0.5
0.6
0.7
0.8
0.9
1
Shrunken Estimates in R
The metafor package uses the function BLUP( ) to
produce ‘shrunken’ or empirical Bayes estimates for
individual studies.
Class Exercise
• Recall GenderMath
Create a forest plot (unsorted data)
Sort the data by G1 then d1
Compute a model with G1 as a moderator
Create a forest plot with data ordered by G1 and d1