advertisement

Practical Meta-Analysis for the Social Sciences Evan J. Ringquist School of Public and Environmental Affairs Indiana University Workshop in Methods Presentation January 11, 2013 Bloomington, IN 47405 1 “Meta-Analysis is not a fad. It is rooted in the fundamental values of the scientific enterprise: replicability, quantification, causal and correlational analysis. Valuable information is needlessly scattered in individual studies. The ability of social scientists to deliver generalizable answers to basic questions of policy is too serious a concern to allow us to treat research integration lightly. The potential benefits of meta-analysis method seem enormous.” (Bangert-Drowns 1986: 398) 2 Meta-Analysis Defined • Meta-analysis is a systematic, quantitative, replicable process of synthesizing numerous and sometimes conflicting results from a body of original studies. • Meta-analysis, then, provides a powerful set of tools for aggregating knowledge. 3 Number of Articles Published Each Year in the “Meta-Analysis” Topic Category as Referenced by the Social Sciences Citation Index 1400 1200 800 600 400 200 2011 2010 2009 2008 2007 2006 2005 2004 2003 2002 2001 2000 1999 1998 1997 1996 1995 1994 1993 1992 1991 1990 1989 1988 1987 1986 1985 1984 1983 1982 1981 0 1980 articles per year 1000 4 Outline of Presentation 1. Motivation for Using Meta-Analysis in the Social Sciences 2. Quick Introduction to Meta-Analysis 3. Meta-Regression Models: The Basics 4. Meta-Regression Analysis for the Social Sciences – CRVE, GEE, and if we have time, HLM 5 Section I: Motivation 6 Cumulative Knowledge and the Scientific Enterprise • A Central Goal of Science is the Accumulation and Aggregation of Knowledge. Yet the Social Sciences in General and Researchers in Public Management and Policy in Particular Have Found Cumulative Knowledge to be an Elusive Goal. 7 “Cumulative Knowledge” in Environmental Justice • “Overwhelming evidence” of significant environmental inequities (Goldman 1993; see also Mohai and Bryant 1994) • “even a reasonably generous reading of the empirical research alleging environment inequity . . . must leave room for profound skepticism” (Foreman 1998; see also Bowen 2001) 8 “Cumulative Knowledge” in Education Policy • “Extensive research has been conducted on the academic success of students enrolled in school choice program nationwide. Rigorous studies show strong gains for voucher and scholarship tax credit recipients . . .” (Alliance for School Choice, 2010) • “The reason that vouchers had subsided as a point of advocacy is because they don’t work.” (Randi Weingarten, President, American Federation of Teachers; Associated Press 2011). • “Few contemporary questions in American education have produced such wide-spread controversy as that regarding the potential of school vouchers to reduce inequality in student outcomes. . . these efforts notwithstanding, answers to the voucher question still appear uncertain.” (Cowen 2008) 9 Cumulative Knowledge in Public Management • the more knowledge we create and diffuse, “the less we know, because our research is not aggregated or accumulated into substantial bodies of knowledge” (Van Slyke, O’Leary, and Kim 2010: 290. • The first step in making Public Administration a stronger and more robust field is the need to aggregate knowledge in the sense of making it cumulative. (Rosenbloom 2010) 10 Cumulative Knowledge in Policy Studies • “political scientists have spent (almost literally) countless articles and books proposing something like ‘laws’ or theories that, taken collectively . . . have produced infinitely more confusion than clarity” (DeLeon 1998: 150). 11 Cumulative Knowledge in Policy Evaluation • Social scientists identify new and important questions having policy relevance. Initial studies examining these questions provide clear answers regarding, for example, the effectiveness of social policy interventions. Subsequent studies cast doubt on these initial conclusions, however, and with the proliferation of studies comes a proliferation of conclusions. The best efforts of social and behavioral researchers generate confusion and uncertainty rather than clarity. In the end, researchers conclude that the phenomena being studied are ‘hopelessly complex’ and move on to other questions. After several repetitions of this cycle, social and behavioral scientists themselves become cynical about their own work and express doubts about whether behavioral and social science in general is capable of generating cumulative knowledge or answers to socially important questions (Hunter and Schmidt 1996: 325-6). 12 Why Do Social Scientists Struggle with Knowledge Accumulation? 1. Epistemological Differences: Social Sciences ought not aim to generate generalizable “scientific” knowledge ala Chemistry. 2. Subject Matter Differences: Social scientists study organizations and institutions devised and populated by strategic actors, not molecules or organizms without agency. 3. Social Scientists Use The Wrong Tools for Aggregating Knowledge. 13 Section II: Introduction to Meta-Analysis 14 Meta-Analysis Defined (again) • Meta-analysis is a systematic, quantitative, replicable process of synthesizing numerous and sometimes conflicting results from a body of original studies. • Meta-analysis, then, provides a powerful set of tools for aggregating knowledge. 15 The Language of Meta-Analysis • Original Study: A piece of original research, published or unpublished, that aims to test a hypothesis and/or estimate a quantity of interest. • Focal Predictor: The independent variable in an original study that measures the key exogenous concept associated with the research question of interest. 16 The Language of Meta-Analysis • Effect Size: A standardized measure of the relationship between the focal predictor and the dependent variable in an original study. Effect sizes are the unit of analysis and the quantity of interest in a metaanalysis, often designated Θi. Without effect sizes there can be no meta-analysis. 17 The Language of Meta-Analysis • Effect Size Variance: A measure of the uncertainty associated with a particular effect size. • Fixed and Random Effects Models: Methods of weighting effect sizes that embody different assumptions about the effect size variance. 18 How Do We Calculate Effect Sizes? • Three Families of Effect Sizes 1. D-based effect sizes (standardized mean differences, common in education and psychology) 2. Odds-based effect sizes (e.g. log odds of an event, common in medicine) 3. R-based effect sizes (partial correlation coefficients, unusual in the literature but 19 most useful for social scientists) Calculating R-based Effect Sizes from Original Studies • r = √[t2 / (t2 + df)] • r = √[Z2 / n] • r = √[Χ21 / n] [1] [2] [3] • V[r] = (1-r2)2 / (n-1) [4] 20 Fisher’s Corrections for R-based Effect Sizes and Variances • Zr = 0.5 ln[(1+r) / (1-r)] [5] • V[Zr] = 1/(n-3) [6] 21 Example: Are Pollution Emissions Higher in Black Neighborhoods? • Ringquist 1997: t=3.06, df=29202 – r=.02 – Zr=.02 – V[Zr] = .000034 • Downey 1998: t=3.38, df=112 – r=.30 – Zr=.31 – V[Zr] = .0072 22 What Can We Do With Effect Sizes Statistically? 1. Communicate results from original research in a more meaningful fashion (not meta-analysis). 2. Combine (average) effect sizes to estimate the population effect size 3. Test null hypothesis that population effect size equals zero 4. Explain variation in effect sizes across original studies. We use “Meta-Regression” to 23 account for this variation. Combining Effect Sizes • We calculate average effect sizes, or our estimate of the population effect size, by calculating a weighted average of all effect sizes from original studies where the weights are inverse variances • Θbar= ΣwiΘi / Σwi [7] • Wi = 1/vi 24 Combining Effect Sizes: Fixed Effects Models • Fixed Effects models assume that effect sizes vary across original studies only due to sampling error: – Θi = Θ + ei – E[Θi]= Θ – ei ~ N(0,vi) [8], so [9], and [10] 25 Combining Effect Sizes: Fixed Effects Models • Fixed effects meta-regression assumes that effect sizes conditional upon moderator variables differ only due to sample size • Fixed effects estimates apply only to sample of original studies in hand 26 Combining Effect Sizes: Random Effects Models • Random Effects models assume that effect sizes are normally distributed random variables – Θ ~ N(µΘ, τ2) [11] – Θi = µΘ + ei [12] – E[Θi] = µΘ [13], and – ei ~ N(0,vi + τ2) [14] – Always use random effects in the social sciences! 27 Estimating the Random Effects Variance Component • Using Restricted Maximum Likelihood – Le(τ2) = -.5 * Σ [ln(vi + τ2) + ((ei2 / (vi + τ2) - .5 * ln |X’v-1X| [15] • Using Method of Moments (MOM) – tr(M) = Σvi-1 – tr [(Σvi-1 XX’)-1 (Σvi-2 XX’)] [16] • Using MOM Approximation – τ2 = [SSEols / (m-k-1)] – vbar [17] 28 What Can We Do With Effect Sizes Substantively? • Estimate severity of problems – E.g., environmental inequities – E.g., effects of climate change • Measure important quantities of interest – E.g., statistical value of a life – E.g., hedonic pricing of environmental amenities 29 What Can We Do With Effect Sizes Substantively? • Program Evaluation – E.g., effectiveness of educational vouchers – E.g., effectiveness of job training programs • Theory Testing and Development – E.g., Top-down vs. bottom-up implementation – E.g., effects of negative campaign ads – E.g., Ricardian equivalence 30 Section III: Introduction to Meta-Regression 31 “exploration of the pattern of variation in effect sizes among studies is a far more important goal of meta-analysis than the construction of powerful tests of null hypotheses.” (Osenberg et al. 1999: 1105) 32 Moving Meta-Analysis to the Social Sciences • Techniques for calculating and combining effect sizes developed for synthesizing the results from experiments. Can these same techniques really be used to synthesize results from the non-experimental multivariate models common in the social sciences? 33 Three Critiques of MetaAnalysis in the Social Sciences 1. Parameter estimates from the general linear model are not comparable. 2. Effect sizes cannot be calculated from multivariate models. 3. Parameter estimates from different models estimate different population parameters, and therefore cannot be combined. 34 Response to Critique #1: • While regression parameters are not comparable, meta-analysis does not combine parameter estimates. Rather, meta-analysis combines effect sizes which are standardized with respect to scale. This is not a valid critique. 35 Response to Critique #2 • In fact, the same formulas that can be used to calculate r-based effect sizes from experimental studies can be used to calculate partial correlations from regression models, probit models, etc. (see Greene 1993: 180) 36 Response to Critique #2 • Example 1: Panel regression model, Y=continuous, K=33, N=307,538 – Pcorr = .0099, Zr = .0096 • Example 2: Probit model, Y=dichotomous, K=8, N=14131 – Pcorr = .1081, Zr = .1062 • Example 3: OLS regression model, Y=continuous, K=24, N=285 – Pcorr = .1619, Zr = .1565 37 Response to Critique #3 • E.g., Study 1 – Y = b0 + b1X1 + b2X2 + b3X3 + e • E.g., Study 2 – Y = b0 + b1X1 + b2X2 + b4X4 + e • E[b11] ≠ E[b12], β11 ≠ β12, and Θ1 ≠ Θ2 • Therefore dependent variable Θi measures fundamentally different quantities 38 Response to Critique #3 • Critique #3 is a valid critique, and this is why calculating average effect sizes is of little value in the social sciences – these average effect sizes combine estimates of different population parameters, and therefore are not valid estimates of any useful quantity of interest. • Thankfully, we can address Critique #3 39 using Meta-Regression. What Do We Need to Conduct a Meta-Regression? • Measure of effect size that is comparable across original studies employing different measures, models, and samples • Measure of (un)certainty or effect size variance • Moderator variables that account for variability in effect sizes 40 Introduction to Meta-Regression • Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi) – Θi is the effect size – X represents moderator variables accounting for differences in effect sizes within and across studies • Scientifically Interesting Moderators (Rubin 1992) – e.g., differences attributable to target characteristics – e.g., differences attributable to program or policy design • Scientifically Uninteresting Moderators – e.g., differences attributable to estimation technique or research design 41 Number of Publications Using Phrase “MetaRegression” in Google Scholar, 1990-2011 42 Example: Synthesizing Research on Educational Vouchers • Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi) – Θi is the estimated point biserial correlation (effect size Zr) between the use of an educational voucher and student standardized test scores from a particular statistical model in a particular original study. 43 Example: Educational Vouchers • Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi) – X1 moderator variable identifying effect sizes from models limiting sample to black children – X2 moderator variable identifying effect sizes from voucher programs available to religious schools – X3 moderator variable identifying effect sizes from models not using random assignment to treatment and control groups. 44 Example: Educational Vouchers • Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi) – b0 average effect of vouchers on student test scores for all students in experimental studies where vouchers use is limited to secular schools – b1 differential effect of vouchers for black students – b2 differential effect from voucher programs that include religious schools – b3 differential voucher effect from quasi-experimental studies 45 Independent Variables Black Student Hispanic Student Religious Schools Matched Control Groups Convenience Control Groups Pretest Included Intercept (Baseline Effect Size) Adjusted R2a Psuedo-R2a I2a Sample Size CRVE Parameter Estimates (St.Err.) 0.0296** (0.0130) -0.0470*** (0.0103) -0.0312 (0.0187) -0.0097 (0.0064) -0.0073 (0.0085) -0.0326*** (0.0115) 0.1097*** (0.0257) .44 CRVE 95% CI 0.0030 0.0562 -0.0680 -0.0259 -0.0693 0.0069 -0.0226 0.0033 -0.0246 0.0101 -0.0560 -0.0092 0.0573 0.1620 GEE Parameter Estimates (St.Err.) 0.0370*** (0.0116) -0.0413*** (0.0084) 0.0180 (0.0201) -0.0069* (0.0040) -0.0082 (0.0082) -0.0228* (0.0120) 0.0830*** (0.0299) na .68 na 25.01% na 611 611 GEE 95% CI 0.0143 0.0598 -0.0577 -0.0248 -0.0574 0.0213 -0.0148 0.0009 -0.0243 0.0078 -0.0464 0.0008 0.0243 0.1416 46 Obtaining Estimates for Meta-Regression Models • OLS Estimator of Meta-Regression Model – E[b] = β = (X’X)-1 X’Θ – var[b] = σ2(X’X)-1 [18] [19] • But OLS is incorrect for two reasons: 1. Θi is heteroskedastic (recall assumptions) 2. OLS weights all observations equally (unequal variances means unequal certainty) 47 Generalized Least Squares in Meta-Regression • In OLS, ee’ = σ2I • In GLS, ee’ = σ2Ω – where Omega is a diagonal matrix with proportional error variances on main diagonal • GLS estimates then are: – E[bgls] = βgls = (X’Ω-1X)-1 X’Ω-1Θ – var[bgls] = σ2 (X’Ω-1X)-1 [20] [21] 48 Weighted Least Squares in Meta-Regression • Find a weight matrix W so that W’W = Ω-1 • Then the WLS meta-regression estimates become: – E[bwls] = βwls = (X’W’WX)-1 X’W’WΘ – var[bwls] = σ2 (X’W’WX)-1 [22] [23] 49 Weighted Least Squares in Meta-Regression (cont.) • The good news is that we have a handy estimate of the diagonal elements of the weight matrix W: the effect size variances (or more properly, their square roots) • The bad news is that fixed effects variance estimates are almost certainly wrong, so we need to use random effect variances. 50 RE Variance Component τ2 • Cannot include moderator variables for all independent variables or all design elements in original studies • Random Effects variance component τ2 can be conceived of as reflecting the impact of unobserved variables on Θi 51 Weights in Random Effects Meta-Regression (v1 + τ2) 0 0 0 0 (v2 + τ2) 0 0 ee’remr = σ2 . . . . 0 0 0 (vn + τ2) • With weights as the square roots of the diagonal elements of this matrix. 52 Estimating the Random Effects Meta-Regression Model in Practice • Option 1: Use the WLS function in Stata, SAS, or some other statistical package. – Don’t do this. All WLS algorithms treat weights as proportional variance weights rather than as absolute variance weights. This means that meta-regression parameter standard errors will be incorrect. 53 Estimating the Random Effects Meta-Regression Model in Practice • Option 2: Use the VWLS command in Stata. – This will work as long as you import the proper random effects variance weights (actually, their standard deviations). These must be calculated outside of the VWLS model. This works because VWLS treats the imported variances as known and absolute rather than as estimated and proportional. 54 Estimating the Random Effects Meta-Regression Model in Practice • Option 3: Use the “Metareg” Command in Stata – While Stata does not include commands for meta-analysis, the Stata User Group (SUG) has produced a number of good metaanalysis ADO files, including a random effects meta-regression algorithm. These commands can be downloaded from the Stata website. 55 Estimating the Random Effects Meta-Regression Model in Practice • Option 4: Use “Manually Weighted” Least Squares – Estimate the REMR model by first manually weighting each observation by the square root of the proper random effects variance, then using OLS on the transformed data. While this is more work than using the VWLS or Metareg commands, it is necessary when estimating more advanced meta-regression models. 56 Comparison of REMR Results in Environmental Justice (n=680) Intercept Pollution Census Tracts Select on DV Metareg (MLE) VWLS (MOM) MWLS (MLE) .068 (.006) -.020 (.008) .011 (.006) -.078 (.019) .073 (.008) -.020 (.011) .009 (.008) -.093 (.025) .068 (.006) -.020 (.008) .011 (.006) -.078 (.019) 57 Consequences of Inadequately Addressing Effect Size Heterogeneity • Inadequately addressing heterogeneity in effect sizes means you have a misspecified REMR model. • Consequences if r(Me,Mk) = 0? • Consequences if r(Me,Mk) ≠ 0? • Consequences of remaining effect size heterogeneity in REMR model are akin to specification error in OLS regression 13 Validity Check: Can MetaRegression Really Recover the Quantity of Interest from Original Studies? 59 Simulation Strategy • Two-level Monte-Carlo simulation – Assume Population Regression Model + β2 X2 + β3 X3 + ε. Y=β0 + β1 X1 • First level simulate 10,800 original studies by • Generate X from distribution N(0,1,Cxx), generate Y using β1 = .2, .7 (X1 is focal predictor), generate ε ~ N (0, 30) • Allow measurement of X and Y to vary across studies [X, X*1000, and ln(X)] • Estimate sample regression models that employ 60 combinations of X1, X2, and X3 Simulation Strategy (cont.) • Second level simulate 1000 meta-analyses of sample size 500 • Estimate meta-regression model – Y=β0 + β1 M1 + β2 M2 + ε – M represent moderator variables • Key Question: Can REMR recover partial correlation parameter from population regression model in original studies? 61 Simulation Results Table 7.10. Ability of REMR Model Recover Population Parameter of Interest in the Presence of Effect Size Heterogeneity and Variation in the Measurement of X and Y in Original Studies Corr Population (Xj,Xk) Parameter of Interest Estimate of b0 Parameter Parameter Estimate Estimate for for Moderator Moderator Variable X2 Variable X3 Predicted Value of Population Parameter of Interest .60 .3234 .5008 -.1648 -.0253 .3107 .60 .0950 .1644 -.0552 -.0065 .1027 62 Section IV: Advanced Meta-Regression 63 “Meta-Analytic techniques developed to date have not addressed the problem of combining information on second-order effects, such as partial regression coefficients. Consequently, meta-analysis has not been very useful in the synthesis of evidence from causal/explanatory research." (Becker 1992: 342) 64 Remaining Problems in Meta-Regression 1. Non-Independence of Observations 2. Unequal Numbers of Effect Sizes per Original Study 65 Problem 1: Non-Independence of Observations • Reasons for Correlated Observations 1. Common data sets across studies 2. Common research teams across studies 3. Single original study produces multiple effect sizes (generally incorporates both 1 and 2) • Consequences of correlated observations for REMR: incorrect parameter st. errors 66 Addressing Problem 1: The Approach in Meta-Analysis • Traditional Solutions in Meta-Analysis 1. Select one effect size per study to use in meta-analysis 2. Calculate average effect size per study and use this in meta-analysis 67 Addressing Problem 1: The Approach in Meta-Analysis • Why the Traditional Solutions are Really Bad Ideas: 1. Assumes each additional effect size from a particular study contains no information 2. Cannot estimate effect of moderators within studies, only across studies 3. Dramatically reduces sample size and statistical power 68 Addressing Problem 1: The Approach in Econometrics • The Traditional Solution in Econometrics: Cluster Robust Variance Estimation (CRVE) – i.e., Huber-White Standard Errors as adjusted for clusters by Liang and Zeger (1986) and Arellano (1987). – Effect sizes Θig clustered by original study G 69 Addressing Problem 1: The Approach in Econometrics • In Matrix Form: V[bREMRg] = (M’W’WM)-1 (Σ Mg’W’(egeg’)WMg) (M’W’WM)-1 [24] 70 (τ2+v 11 ρ1 2 +σ 1) 0 ρ1 (τ2+v . +σ21)21 0 . . . . = . . . . 0 . σ22** ρ2 ρ2 ρ2 . ρ2 . . ρ2 σ22** ρ2 . ρ2 ρ2 σ22** ρ2 . . ρ2 ρ2 ρ2 . σ22** . ee’ . . . . . . . . 0 . σ2g** ρg . ρg σ2g** ρg ρg ρg 0 . . . . . . 0 ρg σ2g** 71 Addressing Problem 1 Using Example REMR Model Intercept Pollution Census Tracts Select on DV REMR CRVE REMR .068 (.006) -.020 (.008) .011 (.006) -.078 (.019) .068 (.011) -.020 (.015) .011 (.010) -.078 (.032) 72 Addressing Problem 1 Using CRVE • CRVE REMR model should be standard practice in the Social Sciences -- BUT • Only three examples of published CRVE CRVE models. Why? • Researchers developing statistics of metaanalysis unfamiliar with developments in econometrics, and vice versa. 73 Extensions to the CRVE REMR Model 1. Small number of clusters? – Use the “Wild Cluster Bootstrap” – Cameron, Gelbach, and Miller 2008 2. Two-way clustering? E.g., effect sizes clustered by study and by data set? – Use “multi-level clustering” – Cameron, Gelbach, and Miller (2011) 74 Problem 2: Unequal Numbers of Effect Sizes Per Study • Reasons for Unequal Numbers of Effect Sizes Per Study – Varying numbers of subgroup analyses – Varying sensitivity tests • Consequences of Unequal Numbers of Effect Sizes Per Study – Few studies may dominate data set and REMR results, threatening external validity 75 Addressing Problem 2: The Approach in Meta-Analysis • Traditional Solutions in Meta-Analysis 1. Select one effect size per study to use in meta-analysis 2. Calculate average effect size per study and use this in meta-analysis • As we have seen, this is really bad advice 76 Addressing Problem 2: The Approach in Econometrics • The Econometric approach to Problem 3 generally takes one of two forms. 1. Relative “importance weights” attached to observations (e.g., weight = 1/g) 2. Generalized Estimating Equations (GEE) 77 Addressing Problem 2 Using GEE • A Simple Introduction to GEE – E[Θi] = h (Mb) [25] • Where h is a link function (identity in regression) – V[Θi|Mi] = σ2i = g([Mbi] / φ [26] • i.e., the variance is a function of the expectation • φ (phi) is a scale parameter that we will ignore today 78 Addressing Problem 2 Using GEE • A Simple Introduction to GEE (cont.) – Assume we have three effect sizes from two studies: [Θ11 , Θ21 , Θ22] • Θ = (1/3) Θ11 + (1/3) Θ21 + (1/3) Θ22 • Θ = (1/2) Θ11 + (1/2) Θ21 • Θ = (1/2) Θ11 + (1/4) Θ21 + (1/4) Θ22 [27] [28] [29] – All are unbiased estimates of population effect size Θ, but with different variances 79 Addressing Problem 2 Using GEE • A Simple Introduction to GEE (cont.) – GEE estimates Θ using an optimal weighting scheme employing the correlation among observations from the same cluster (ρ). In this simple example: Θ = [((1+ρ) / (3+ρ)) Θ11] + [(1 / (3+ρ)) Θ21] + [(1 / (3+ρ)) Θ22] [30] 80 Addressing Problem 2 Using GEE • A Simple Introduction to GEE (cont.) – We still need an estimate of the conditional variance of the sample effect size Θi. GEE estimates this variance (σ2) using: • σ2 = (A)1/2 R (A)1/2 [31] • A has E[Θi ] = h Mb on the diagonal • R is the “working correlation matrix” • For REMR, the exchangeable working correlation 81 matrix is most appropriate Addressing Problem 2 Using GEE • Obtaining GEE Estimates • Estimate a naïve model for Θ assuming independent observations to obtain starting values for b and e. • Use e to estimate the elements of the working correlation matrix R. • Re-estimate the structural model with weights derived from the estimated values of ρ in R. • Iterate steps 2 and 3 to convergence. 82 Addressing Problem 2 Using GEE A Bonus from Using GEE to Estimate the REMR Model: If we use the “empirical” matrix ee’ rather than the “model” matrix ee’, GEE also controls for Problem 2 – GEE calculates parameter estimates controlling for clustering – Empirical GEE calculates parameter variance estimates controlling for clustering 83 Addressing Problem 4 Using Example REMR Model Intercept Pollution Census Tracts Select on DV REMR CRVE REMR GEE REMR .068 (.006) -.020 (.008) .011 (.006) -.078 (.019) .068 (.011) -.020 (.015) .011 (.010) -.078 (.032) .067 (.011) -.015 (.014) .008 (.009) -.072 (.021) 84 Addressing Problem 2 Using GEE • GEE is an excellent choice for estimating the REMR model. • GEE REMR is quite rare – fewer than half a dozen in the published literature • Preferred over CRVE when number of effect sizes are highly unequal across studies. CRVE preferred in balanced 85 samples. Estimating Advanced REMR Models in Stata • Stata cannot estimate directly either the CRVE REMR or the GEE REMR model. • Neither VWLS nor Metareg can estimate these models. • Models must be estimated using MWLS data in conjunction with “robust cluster” or “xtgee” commands in Stata. 86 The Take Aways 1. Meta-regression is a powerful and underutilized tool for synthesizing research in the social sciences. 2. When estimating meta-regression models, social scientists should use the CRVE REMR or the GEE REMR model. 3. Social scientists should not use the standard REMR model or the HLM REMR model. 87 “(Meta-analysis) is going to revolutionize how the sciences . . . handle data. And it is going to be the way many arguments will end.” T. Chalmers, quoted in Mann 1990: 480 88 Evan Ringquist eringqui@indiana.edu 89