Significance and effect sizes
 What is the problem with just using p-levels to determine
whether one variable has an effect on another?
 Don’t EVER just give p-range!
 Sample results:
 For boys, r (87) = .31, p = .03
 For girls, r (98) = .24, p = .14
 Significance test = effect size x study size
 Why are effect sizes important?
 What is the difference between statistical, practical, and
clinical significance?
What should you report?
 2 group comparison—treatment vs. control on anxiety




symptoms
3 group comparison—positive prime vs. negative
prime vs. no prime on number of problems solved
2 continuous variables—relationship between
neuroticism and goal directedness
3 continuous variables—anxiety as a function of selfesteem and authoritarian parenting
2 categorical variables—relationship between answers
to 2 multiple choice questions
Narrative vs. quantitative reviews
 When was the first meta-analysis?
 When was the term first used?
 What are the advantages of quant reviews?
 What are particular critiques of them?
 What are the three basic principles to guide meta-
analysis?
Steps to meta-analysis
1. define your variables/question
 1 df contrasts
 What is a contrast?
2. Decide on inclusion criteria
 What factors do you want to consider here?
3. Collect studies systematically
 Where do you find studies?
 File drawer problem
 Rosenthal’s fail-safe N
 # studies needed at p < .05= (K/2.706) (K(mean Z squared) = 2.706)


Z = Z for that level of p
K = number of studies in meta-analysis
 Funnel plot
 Rank correlation test for pub bias
 What can you do if publication bias is a problem?
 Trim and fill
 Sensitivity analysis
 Weight studies
Fig. 3. Funnel plots of 11 (subsets of) meta-analyses from 2011 and Greenwald, Poehlman,
Uhlman, and Banaij (2009).
Marjan Bakker et al. Perspectives on Psychological Science
2012;7:543-554
Copyright © by Association for Psychological Science
3. Calculate effect sizes
 If there is more than 1 effect per study, what do you do?
 What does the sign mean on an effect size?
 What are small, medium, and large effects?
 How can you convert from one to another?
 r or d?
 http://www.soph.uab.edu/Statgenetics/People/MBeasl
ey/Courses/EffectSizeConversion.pdf
Families of effect sizes
 2 group comparisons (difference between the means)
 Cohen’s d
 Hedge’s g
 Glass’s d or delta
 Continuous or multi-group (proportion of variability)
 Eta squared η2
 Partial eta-squared ηp2
 Generalized eta-squared η G2
 r, fisher’s z, R2, adjusted R2
 ω2 and its parts
 difference between η2 and R2 family
 Nonparametric effect sizes
 Nonnormal data: convert z to r or d
 Categorical data:



Rho
Cramer’s V
Goodman-Kruskal’s Lambda
 How can you increase your effect sizes?
 How can you calculate confidence intervals around your
effect sizes?
 http://www.latrobe.edu.au/psy/research/cognitive-and-
developmental-psychology/esci
 http://www.cem.org/effect-size-calculator
Interpretation of effect sizes
 Recommended for at least most important findings
 PS
U
 Binomial effect size display (p. 76)
 Relative risk
 Odds ratio
 Risk difference
4. Look at heterogeneity of effect
sizes
 Chi-square test
 I2 (measure based on Chi-square)
 Cochran’s Q
 Standard deviations of effect sizes
 Stem and leaf plot (p. 671)
 Box plot
 Forest plot
 What are common moderators you might test? How
would you do that?
Forest plot
5. Combine effect sizes
 When should you do fixed vs. random effects?
 Should you weight effect sizes, and if so, on what?
 How can you deal with dependent effect sizes?
 Hunter and Schmidt method vs. Hedges et al. method
 Credibility intervals vs. confidence intervals
6. Calculate confidence intervals/
7. Look for moderators
 What are common moderators you might test?
 How do you compare moderators?
“Meta-analysis”
 Comparing and combining effect sizes on a smaller




level—when might you want to do this?
How would you do it?
Average within-cell r’s with fisher z transforms
To compare independent r’s: Z = z1-z2/sqrt ((1/n-3) +
(1/n-3))
To combine independent r’s: z = z1+z2/2
Write-up
 Inclusion criteria, search, what effect size
 Which m-a tech and why
 Stem and leaf plots of effect sizes (and maybe mods)
 Forest plots
 Stats on variability of effect sizes, estimate of pop
effect size and confidence intervals
 Publication bias analyses
Side note
 Analysis of power (Appendix)
Terms
 Evolutionary epistemology
 Evidence-based practice
 Systems thinking
 Dynamical systems approaches
 Evaluation research
Issues with evaluation research
 What questions are asked?
 What methods are used?
 What unique issues emerge?
Types of evaluation
 Formative
 Needs assessment
 Evaluability assessment
 Structured conceptualization
 Implementation evaluation
 Process evaluation
 Summative
 Outcome evaluation
 Impact evaluation
 Cost-benefit analysis
 Secondary analysis
 Meta-analysis
Methods used for different ?s
 What is the scope of the problem?
 How big is the problem?
 How should we deliver the program?
 How well did we deliver it?
 What type of evaluation can we do?
 Was the program effective?
 What parts of the program work?
 Should we continue the program?
Evidence based medicine (Sackett
et al.)
 Convert problem into question
 Find evidence
 Evaluate validity, impact, applicability
 Integrate patient experience and clinical judgment
 Review evaluation
What does the book author
 Mean by an “evaluation culture”?
 Is it a good thing?
Post spring break
 Readings on analyses (some to be emailed out)
 Quant article critique is separate from thought paper
(look for questions at end of syllabus)
 One more week then rough drafts due