Texas A&M University
EPSY 642
FALL 2009
Victor L. Willson, Instructor
 EPSY 642 Meta Analysis of Social Science Research Syllabus
 FALL 2009
 TUESDAYS 4:45 – 7:35
 Victor L. Willson, Professor:
Office: by appt
 704 Harrington 845-0904 /fax: 862-1256
email:v-willson@tamu.edu

 Course description: Theory and methodology of current state of the topic of
meta-analysis of research documents.
 Course objectives: Students will learn meta-analysis methods by conducting a
meta-analysis on a circumscribed topic of their choice. Students will be
required to present their findings at the end of the course and produce a
research paper on their topic. Students will be encouraged to develop a
potentially publishable research article from their work. Components for the
project will be reported by students during the semester.

 Texts: Lipsey, M. W., & Wilson, D. B. (2001). Practical Meta-Analysis. Sage.

Cooper, H. (2010). Research Synthesis and Meta-Analysis. 4th Ed.
Sage
 Date
due
Topic
 SEP 1
Introduction: Purposes, Approaches HC 1


Assignment
Activity: Online Google Scholar search 5 articles: samples description, variable list
8
 15


22

Reading
NO CLASS
Problem finding, Study Selection
LW2, HC 2-3
Study coding, menu elements
LW4, 5, HC 4, handout
29
Outcome variable effect computation LW3, HC 6, handouts
 OCT 6
Study Design variables
HS 2, HC 5

13
Correlational study analysis
LW 6-7

Experimental study analysis
LW 7
#1

27
Findings cumulation and reporting HC 6-9, LW 8
#2

handouts & examples
 NOV 3
Advanced Modeling
handouts
#3

10-17
Individual meetings
#4 Nov 10, #5 Nov 17

24
Presentation of student papers using powerpoint
 Grades: Based on completion of intermediate assignments (20%) and quality of paper
(80%)
 U= < 60% S = 61% - 100%

 Intermediate assignments:
 1




Proposed meta-analysis topic with preliminary list of at least 20 articles
2
Preliminary list of proposed variables to be extracted from articles
3
Proposed Coding menu for variables
4
Preliminary analysis of outcomes using Q method, funnel plot of one outcome
variable
5
Preliminary covariate analysis using QB and QW method (at least 1 covariate)

 Project and paper: conduct a meta-analysis using the procedures of this course on a topic
of your choice; limit the number of studies to be finally included to no more than about
20 but at least 8 as appropriate to the topic. Produce a finished paper in APA format
summarizing the procedures and results; produce a powerpoint of your paper for
presentation.

 In order to receive a grade a presentation must be made on the dates listed; otherwise an
incomplete will be given and a presentation to me and a group of students must be made
before a final grade can be submitted. Final papers may be turned in after the last day to
turn in grades for the semester with a grade of I given and no penalty, but after a
presentation has been made and all intermediate assignments are completed. After one
semester a grade of F will be given if these conditions are not met.
 Note: The handouts and web-based files used in this course are copyrighted. By
“handouts” I mean all materials generated for this class, which includes but is
not limited to syllabi, quizzes, exams, lab problems, in-class materials, review
sheets, and additional problem sets, in paper or electronic form. Because these
materials are copyrighted, you do not have the right to copy the handouts
unless I expressly grant permission.

As commonly defined, plagiarism consists of passing off as one’s own ideas,
words, writings, etc. which belong to another. In accordance with this
definition, you are committing plagiarism if you copy the work of another
person and turn it in as your own, even if you should have the permission of
that person. Plagiarism is one of the worst academic sins, for the plagiarist
destroys the trust among colleagues, without which research cannot be safely
communicated.
 If you have any questions regarding plagiarism, please consult the latest issue
of the Texas A&M University Student Rules, under the section “Scholastic
Dishonesty”. Honor Code and refers the student to the Honor Council Rules
and Procedures on the web:
 http://www.tamu.edu/aggiehonor.)
 “An Aggie does not lie, cheat or steal, or tolerate those who do.”
Purposes for Meta-Analysis
• Cumulate findings of studies on a particular topic
• Examine homogeneity of outcomes
• Estimate effects of independent variables on outcomes
in a standardized format
– Evaluate moderator and mediator effects on outcomes
– Differentiate different types or classes of outcome effects
Historical background
 Criticism of traditional narrative reviews of research
 Exasperation in social sciences with constructs
measured different ways in terms of determining
consistencies
 Need to formulate theoretical relationships based on
many studies
History part 2
 Early 1970s efforts focused on significance testing and
“vote counts” of significance
 Glass (1976) presented a method he called “metaanlaysis” in Am. Ed. Research Assn. presidential
address
 Others proposed related methods, but Glass and
colleagues developed the most widely used approach
(Glass, McGaw, & Smith, 1981)
Meta-Analysis as Survey Research
• Research articles as unit of focus
• Population defined
– Conditions for inclusion of articles
•
•
•
Data requirements needed for inclusion
Completeness of data available in article or estimable
Publication sources available, selected
• Sample vs. population acquisition
– Availability of publications and cost
– Time to acquisition (how long to wait for retrieval)
Strengths of Meta-Analysis
 Definition of effect and effect size beyond “significant
or not”
 Focus on selection threats in traditional reviews (bias
in selection of articles for review)
 Systematic consideration of potential mediators and
moderators of effects
 Data organization of articles for public review
Weaknesses of Meta-Analysis
 Methodologically sophisticated and expensive
 Potential ignoring of contextual effects not easily
quantified; eg. historical/environmental placement
of research
 Potential improper mixing of studies
 Averages hiding important subgroupings
 Improperly weighting studies with different
methodological strength/rigor
Research focus for meta-analysis
 Defining and delineating the construct
 Determining a research outlet
 Meta-Analysis as an interactive, developing process
Recent Criticism
 Suri & Clarke (2009): Advancements in Research
Synthesis Methods: From a Methodologically Inclusive
Perspective (Review of Educational Research, pp. 395-430)
 They propose 6 overlapping approaches:
1. Statistical research syntheses (eg. meta-analysis)
2. Systematic reviews
3. Qualitative research syntheses
4. Qualitative syntheses of qualitative and quantitative
research
5. Critical impetus in reviewing research
6. Exemplary syntheses
Some critical comments on Suri & Clarke (2009)
 Systematic reviews- original Glass criticisms hold: what is
the basis for inclusion and exclusion; why are certain
articles privileged?
 Qualitative research syntheses- how can these be done
with situated contexts, small samples, environmentallydeveloped variables, sources, etc.? Will there be a review for
every reader, or for every researcher? Same limitation as all qual
research
 Qual syntheses of quant and qual research- potentially
doable, with an alternating order: qual first to focus emphases
in the quant analysis, or quant first to be validated with the qual studies
of particular environments and populations- do they fit/match in
reasonable ways?
 Critical impetus- code words for critical theory/Marxist etc.
Answer is already known, why do the research?
 Exemplary syntheses- what is the purpose?
Defining and Delineating the
Research Topic
• Outcome construct definition
– Importance to the field to know what has been learned
– How big is it? How many potential studies?
– Conduct preliminary searches using various databases
• Refining the construct
– How much resource is available? Eg. 1000 studies = 2-3 years
work
– Are there specific sub-constructs more important than
others? Select them or one of them
– Are there time-limitations (no studies before 19xx)
– Are there too few studies for the given construct, should it be
broadened? Too few-> less than 10?
Defining and Delineating the
Research
Topic
 What is the typical research approach for the topic
area?
 All quantitative
 All qualitative
 Mixed quantitative and qualitative
 Are there sufficient quantitative studies to provide
evidence for findings?
 Can qualitative studies be included as a separate part of
the study? How?
Determining Research Outlet
• Does the proposed journal
– publish research on the construct?
– Publish reviews or meta-analyses?
• Is there a journal devoted to reviews that your project
would fit with?
• Has a recent similar meta-analysis been published? If so,
will yours add anything new?
– Ex. Allen, et al (under review) evaluated articles on first grade
retention after 1990 focusing on the quality of the research
design in each study to determine if the effects were different
from a fairly recent meta-analysis by Jimerson (2001)
Meta-Analysis as an interactive,
developing process
• View meta-analysis as evolutionary
– As studies are reviewed and included, purpose and
scope may change
• Assume initial conceptualizations about both
outcomes and potential predictors may change over
time
– Definitions, instruments, coding may all change as
studies are found and included
• Plan for revisions to all aspects of the meta-analysis
ACTIVITY
 Get into groups of 3 or 4
 Use GOOGLE SCHOLAR to find research articles from
the following:
 Group 1: elementary school grade retention
 Group 2: teenage depression treatments
 Group 3: IQ and socioeconomic status
 Groups 4+: one of above
 Select 5 articles that have empirical data
 For each article
 define the population being samples
 List with specific names the variables used in the study
 Discuss within your group the similarities and differences of the
populations and variables; consider difficulties you would have in
summarizing research for these
 Plan to report to the class what you found