Professor Tod Mijanovich
tod.mijanovich@nyu.edu
Puck Building Room 2303
Office hours: Wednesdays 4 – 6pm or by appointment
212-998-7467 (voice) 212-995-4164 (fax)
Estimating Impacts in Policy Research (P11.2875)
Fall 2009
Wednesdays 6:45 – 8:25, Room 207, 194 Mercer St.
This course covers selected analytic and design issues that are relevant to policy research
and program evaluation. It applies and extends skills that are developed in other courses
that are offered at the Wagner School. For example, the concepts of experimental and
quasi-experimental design that are introduced in Program Evaluation and Analysis
(P11.2171) are applied in understanding and critiquing research reports, and in analyzing
data. Multivariate analytic skills introduced in Statistical Data Analysis: Multiple
Regression (P11.2902) are extended to various types of research designs and analytic
situations.
Your goals in this course should be to:
get hands-on experience in analyzing and presenting data, including
managing data, selecting appropriate analyses, interpreting computer
output, and summarizing and presenting results. In this course, you will
become "bilingual", using both SPSS and Stata;
extend your familiarity with methodological issues in research, including
various study designs, measurement problems and analytic approaches;
apply your skills by reading about and reporting on policy-relevant impact
analyses written by others.
The course is not a comprehensive or exhaustive review of the field of policy-relevant
research or program evaluation, nor is it a course in how to evaluate a program. The
focus is on impact analysis (rather than process evaluation, performance monitoring, cost
effectiveness analysis, or evaluation synthesis, all of which are covered in other Wagner
courses). To that end, there is a substantial amount of data analysis both in and out of
class. There is also a significant amount of new statistical material presented. All of this
is done using real world examples, to solidify the base as you build your career as a
practitioner and consumer of the research that informs public policy.
2875 Syllabus – Fall 2009
1
Course prerequisites (neither may be taken concurrently)
1. Program Evaluation and Analysis (P11.2171)
2. Statistical Data Analysis: Multiple Regression (P11.2902)
Course texts & software: No required new purchase.
1. Healey JF. Statistics: A Tool for Social Research, or The Essentials of Statistics
(whichever you used for your Stat 1 textbook; hereafter HJF)
2. Stock JH and Watson MW. Introduction to Econometrics. 2007. Boston: Addison
Wesley. (This hasn’t been used in a prior class; you don’t need to purchase;
chapters will be made available; hereafter S&W).
3. SPSS & Stata – versions to be discussed; purchase not mandatory.
4. Blustein J. SPSS: The Wagner Way. (available at the WagnerWorks café @ Puck;
may be useful to review basic data analysis).
Additional information
Blackboard (BB) A number of resources have been posted on the course BB. These
include the syllabus, datasets, assignments, articles, artcile links and other core course
documents. There are also links to supplementary materials (primarily in the form of
weblinks) that extend some of the topics discussed in class.
As the semester progresses, assignments and teaching resources will be added to the BB.
Note that some posted material will be modified as the semester progresses, so you
should not download and print everything at the outset. BB will be also be used to
communicate urgent matters such as assignment changes or glitches, class cancellations,
and changes in office hours. You should check the site frequently. Also, you will be
receiving email via BB, which goes to your NYU mailing account by default. You can
arrange forwarding to another account via “Preferences” (upper right hand corner,
NYUhome).
Access to SPSS and Stata. A key goal of the course is to improve your data analysis
skills. As you have found in prior courses, analyzing data means trying things out,
making mistakes, consulting notes and books, talking to peers and instructors, and redoing and refining your work. In order to do this at the level that we will be working, you
must have ready access to a computer that has both SPSS and Stata installed. If you
wish, you may use the NYU labs, but as you know, those can get crowded. Both SPSS
and Stata are available for installation to graduate students at a reasonable rate (see
course BB for details). You will not be required to use Stata until the end of the course,
but the earlier you start, the more you will learn.
Missed classes. This course moves quickly. There is a lot of work! While the reading is
important, there is a fair bit of informal in-class give-and-take. An effort is made to
capture class work in handouts, but this is not always feasible. Students who miss a class
2875 Syllabus – Fall 2009
2
should ask a partner student for copies of notes and handouts. Inasmuch as is possible,
handouts for each week will be posted on BB.
Non-graded assignments. There are many non-graded assignments. Most are designed
to help you to prepare for class. In discussing them, I reserve the right to “cold call” on
students (as is often done in schools of law). Come to class prepared to contribute!
Practice problems. Practice problems (and answers) will be distributed early on in the
course, giving you the opportunity to use some of the concepts and techniques that will
be covered on the in-class exam. While you will not submit the answers to the practice
problems for grading, you will probably find it quite helpful to complete them.
Course grades will be based on:
1. Assignment 1 (10%)
2. Assignment 2 (10%)
3. Assignment 3 (15%)
4. Assignment 4 (10%)
5. Assignment 5 (15%)
6. In-class examination
7. Class participation
Supplemental insurance and Medicare expenditures
Topic TBA
Impact of a mentoring program in New York City
Topic TBA
Enterprise zones and employment rates
(30% of grade)
(10% of grade)
2875 Syllabus – Fall 2009
3
TOPICS
Week 1. September 9. A framework for estimating impacts. Goals of the course,
course mechanics. Impacts, outcomes and the counterfactual. Review of estimation
concepts with extension to the regression framework.
To prepare:

Take several hours to review your class notes and textbooks from P11.2902
(Multiple Regression) and P11.2171 (Program Analysis and Evaluation)
Week 2. September 16. Validity in social research. Causality and validity in social
research. Internal validity, external validity, validity in measurement, conclusion
validity. Basic study designs and threats to internal validity. Discussion of Newcomb
paper.
To prepare:


Review basic study designs and threats to validity from Program Analysis and
Evaluation (P11.2171)
Read Newcomb, T.M. "Conservation program evaluations: The control of self
selection bias". Evaluation Review. June 1984 Volume 8, Number 3. Pages 425
– 440.
o Non-graded assignment:
Worksheet on Newcomb paper (posted on
Blackboard).

Review material on Type I and Type II errors from Stat 1, and the
read the Trochim material that is posted on Blackboard.

Download and preview Assignment 1; come to class with
questions
If you want to read further to understand any of the concepts presented in the first two
weeks of class, the Trochim website is an excellent source – see the class Blackboard
for the weblink.
Week 3. September 23. Continuous outcomes, cross sectional data (1). Thinking
further about what it means to "adjust for" or "hold constant". Discussion of Devaney
article. Beginning of work on Assignment 1.
To prepare:

Read Devaney B, Bilheimer L, Schore J. “Medicaid costs and birth outcomes:
The effects of prenatal WIC participation and the use of prenatal care". Journal
2875 Syllabus – Fall 2009
4
of Policy Analysis & Management. 1992; Volume 11, Number 4. Pages 573-592,
and
Blustein J. "Medicare coverage, supplemental insurance, and the use of
mammography by older women". New England Journal of Medicine. 1995;
332:1138-1143.
o Non-graded assignment: Prepare an "11-points" discussion of the Devaney et
al. and the Blustein articles (the 11-points document is posted on Blackboard).
o Begin Assignment 1 during this week; come to class with questions.
Week 4. September 30. Continuous outcomes, cross sectional data (II). Continued
discussion of the problem of selection bias.
ASSIGNMENT 1 DUE
To prepare:

Read Bitler MP, Currie J. “Does WIC work? The effects of WIC on pregnancy
and birth outcomes”. Journal of Policy Analysis and Management. 2005;
Volume 24, No. 1, Pages 73-91. (note that this paper has some advanced
material, including odds ratios and 2-stage least squares – don’t worry about
this);
Joyce T, Gibson D, Colman S. “The changing association between prenatal
participation in WIC and birth outcomes in New York City. 2005; Volume 24;
No. 4; and
Bitler MP, Currie J. 2005. The changing association between prenatal
participation in WIC and birth outcomes in New York City: What does it mean?
Journal of Policy Analysis and Management. Vol 24, No. 4. 687-690.
o Non-graded assignment: Worksheet on WIC articles by Bitler et al and Joyce et
al. (Posted under Week 4 on Blackboard)
Week 5. October 7. Dichotomous outcomes (I): The econometrician’s perspective.
Review of the concept of association from Stat 1 (strength versus magnitude).
Dichotomous variables and the fourfold table. The linear probability, logit and probit
models. Clarification of what “non-linearity” means.
To prepare:
o Non-graded assignments – make sure that your math skills are up to par by
completing these class Blackboard postings. ** If you don’t get this, you won’t
get what follows **
o “Review of percent change and percentage point change”
o “Review of logarithms” – note that you should perform a calculator check;
bring your calculator to class.
 Read: S&W Chapter 11, “Regression with a binary dependent variable”
2875 Syllabus – Fall 2009
5

Read: Munnell A.H. et al. "Mortgage lending in Boston: Interpreting the HMDA
data". American Economic Review. 1996; Vol. 86, no. 1, pages 25-53 – paying
special attention to the tables, and then:
o Non-graded assignment: Prepare an 11-points discussion of the Munnell article
Week 6. October 14. Dichotomous outcomes (II): The health researcher’s
perspective. The fourfold table. Odds versus risk. The RD, RR and OR as measures of
the magnitude of association.
To prepare:




Re-read S&W Chapter 11 from last week, and review your class notes
o Complete the non-graded assignment that is posted on Blackboard
(“Probit Practice”).
Read: Background material on the fourfold table (posted on Blackboard)
o Non-graded assignment: Work the problems on pages 3 & 4 of the posted
document (fourfold table.doc).
Download and preview Assignment 2; bring questions to class.
Read: Guttmatcher S. et al. "Condom availability in NYC public high schools:
Relationships to condom use and sexual behavior." American Journal of Public
Health. 1997. Volume 87, no. 9, pages 1427-143, paying special attention to the
tables, which we will discuss.
o Prepare an 11-points discussion of the Guttmacher et al. article
Week 7. October 21.
Finishing up dichotomous outcomes and non-linearity;
Interaction as another form of non-linearity. Subgroup effects and coding of
interactions in regressions.
To prepare:
Review material on interactions from your Stat 2 course. There will be a review reading
posted on the website, if you are not confident on this.
 Read and be ready to summarize the issues from:
Schulman, K. A. et al. "The effect of race and sex on physicians'
recommendations for cardiac catheterization". New England Journal of
Medicine. 1999. Vol. 340, no. 8, pages 618-26.
and
 Schwartz L.M. et al. "Misunderstandings about the effects of race and sex on
physicians' referrals for cardiac catheterization. New England Journal of
Medicine. 1999. Volume 341, no. 4, pages 279-283.
 Download and preview Assignment 3, and bring questions to class.
2875 Syllabus – Fall 2009
6
Week 8. October 28. Panel data. Simplest panel data analysis (pooled cross sections).
Two period panel data, multi period panels. Method of first differences; fixed effects
estimation.
ASSIGNMENT 2 due
To prepare:
Carefully read Chapter 10 of S&W (“Regression with Panel Data”) and also these
papers:
 Ruhm, CJ. “Alcohol policies and highway vehicle fatalities”. Journal of Health
Economics. 1996;15(4):435-454;
 Blustein J, Chan S, Guanais FC. “Depression among caregiving grandparents” .
HSR: Health Services Research. 2004; 39(6):1672-1689; and
 Laxminarayan R. “Does reducing malaria improve household living standards?”
Tropical Medicine and International Health. 2004;9(2):267-272.
 Geronimus AT, Korenman S. "The socioeconomic consequences of teen
childbearing reconsidered". Quarterly Journal of Economics. 1992. 101: 1187 –
1214;
and then:
o Prepare an 11 – points discussion of the Laxminariyan paper, and the
Geronimus & Korenman paper.
Week 9. November 4. Experiments. The rationale for social experiments. New wave
of interest in social experiments: why? Residual threats to validity. Logistical issues in
implementation.
To prepare, read these:





Burghardt J. et al. "Does Job Corps Work? Summary of the National Job Corps
Study." Princeton, NJ: Mathematica Policy Research, Inc., June 2001. Document
No. PR01-50.
Burghardt J. et al. "National Job Corps Study: Report on Study Implementation."
Princeton, NJ: Mathematica Policy Research, Inc., April 1999. Document No.
PR99-28.
Lynn J. et al. "A Controlled Trial to Improve Care for Seriously Ill Hospitalized
Patients: The Study to Understand Prognoses and Preferences for Outcomes and
Risks of Treatments (SUPPORT)." Journal of the American Medical Association.
1995:274(20):1591-1598.
Lynn J. et al. "Ineffectiveness of the SUPPORT Intervention: Review of
Explanations". Journal of the American Geriatrics Society - Volume 48, Issue 5
(May 2000).
Lynn J. et al. “Rethinking Fundamental Assumptions: SUPPORT's Implications
for Future Reform.” Journal of the American Geriatrics Society - Volume 48,
Issue 5 (May 2000).
2875 Syllabus – Fall 2009
7

Gertler P. Final Report: Impact of Progresa on Health. International Food
Policy Research Institute, Food Consumption and Nutrition Division ( 2008).
Week 10. November 11. Ethics in social research. Overview of issues in social
research ethics. Federal regulations to protect human subjects; the Institutional Review
Board (IRB) and criteria for review. Case discussion: The Kennedy Krieger Lead Paint
Abatement Study.
ASSIGNMENT 3 DUE
To prepare:

Begin by downloading the non-graded assignment sheet posted on the course BB.
You can complete the non-graded assignment as you read the following:
Oakes JM. “Risks and wrongs in social science research: An Evaluator’s Guide
to the IRB.” Evaluation Review Vol. 26 No. 5, October 2002 443-479.
Blustein J. “Toward a more public discussion of the ethics of federal social
program evaluation”. Journal of Policy Analysis and Management. 2005; 24 (4),
824-846; and 851-852; and the exchange that followed.
Johns Hopkins University, “Lead based paint study”. A fact sheet posted by the
university on the School of Medicine website. Our BB includes the link to lots of
other material that Hopkins has made available.
Court of Appeals of Maryland. Erika Grimes vs. Kennedy Krieger Institute.
Decision. Nos. 128, 129, September; Term 2000. August 16, 2001.
Buchanan DR, Miller FJ. “Justice and fairness in the Kennedy Krieger Institute
lead paint study: the ethics of public health research on less expensive, less
effective interventions”. American Journal of Public Health. 2006;96:781–787.
Week 11. November 18. In-class late midterm exam
Week 12. November 25. Hands on Stata Session I
Introduction to Stata. Demonstration of some aspects of the Stata interface not covered
in the UCLA Stata material. Presentation of new material on data management; in-class
work on Assignment 4.
To prepare: (TBA, tentatively)


Complete the UCLA Stata material
Download and preview Assignment 4
2875 Syllabus – Fall 2009
8
Week 13. December 2 Hands on Stata Session II. Specifying fixed effects models in
Stata. In-class work on Assignment 5.
To prepare:

Download and preview Assignment 5.
Week 14. December 9. Hands on Stata Session III.
complete all assignments.
Optional in-class lab period to
ASSIGNMENTS 4 and 5 due by December 18.
2875 Syllabus – Fall 2009
9