Professor Tod Mijanovich tod.mijanovich@nyu.edu Puck Building Room 2303 Office hours: Wednesdays 4 – 6pm or by appointment 212-998-7467 (voice) 212-995-4164 (fax) Estimating Impacts in Policy Research (P11.2875) Fall 2009 Wednesdays 6:45 – 8:25, Room 207, 194 Mercer St. This course covers selected analytic and design issues that are relevant to policy research and program evaluation. It applies and extends skills that are developed in other courses that are offered at the Wagner School. For example, the concepts of experimental and quasi-experimental design that are introduced in Program Evaluation and Analysis (P11.2171) are applied in understanding and critiquing research reports, and in analyzing data. Multivariate analytic skills introduced in Statistical Data Analysis: Multiple Regression (P11.2902) are extended to various types of research designs and analytic situations. Your goals in this course should be to: get hands-on experience in analyzing and presenting data, including managing data, selecting appropriate analyses, interpreting computer output, and summarizing and presenting results. In this course, you will become "bilingual", using both SPSS and Stata; extend your familiarity with methodological issues in research, including various study designs, measurement problems and analytic approaches; apply your skills by reading about and reporting on policy-relevant impact analyses written by others. The course is not a comprehensive or exhaustive review of the field of policy-relevant research or program evaluation, nor is it a course in how to evaluate a program. The focus is on impact analysis (rather than process evaluation, performance monitoring, cost effectiveness analysis, or evaluation synthesis, all of which are covered in other Wagner courses). To that end, there is a substantial amount of data analysis both in and out of class. There is also a significant amount of new statistical material presented. All of this is done using real world examples, to solidify the base as you build your career as a practitioner and consumer of the research that informs public policy. 2875 Syllabus – Fall 2009 1 Course prerequisites (neither may be taken concurrently) 1. Program Evaluation and Analysis (P11.2171) 2. Statistical Data Analysis: Multiple Regression (P11.2902) Course texts & software: No required new purchase. 1. Healey JF. Statistics: A Tool for Social Research, or The Essentials of Statistics (whichever you used for your Stat 1 textbook; hereafter HJF) 2. Stock JH and Watson MW. Introduction to Econometrics. 2007. Boston: Addison Wesley. (This hasn’t been used in a prior class; you don’t need to purchase; chapters will be made available; hereafter S&W). 3. SPSS & Stata – versions to be discussed; purchase not mandatory. 4. Blustein J. SPSS: The Wagner Way. (available at the WagnerWorks café @ Puck; may be useful to review basic data analysis). Additional information Blackboard (BB) A number of resources have been posted on the course BB. These include the syllabus, datasets, assignments, articles, artcile links and other core course documents. There are also links to supplementary materials (primarily in the form of weblinks) that extend some of the topics discussed in class. As the semester progresses, assignments and teaching resources will be added to the BB. Note that some posted material will be modified as the semester progresses, so you should not download and print everything at the outset. BB will be also be used to communicate urgent matters such as assignment changes or glitches, class cancellations, and changes in office hours. You should check the site frequently. Also, you will be receiving email via BB, which goes to your NYU mailing account by default. You can arrange forwarding to another account via “Preferences” (upper right hand corner, NYUhome). Access to SPSS and Stata. A key goal of the course is to improve your data analysis skills. As you have found in prior courses, analyzing data means trying things out, making mistakes, consulting notes and books, talking to peers and instructors, and redoing and refining your work. In order to do this at the level that we will be working, you must have ready access to a computer that has both SPSS and Stata installed. If you wish, you may use the NYU labs, but as you know, those can get crowded. Both SPSS and Stata are available for installation to graduate students at a reasonable rate (see course BB for details). You will not be required to use Stata until the end of the course, but the earlier you start, the more you will learn. Missed classes. This course moves quickly. There is a lot of work! While the reading is important, there is a fair bit of informal in-class give-and-take. An effort is made to capture class work in handouts, but this is not always feasible. Students who miss a class 2875 Syllabus – Fall 2009 2 should ask a partner student for copies of notes and handouts. Inasmuch as is possible, handouts for each week will be posted on BB. Non-graded assignments. There are many non-graded assignments. Most are designed to help you to prepare for class. In discussing them, I reserve the right to “cold call” on students (as is often done in schools of law). Come to class prepared to contribute! Practice problems. Practice problems (and answers) will be distributed early on in the course, giving you the opportunity to use some of the concepts and techniques that will be covered on the in-class exam. While you will not submit the answers to the practice problems for grading, you will probably find it quite helpful to complete them. Course grades will be based on: 1. Assignment 1 (10%) 2. Assignment 2 (10%) 3. Assignment 3 (15%) 4. Assignment 4 (10%) 5. Assignment 5 (15%) 6. In-class examination 7. Class participation Supplemental insurance and Medicare expenditures Topic TBA Impact of a mentoring program in New York City Topic TBA Enterprise zones and employment rates (30% of grade) (10% of grade) 2875 Syllabus – Fall 2009 3 TOPICS Week 1. September 9. A framework for estimating impacts. Goals of the course, course mechanics. Impacts, outcomes and the counterfactual. Review of estimation concepts with extension to the regression framework. To prepare: Take several hours to review your class notes and textbooks from P11.2902 (Multiple Regression) and P11.2171 (Program Analysis and Evaluation) Week 2. September 16. Validity in social research. Causality and validity in social research. Internal validity, external validity, validity in measurement, conclusion validity. Basic study designs and threats to internal validity. Discussion of Newcomb paper. To prepare: Review basic study designs and threats to validity from Program Analysis and Evaluation (P11.2171) Read Newcomb, T.M. "Conservation program evaluations: The control of self selection bias". Evaluation Review. June 1984 Volume 8, Number 3. Pages 425 – 440. o Non-graded assignment: Worksheet on Newcomb paper (posted on Blackboard). Review material on Type I and Type II errors from Stat 1, and the read the Trochim material that is posted on Blackboard. Download and preview Assignment 1; come to class with questions If you want to read further to understand any of the concepts presented in the first two weeks of class, the Trochim website is an excellent source – see the class Blackboard for the weblink. Week 3. September 23. Continuous outcomes, cross sectional data (1). Thinking further about what it means to "adjust for" or "hold constant". Discussion of Devaney article. Beginning of work on Assignment 1. To prepare: Read Devaney B, Bilheimer L, Schore J. “Medicaid costs and birth outcomes: The effects of prenatal WIC participation and the use of prenatal care". Journal 2875 Syllabus – Fall 2009 4 of Policy Analysis & Management. 1992; Volume 11, Number 4. Pages 573-592, and Blustein J. "Medicare coverage, supplemental insurance, and the use of mammography by older women". New England Journal of Medicine. 1995; 332:1138-1143. o Non-graded assignment: Prepare an "11-points" discussion of the Devaney et al. and the Blustein articles (the 11-points document is posted on Blackboard). o Begin Assignment 1 during this week; come to class with questions. Week 4. September 30. Continuous outcomes, cross sectional data (II). Continued discussion of the problem of selection bias. ASSIGNMENT 1 DUE To prepare: Read Bitler MP, Currie J. “Does WIC work? The effects of WIC on pregnancy and birth outcomes”. Journal of Policy Analysis and Management. 2005; Volume 24, No. 1, Pages 73-91. (note that this paper has some advanced material, including odds ratios and 2-stage least squares – don’t worry about this); Joyce T, Gibson D, Colman S. “The changing association between prenatal participation in WIC and birth outcomes in New York City. 2005; Volume 24; No. 4; and Bitler MP, Currie J. 2005. The changing association between prenatal participation in WIC and birth outcomes in New York City: What does it mean? Journal of Policy Analysis and Management. Vol 24, No. 4. 687-690. o Non-graded assignment: Worksheet on WIC articles by Bitler et al and Joyce et al. (Posted under Week 4 on Blackboard) Week 5. October 7. Dichotomous outcomes (I): The econometrician’s perspective. Review of the concept of association from Stat 1 (strength versus magnitude). Dichotomous variables and the fourfold table. The linear probability, logit and probit models. Clarification of what “non-linearity” means. To prepare: o Non-graded assignments – make sure that your math skills are up to par by completing these class Blackboard postings. ** If you don’t get this, you won’t get what follows ** o “Review of percent change and percentage point change” o “Review of logarithms” – note that you should perform a calculator check; bring your calculator to class. Read: S&W Chapter 11, “Regression with a binary dependent variable” 2875 Syllabus – Fall 2009 5 Read: Munnell A.H. et al. "Mortgage lending in Boston: Interpreting the HMDA data". American Economic Review. 1996; Vol. 86, no. 1, pages 25-53 – paying special attention to the tables, and then: o Non-graded assignment: Prepare an 11-points discussion of the Munnell article Week 6. October 14. Dichotomous outcomes (II): The health researcher’s perspective. The fourfold table. Odds versus risk. The RD, RR and OR as measures of the magnitude of association. To prepare: Re-read S&W Chapter 11 from last week, and review your class notes o Complete the non-graded assignment that is posted on Blackboard (“Probit Practice”). Read: Background material on the fourfold table (posted on Blackboard) o Non-graded assignment: Work the problems on pages 3 & 4 of the posted document (fourfold table.doc). Download and preview Assignment 2; bring questions to class. Read: Guttmatcher S. et al. "Condom availability in NYC public high schools: Relationships to condom use and sexual behavior." American Journal of Public Health. 1997. Volume 87, no. 9, pages 1427-143, paying special attention to the tables, which we will discuss. o Prepare an 11-points discussion of the Guttmacher et al. article Week 7. October 21. Finishing up dichotomous outcomes and non-linearity; Interaction as another form of non-linearity. Subgroup effects and coding of interactions in regressions. To prepare: Review material on interactions from your Stat 2 course. There will be a review reading posted on the website, if you are not confident on this. Read and be ready to summarize the issues from: Schulman, K. A. et al. "The effect of race and sex on physicians' recommendations for cardiac catheterization". New England Journal of Medicine. 1999. Vol. 340, no. 8, pages 618-26. and Schwartz L.M. et al. "Misunderstandings about the effects of race and sex on physicians' referrals for cardiac catheterization. New England Journal of Medicine. 1999. Volume 341, no. 4, pages 279-283. Download and preview Assignment 3, and bring questions to class. 2875 Syllabus – Fall 2009 6 Week 8. October 28. Panel data. Simplest panel data analysis (pooled cross sections). Two period panel data, multi period panels. Method of first differences; fixed effects estimation. ASSIGNMENT 2 due To prepare: Carefully read Chapter 10 of S&W (“Regression with Panel Data”) and also these papers: Ruhm, CJ. “Alcohol policies and highway vehicle fatalities”. Journal of Health Economics. 1996;15(4):435-454; Blustein J, Chan S, Guanais FC. “Depression among caregiving grandparents” . HSR: Health Services Research. 2004; 39(6):1672-1689; and Laxminarayan R. “Does reducing malaria improve household living standards?” Tropical Medicine and International Health. 2004;9(2):267-272. Geronimus AT, Korenman S. "The socioeconomic consequences of teen childbearing reconsidered". Quarterly Journal of Economics. 1992. 101: 1187 – 1214; and then: o Prepare an 11 – points discussion of the Laxminariyan paper, and the Geronimus & Korenman paper. Week 9. November 4. Experiments. The rationale for social experiments. New wave of interest in social experiments: why? Residual threats to validity. Logistical issues in implementation. To prepare, read these: Burghardt J. et al. "Does Job Corps Work? Summary of the National Job Corps Study." Princeton, NJ: Mathematica Policy Research, Inc., June 2001. Document No. PR01-50. Burghardt J. et al. "National Job Corps Study: Report on Study Implementation." Princeton, NJ: Mathematica Policy Research, Inc., April 1999. Document No. PR99-28. Lynn J. et al. "A Controlled Trial to Improve Care for Seriously Ill Hospitalized Patients: The Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatments (SUPPORT)." Journal of the American Medical Association. 1995:274(20):1591-1598. Lynn J. et al. "Ineffectiveness of the SUPPORT Intervention: Review of Explanations". Journal of the American Geriatrics Society - Volume 48, Issue 5 (May 2000). Lynn J. et al. “Rethinking Fundamental Assumptions: SUPPORT's Implications for Future Reform.” Journal of the American Geriatrics Society - Volume 48, Issue 5 (May 2000). 2875 Syllabus – Fall 2009 7 Gertler P. Final Report: Impact of Progresa on Health. International Food Policy Research Institute, Food Consumption and Nutrition Division ( 2008). Week 10. November 11. Ethics in social research. Overview of issues in social research ethics. Federal regulations to protect human subjects; the Institutional Review Board (IRB) and criteria for review. Case discussion: The Kennedy Krieger Lead Paint Abatement Study. ASSIGNMENT 3 DUE To prepare: Begin by downloading the non-graded assignment sheet posted on the course BB. You can complete the non-graded assignment as you read the following: Oakes JM. “Risks and wrongs in social science research: An Evaluator’s Guide to the IRB.” Evaluation Review Vol. 26 No. 5, October 2002 443-479. Blustein J. “Toward a more public discussion of the ethics of federal social program evaluation”. Journal of Policy Analysis and Management. 2005; 24 (4), 824-846; and 851-852; and the exchange that followed. Johns Hopkins University, “Lead based paint study”. A fact sheet posted by the university on the School of Medicine website. Our BB includes the link to lots of other material that Hopkins has made available. Court of Appeals of Maryland. Erika Grimes vs. Kennedy Krieger Institute. Decision. Nos. 128, 129, September; Term 2000. August 16, 2001. Buchanan DR, Miller FJ. “Justice and fairness in the Kennedy Krieger Institute lead paint study: the ethics of public health research on less expensive, less effective interventions”. American Journal of Public Health. 2006;96:781–787. Week 11. November 18. In-class late midterm exam Week 12. November 25. Hands on Stata Session I Introduction to Stata. Demonstration of some aspects of the Stata interface not covered in the UCLA Stata material. Presentation of new material on data management; in-class work on Assignment 4. To prepare: (TBA, tentatively) Complete the UCLA Stata material Download and preview Assignment 4 2875 Syllabus – Fall 2009 8 Week 13. December 2 Hands on Stata Session II. Specifying fixed effects models in Stata. In-class work on Assignment 5. To prepare: Download and preview Assignment 5. Week 14. December 9. Hands on Stata Session III. complete all assignments. Optional in-class lab period to ASSIGNMENTS 4 and 5 due by December 18. 2875 Syllabus – Fall 2009 9