NEW YORK UNIVERSITY - ROBERT F. WAGNER GRADUATE SCHOOL OF PUBLIC SERVICE,
PADM-GP 2171/GPH-GU 2371 Sec3
Program Analysis and Evaluation -- Spring 2016
Wednesdays 6:45-8:25 pm; 19 West 4th St Rm 102
Adjunct Prof Jodi F Paroff
To contact instructor: Jodi.Paroff@nyu.edu, Phone: (973) 934-7214
Office Hrs: Wed 4:30-6:30pm, by appointment, Puck Building Adjunct Offices (Please request appts
via email in advance.)
Course Pre-requisites: You must have completed (or waived) P11.1011 (Statistical Methods) and
P11.1022 (Introduction to Public Policy). This course builds on these introductory courses and lays
the foundation for P11.2875 (Eval of Health & Social Programs).
Course Description and Objectives: Program evaluation is a critical part of designing and operating
effective programs. Evaluations supply information to program managers and policymakers that
can assist them in making decisions about which programs to fund, modify, expand or discontinue.
Evaluation can be an accountability tool for program managers and funders. This course serves as
an introduction to evaluation methodology and evaluation tools commonly used to assess publicly
funded programs. Students taking this course can expect to
 become familiar with the concepts, methods and applications of evaluation results;
 learn how to read evaluation research critically;
 understand how to use evaluation results to improve program performance; and
 be able to propose an appropriate evaluation plan to assess the implementation and
effectiveness of a program.
Course Structure: The class includes lecture, readings, and discussion. There is no specific policy
or sector focus to this course, as evaluation tools are used in all policy areas and by public
(government) and private (foundation) funders as well as by public and private sector program
managers. You are encouraged to relate the general material of the course to your specific policy
interests.
Readings: Required textbook: Carol H. Weiss (1998) Evaluation: Methods for Studying Programs
& Policies 2nd edition. Prentice Hall
Optional and recommended text is: Peter Rossi, Howard Freeman, and Mark Lipsey (2004)
Evaluation: A Systematic Approach, 7th ed. Sage Publications. (abbreviated in syllabus as “RFL”)
In addition to the required text, you will need to read a few chapters from optional textbooks and
many journal articles. All of the articles are available through Bobst electronic journals and I will
make them available on NYU Classes. The chapter readings are available on NYU Classes.
There is a sizable body of literature which deals with program evaluation and policy analysis. The
journal Evaluation Review (previously Evaluation Quarterly) is an especially rich source on the
subject, as is the Evaluation Studies Review Annual (Sage, more or less annually). Evaluation
Practice, Evaluation and Program Planning, New Directions for Program Evaluation, and Journal of
Policy Analysis and Management are also recommended. There are also evaluation journals for
specific fields, including Evaluation and the Health Professions, Evaluation in Education, and
Evaluation and Human Services.
Course requirements: Class preparation and participation are important for this course. You need
to read required text and articles in advance and be prepared to participate in class discussion. In
addition to class participation, you will: write 2 brief memos, complete 1 take-home exam,
facilitate at least 1 small group discussion, create a poster with your group for peer review, and
write a final evaluation design paper. It is expected that you abide by NYU’s honor code and
uphold the university’s standards for academic honesty in all of your work. See
https://wagner.nyu.edu/students/policies/academic-code.
Attendance: No grades are given for attendance, but your regular participation in reading
critiques is factored into your class participation grade. You are responsible for any missing
information and meeting all assignment deadlines during your absence.
Program Statement Memo, due Wed Feb 24th
Student groups will submit a short (2 - 3 pp) description of a selected program, indicating the
problem/need addressed by the intervention, the intended beneficiaries of the program, the
intended benefits, and the causal model/program theory underlying the program. This memo is a
preliminary step in writing the final design paper.
Midterm Examination: Take-home essay exam due Mon Mar 21st
Measurement Memo, due Wed Apr 6th
Using the program model developed in the first memo, student groups will specify the research
questions, operational definitions, and specific measures they would use in an evaluation of the
program.
Poster Sessions & Feedback due Wed Apr 20th
Each group will create a poster that presents the evaluation design proposal of their final
paper/project to be submitted for peer critique. Posters will include program theory and
descriptions and outcome measures from earlier assignments, as well as design rationale, and data
collection strategies. Clarity, application of concepts discussed in class, and the rationale behind
your evaluation design choices will be emphasized.
OPTIONAL Evaluation Review (for 4 pts extra credit) due Wed May 4th
It is important to become a good evaluator and a good consumer of evaluations. Review one of
three selected evaluation articles. In 2 - 3 pages, summarize the type of evaluation described, its
design and methods, and write a critique of the evaluation.
Final Paper: Impact Evaluation Design, due Mon May 9th
The final paper builds on earlier assignments. Student groups will design a comprehensive
evaluation plan/design for a chosen program. The design proposal will focus on outcome or
impact evaluation, but will also include a section on process evaluation.
2
Relative Weight of Assignments
Midterm Exam
35%
Final Paper
35%
Two memos & Poster presentation 20%
Class Participation
10%
Note: the above descriptions are not enough to complete assignments adequately. More detailed
instructions will follow.
SCHEDULE OF COURSE TOPICS, MEETINGS, READINGS, AND ASSIGNMENTS
DATE & THEMES
READINGS & ASSIGNMENTS
Pre-Course Reading  Guiding Principles for Evaluators. American Evaluation Association.

Rogers, P. (2012). Introduction to Impact Evaluation. Read to page 5.

Weiss Chapter 1, and Chapter 5 (pp. 100-108).

“Michael Scriven on the Differences between Evaluation and Social Science
Research” (www.hfrp.org).

Optional: Preskill, H. and N Jones. (2009). A Practical Guide for Engaging
Stakeholders in Evaluation. Robert Wood Johnson.
1/27/16-CLASS 1

Course Overview,
Evaluation Purpose, 
Engaging
Stakeholders

2/3/16-CLASS 2
Ethical Issues in
Program
Evaluation, Needs
Assessment
Markiewicz, A. (2005). A balancing act: Resolving multiple stakeholder
interests in program evaluation. (4) No. 1 & 2. March/April.
Oakes, J.M. (2002). Risks and wrongs in social science research: an
evaluator’s guide to the IRB. (only pp. 443-454)
Optional: Sieber, J.E. and M.B. Tolich (2013). Planning Ethically Responsible
Research. pp. 77-92.

Weiss Chapter 4, pp. 72-74

Watkins, R and J Kavale (2014). Needs: Defining What You Are Assessing.
New Directions for Evaluation, 144, 19-31.

Review: CommonNeedsAssessmentMethods.pptx (see NYU Classes)

Collier AF, Munger, M, & YK Moua. (2012) Hmong mental health needs
assessment: a community-based partnership in a small mid-western
community. American Journal of Comm Psychology, 49(1):73–86. (CS)

Lascher, S. et al. (2013). Government Public Housing Health Needs
Assessment: Focus on Race, Ethnicity, and the Older Adult.

http://india.blogs.nytimes.com/2012/07/22/mapping-toilets-in-a-mumbaislum-yields-unexpected-results/?emc=eta1
Optional Readings:

Engle, M. and J Altschuld (2014). Needs Assessment: The Perspective From
the Public Sector. New Directions for Evaluation, 144, 33-45. (First few
pages offer excellent context).
3
DATE & THEMES
READINGS & ASSIGNMENTS
 O’Toole, J. (2013). Closing the Gap: Needs Assessment of Medical Students
& Handoff Training.

Berberet, HM. (2006). Putting the pieces together for queer youth.

Weiss Chapter 2 (Purposes of eval) & Chap 3 (Program Theory)

Chen, Wang & Lin (1997). Evaluating the process and outcome of a
garbage reduction program in Taiwan. (CS)

Knowlton & Phillips (2013). The Logic Model Guidebook. Chapter 1.

Optional: Proscio, T. (2000). In Other Words. Edna McConnell Clark
Foundation. New York, NY.

Optional: RFL Chapter 5

Optional: Kaplan, SA. and Garrett, KE. (2005). The use of logic models by
community-based initiatives.
2/17/16-CLASS 4

Process Evaluation,
Program

monitoring, Data
Collection methods
Mye, SC and Moracco, KE. (2015). Compassion, pleasantry, and hope: A
process evaluation of a volunteer-based nonprofit. (CS)
2/10/16-CLASS 3
Teasing out
Program Theory
2/24/16-CLASS 5
Impact Evaluation
I: Evaluation
Design, Internal
and External
Validity,
3/2/16-CLASS 6
Randomized
Experimental
Designs
Curran, Gittelsohn, Anliker, et al (2005). Process evaluation of a storebased environmental obesity intervention on two American Indian
reservations.

Weiss Chapter 11 (Qualitative Methods)

Optional: Heinz & Grant (2003). A process evaluation of a parenting group
for parents with intellectual disabilities.

Optional: Giorgio, Kantor, Levine, & Arons (2013). Using chat and text
technologies to answer sexual and reproductive health questions: Planned
Parenthood pilot study. Journal of Medical Internet Research.

Weiss Chapter 8 (Design of evaluation, pp 180-214)

Ester Duflo’s TED talk:
http://www.ted.com/talks/esther_duflo_social_experiments_to_fight_poverty.html
▪
Edna McConnell Clark Foundation. Levels of Evidence:
http://www.emcf.org/fileadmin/media/PDFs/emcf_levelsofeffectiveness.pdf
▪
DUE: Program Memo

Weiss Chapter 9

Seron, C., Ryzin, G.V., Frankel, M., & Kovath, J. (2001). The impact of legal
counsel on outcomes for poor tenants in NYC’s housing court: results of a
randomized experiment. (CS)

Comparing experimental, quasi-experimental and non-experimental
designs: http://go.worldbank.org/7M4NUSKE10
4
DATE & THEMES
READINGS & ASSIGNMENTS
 Optional: Robertson, A., St. Lawrence, J., & Morse, D.T. et al (2011).
Comparison of Health Education and STD Risk Reduction Intervention for
Incarcerated Adolescent Females.

Optional: RFL Chapter 8
▪
MIDTERM question pool released for your review.
3/9/16-CLASS 7

Quasi-experimental 
Design, Comparison

Groups
RFL Chapter 9 pp 265-286
Moore, K. (2008). Quasi-Experimental Designs. Child Trends.
Smith, G and J Pell (2003). Parachute Use to prevent death and major
trauma. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/

Optional: Cumberland, P., Edwards, T., et al. (2008). Community level
treatment and trachoma prevalence in Ethiopia.

Optional: Nakamura, R., et al (2014). Impact on alcohol purchasing of a
ban on multi-buy promotions. 109(4):558-567.
http://onlinelibrary.wiley.com/doi/10.1111/add.12419/abstract

Optional: McMillan, J. (2007). Randomized Field Trials and Internal
Validity: Not so Fast My Friend.

Actual MIDTERM questions released after class.
3/16/16
SPRING BREAK – NO CLASS
3/21/16-MONDAY

MIDTERMS due.
3/23/16-CLASS 8
Formulating
Evaluation
Questions &
Measurement

Weiss, Chapter 6 (Measures), Chap 4 (Planning) pp. 74-82.

Engle & Schutt (2014). Chap 4 (Conceptualization & Measurement, skip pp
85-86 on cutoff scores)

http://www.nytimes.com/2011/09/23/opinion/why-cyberbullyingrhetoric-misses-the-mark.html

Online lecture: LevelofMeasurement. pptx (see NYU Classes)

Optional: Litwin, M.S. (2003). How to Assess and Interpret Survey
Psychometrics, Ch 2 (Reliability) & 3 (Validity)

Optional: Chandrika Ismail A., et al. (2009). A model of substance abuse
risk: Adapting to the Sri Lankan context.

Lester, P. et al. (2012). Evaluation of a family-centered prevention
intervention for military children and families facing wartime deployments.
(CS)

Fisher-Maltese, C and T. Zimmerman. (2015). A Garden-Based Approach to
Teaching Life Science Produces Shifts in Students’ Attitudes toward the
Environment.
3/30/16-CLASS 9
Measurement
(continued)
5
DATE & THEMES
4/6/16-CLASS 10
Sampling
4/13/16-CLASS 11
Strengthening
Evaluation Designs
4/20/16-CLASS 12
Ethical Evaluation
(continued)
READINGS & ASSIGNMENTS
 Optional: Dufrene, R. (2000). An evaluation of a patient satisfaction survey:
validity and reliability.

Optional: Fisher, A. Designing HIV/AIDS Intervention Studies. Chapter 6
(Operational Definitions) See:
http://www.popcouncil.org/pdfs/horizons/orhivaidshndbk.pdf

Babbie, E. (2013). The Practice of Social Research, Chapter 7.

Patton, MQ. (1990). “Purposeful Sampling” in Qualitative evaluation and
research methods. (pp. 169-186). Beverly Hills, CA: Sage.

Weiss Chapter 7, pp. 163-166.
▪

Optional: Taylor-Powell, E. (1998). Sampling. University of WisconsinExtension.
DUE: Measurement memo.

Weiss, review Chapter 8 pp. 191-199

RFL Chapter 9 pp. 289-295

Pettifor, A., Taylor, E., Nku, D., Duvall, S., Tabala, M., Mwandagalirwa, K.,
Meshnick, S., & Behets, F. (2009). Free distribution of insecticide treated
bed nets to pregnant women in Kinshasa. (CS)

Ballart, X. & Riba, C. (1995), Impact of legislation requiring moped and
motorbike riders to wear helmets.

Optional: Potter, S.J. (2012) Using a multimedia social marketing campaign
to increase active bystanders on the college campus. Journal of American
College Health, 60(4): 282-295.
Weiss, Chapter 14 (Ethics)


Bluestein, J. (2005). Toward a more public discussion of the ethics of
federal social program evaluation. (pp. 823-840)

Oakes, J.M. (2002). Risks and wrongs in social science research: an
evaluator’s guide to the IRB. (pp. 460-467)
4/27/16-CLASS 13

Student Poster Session – mandatory attendance
5/4/16-CLASS 14
Meta-Evaluation

Weiss Chapter 10, pp 235 – 244 (good design)

Head, K. et al. (2013). Efficacy of text messaging-based interventions for
health promotion: A meta-analysis. Social Science & Medicine 97, pp 4148. (CS)

Goldacre, Ben. (2013). Healthcare’s Trick Coin. The New York Times.
http://www.nytimes.com/2013/02/02/opinion/health-cares-trickcoin.html

Gansle, K.A. (2005). The effectiveness of school-based anger interventions
6
DATE & THEMES
5/9/16-MONDAY
READINGS & ASSIGNMENTS
and programs: a meta-analysis.

Optional: Independent Evaluation Group. (2010). What Can We Learn
from Nutrition Impact Evaluations? A Review of Interventions to Reduce
Child Malnutrition in Developing Countries. Washington, DC: World Bank.

Optional: Visher, C.A., Winterfield, L., & Coggeshall, M.B. (2005). Exoffender employment programs and recidivism: a meta-analysis.

DUE: Optional Evaluation Review Memo.
NO CLASS - Team Ratings & Final Evaluation Proposals DUE
7