MSSP 629: Research and Evaluation Design
______________________________________________________________________________
Instructor
Course Information
Geri Summerville, MSW, LSW
MSSP 629 Research and Evaluation Design
gsum@sp2.upenn.edu
School of Social Policy and Practice, Room A19
Office Hours – Room TBD
Spring 2015
Immediately before/after class or
Mondays 12:00-2:30 PM
additional hours can be
January 14* – April 27, 2015
scheduled upon request.
(*Note: January 14th is a Wednesday.)
______________________________________________________________________________
I. Course Purpose
This course introduces program evaluation in the context of social research methods. Students learn
about design and the application of data collection skills to all phases of program/service delivery from
needs assessment to analysis of findings to implementation of changes based on results. Students learn to
appreciate how these skills can be used as practical tools for identifying problems to developing and
implementing programs including taking a reflective practice approach, ensuring equity and fairness in
program delivery (i.e., combating disparities). This applied course provides students with practical
experiences while learning to apply guidelines of evaluation and research ethics to social policy settings.
II. Educational Objectives
The course asks students to practice the conceptualization skills that inform program planning,
evaluation and research. Development of these skills enhances all areas of social policy and practice.
Educational objectives for this course are:

Students will attend to program evaluation ethics while conducting all work for this course;

Students will use an array of intellectual and professional resources for information,
including textbooks, journals, web resources, consumers and professional colleagues; and

Students will work collegially with their teams and engage in collegial critique of the work
and efforts of peers.
Specific learning objectives for this course are:
1. Describe the difference between research and program evaluation;
2. Identify the key concepts in program evaluation;
3. Discuss the component parts of program evaluation within a working framework;
4. Create a logic model linking program components;
5. Design a basic program evaluation that is practical and feasible;
6. Critique and recommend practical improvements to existing or proposed program evaluations;
7. Design a basic performance measurement system to assess program/organizational processes or
outcomes;
8. Develop critical thinking skills for reviewing and interpreting peer-reviewed publications and
grey literature.
III. Course Specifics
A. Prerequisites
This course assumes prior coursework or basic knowledge of research methods.
1
B. Expectations
Classroom learning is a group activity that depends upon everyone’s full participation in order to
succeed. Students are expected to be prepared to begin class on time; silence or turn off and put
away cell phones and other electronic devices; complete assigned readings prior to class and be
prepared to discuss content in class homework; submit assignments at the start of class; and assist
your classmates. In return, students can expect that the instructor will be on time; be prepared for
every class; be available via email between classes; have appointment times available to answer
questions and help with learning and assignments; make every class engaging and valuable; and
respect student contributions to class.
C. Course Format
The course will incorporate diverse learning activities including lectures, videos, skills training, guest
speakers, student presentations and applied discussions.
D. Required Text and Other Readings
This course will use the following text:
Wholey, J.S., Hatry, H.P., & Newcomer, K.E. (2010). Handbook of practical program evaluation
(3rd ed.). San Francisco, CA: Jossey-Bass.
Additional readings, documents, and handouts will be made available by the instructor throughout
the course.
E. Attendance and Class Participation
This class will meet on Mondays. Classroom learning is a fundamental component of your professional
education and full and active participation in all classroom exercises and activities is required to pass the
course. Attendance is, therefore, required at all classes and mandatory for a passing grade. If you cannot
attend class you are to notify your class instructor prior to the class session. Email is the desired form of
communication. If an emergency arises and you must miss class, please contact the instructor as soon as
reasonably possible. Two or more missed classes may require a make-up assignment to demonstrate the
content was reviewed and learned.
F. Academic Integrity
Students are expected to conduct themselves consistent with the University of Pennsylvania’s Code
of Academic Integrity, which presents standards regarding plagiarism, multiple submissions and other
actions. Students are expected to be familiar with the Code, which can be found at
http://www.vpul.upenn.edu/osl/acadint.html
G. Grading policy
Please see the instructor as soon as possible in the semester if you require formal accommodations
through the Office of Students with Disabilities or informal accommodations that can be arranged on a
situation-by-situation basis. Please also indicate if you anticipate having any problems completing
assignments for any reasons. The late assignment policy is stated below. Exceptions to this late penalty are
rare and up to the discretion of the course instructor. Plagiarism and other forms of student misconduct
will automatically result in an “F” in the course.
Policy on Late Assignments
All assignments will be penalized 10% of the possible points for each day of tardiness. Students should
contact the instructor at least three days prior to the due date if there are extenuating circumstances. For
such circumstances, new dates may be negotiated. Students who miss class when an assignment is
due should e-mail, fax, or otherwise deliver the assignment prior to class by the due date.
2
Midterm Assessment
The midterm assessment will be based on satisfactory course participation, completion of the logic
model, and satisfactory performance in the group exercises to that point. It is presumed that you are
in good standing and on target to receive a passing grade unless you receive an e-mail indicating
otherwise by class 5.
Final Grade
Points are given for all assignments completed outside of class. The following distribution will be
used for grading:
A+ 98 - 100
A 93 - 97
A- 90 - 92
B+ 87 - 89
B 83 - 86
B- 80 - 82
C+ 77 - 79
C 73 – 76
C- 70 - 72
D 60 – 70
F under 60
H. Course Assignments
Your course grade will be determined by the following:
In-class group data collection exercises…………………………………………25%
Evaluation and research critique.....................................................................................10%
Evaluation plan
Theory of Change and Logic Model…………………………………………15%
Advance preparation: problem statement, evaluation question, and outline.....10%
Final presentation........................................................................................................10%
Final report.......................................................................................................30%
Session &
Date
Class 1 – (Wed.)
January 14th
Class 2
January26th
Topic
Course Overview
Readings to be
Completed Prior to
Class
In-Class
Learning
Activities
Assignments
Due the Start
of Class
None
Statement of
professional
focus/interest
None
Intersection between
Research, Program
Evaluation and Policy
What is a program?
Program theory of change
Direct observation data
collection
Centers for Disease Control and
Prevention. Recommendations
and Reports. September 17,
1999 (RR11);1-40Framework for
program evaluation in public
http://www.cdc.gov/mmwr/pr
eview/mmwrhtml/rr4811a1.htm
Collecting Evaluation Data:
Direct Observation
http://Learningstore.uwex.edu/
Assets/pdfs/G3658-05.pdf
3
Observational
exercise
Watch and
analyze: Harlem
Children’s Zone
Observational
data collection
group exercise
Discuss
program choice
for final
assignment
None
Class 3
February 2nd
Class 4
February 9th
Class 5
February 16th
Class 6
February 23rd
Class 7
March 2nd
Describe/Develop your
program theory of change
and logic model
Planning the
Evaluation/Research
 Review of Literature
 Problem Statement
 Stakeholders and
audience
Methodology and
Measurement
 Experimental, Social
Experiments and
Quasi-experimental
designs
Guest Lecturer: Jackie W.
Kaye, Director of
Evaluation and Research
Wellspring Advisors
Methodology and
Measurement (continued)
 Use of quantitative vs.
qualitative
 Introduction to
interviews and focus
groups
Introduction to Survey
Research
W.K. Kellogg Foundation
(2001). Logic Model Development
Guide.
http://www.wkkf.org/~/media
/36693510092544928c454b5778
180d75/logicmodel.pdf
Review theory of
change and logic
model
assignment
Submit a paragraph
describing your
program choice.
Wholey et al. (2010)
Chapter 1 Planning and Designing
Useful Evaluations
Draft problem
statement.
Identify
stakeholders and
audience
Theory of change
and logic model
assignment due
Wholey et al., (2010) – Chapter 6
Comparison group designs; and
Chapter 7 Randomized control trials
and nonrandomized designs.
Review quasiexperimental
evaluation
critique exercise
Problem statement
due
Wholey et al., (2010) – Chapter
14 Collecting data in the field and
Chapter 17 Focus Group
Interviewing
In class
discussion of
evaluation
critique findings
Evaluation critique
1-page report due.
Additional readings will be
emailed and posted on canvas
Interview
protocol
development
exercise
Survey data
collection group
exercise
Additional readings will be
emailed and posted on canvas
Wholey et al., (2010) Chapter 12
Using surveys and Chapter 19
Qualitative data analysis
None
March 9th
NO CLASS
SPRING BREAK
Class 8
March 16th
Performance Monitoring
and Improvement
Class 9
March 23rd
Class 10
March 30th
Cost Effectiveness and
Cost/Benefit Analysis
Guest Lecturer: Lisa Gale
Reyes, Northeast Director
for Nurse Family
Partnership
Writing effective evaluation
reports and maximizing use
of results
Storytelling
Wholey et al., (2010) – Chapter 5
Performance Monitoring
Wholey et al., (2010) – Chapter
21 Cost-effectiveness and cost-benefit
analysis
Performance
improvement
exercise
Discuss
Evaluation
Question due
next week
Cost/benefit
analysis exercise
None
Evaluation question
due
Additional readings will be
emailed and posted on canvas
Wholey et al., (2010) – Chapter
25 Writing for impact.
www.issuelab.org/...labor_marke
ts_findings_from_the_sectoral_
employment_impact_study
4
In-class critique
on Tuning in to
Local Labor
Markets
Discuss outline
due next week
None
Class 11
April 6th
Collective Impact
Using Agency Data
Guest Lecturer: Meridith
Polin, Senior Consultant,
McClanahan Associates
Using Data in Multi-Agency
Collaboratiaons
http://www.issuelab.org/researc
h/using_data_in_multi_agency_
collaborations_guiding_perform
ance_to_ensure_accountability_
and_improve_programs
Collective
Impact Exercise
Additional readings will be
emailed and posted on canvas.
Class 12
April 13th
Pitfalls of Evaluations.
Review of ethics in
program evaluation and
research
Class 13
April 20th
Class 14
April 27th
Ethical Dilemmas in Program
Evaluation and Research Design
http://www.socialresearchmeth
ods.net/tutorial/Lapolt/lizhtm.h
tm
Guest Lecturer: Lauren
Ferreira Cardoso
PhD Candidate(will discuss
international research)
Final student presentations
(15 minutes/person
including Q&A)
Guiding principles for evaluators
http://www.eval.org/Publicatio
ns/GuidingPrinciplesPrintable.a
sp
None
Final student presentations
(15 minutes/person
including Q&A)
None
5
Case Study
Outline for
evaluation plan due
(including full write
up of your problem
statement, program
description,
stakeholder
overview,
evaluation question
and bullets for each
remaining section
in the proposal
(None – prepare
for final
presentation and
evaluation plan)
Learn from each
other
(None – prepare
for final
presentation and
evaluation plan)
Learn from each
other
All evaluation plans
due by Friday, May
1st.