THE UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL
SCHOOL OF SOCIAL WORK
SOWO 810: EVALUATION OF SOCIAL WORK INTERVENTIONS
Section 003
Spring Semester, 2012
Instructor
Aaron M. Thompson
Office: Room 245C
TTK Building
CB #3550,
School of Social Work, Chapel Hill, NC 27599-3550
Phone: (919) 962-9808
Email: athom@email.unc.edu
Class Meeting Times & Office Hours:
Class meets on Tuesdays, 10:30-11:50 a.m. in Room 500
Office hours are Tuesdays, 9:30 – 10:00 and 12:00 to 1:00 or by appointment
Course Description:
Students gain advanced knowledge of evaluation methods and evidence-based practices that
can be applied to the evaluation of social work interventions in their areas of practice.
Learning Objectives:
Upon completion of this course, students will be able to draw from knowledge of research
methods and evidence-based practice, and work collaboratively with supervisors, colleagues,
peers, and consumers, to design, implement, and disseminate the results of an evaluation of
an intervention, program, service or policy that may be needed by current or prospective
consumers or program or policy planners. Specifically, students will demonstrate:
1) Skills for reviewing and presenting information about the importance of a social problem
addressed by an agency, including its relation to discrimination based on culture,
economic status, gender, religion, age, or sexual orientation;
2) The ability to consider this social problem in the context of current service delivery, and
critically determine a matter requiring evaluation related to client need, the process of
service delivery, or service outcomes;
3) The ability to design an appropriate evaluation, including identifying a sample,
determining a suitable study approach; and selecting appropriate, reliable, and valid
measures (including consideration of validity for the targeted population);
1
4) The ability to plan, conduct, and synthesize appropriate qualitative or quantitative
analyses;
5) The ability to apply knowledge of social work ethics and values to the design of a
practice intervention or other types of evaluations;
6) The ability to consider the implications of results for practice and policy (including
implications related to the agency’s success in working with clients from different
groups);
7) The ability to summarize evaluation results and program improvement plans in a concise
written document; and
8) The ability to present evaluation plans and program improvement plans to appropriate
audiences.
Available Resources:
1. Royse, D., Thyer, B. A., Padgett, D. K. (2010). Program evaluation: An introduction (5th
ed). Belmont, CA: Brooks-Cole.
2. Rubin, R., & Babbie, E. R. (2010). Essential research methods for social work (2nd ed).
Belmont, CA: Brooks-Cole.
3. Hatry, H., Van Houten, T., Plantz, M. C., & Greenway, M. T. (1996). Measuring
program outcomes: A practical approach. Alexandria, VA: United Way of America.
4. On line course materials http://sakai.unc.edu
Accommodations for Students with Disabilities:
Students with disabilities, which affect their participation in the course, should notify the
instructor if they wish to have accommodations in instructional format, examination format, etc.,
considered. Accommodations and services are provided by Disability Services (Voice/TDD:
919.962.8300/966.4041). Learning Disability Services (919.962.7227) provides supportive
services for students with learning disabilities and attention-deficit/hyperactivity disorders.
Policy on Academic Dishonesty:
Academic dishonesty is cheating, contrary to the ethics of the social work profession, unfair to
your peers, and generally unacceptable. All written assignments per UNC policy must include
the following signed pledge: “I have neither given nor received unauthorized aid in preparing
this written work.”
The UNC School of Social Work APA Quick Reference Guide, available at
http://ssw.unc.edu/files/web/pdf/APA_Quick_Reference_Guide.pdf , contains basic information
on how to avoid misuse of the words and ideas of others in your written work. Students are
strongly encouraged to review the section on plagiarism carefully. All instances of academic
dishonesty will result in disciplinary measures by the School of Social Work and the University.
Teaching Methods and Accountability:
SOWO 810 is a hybrid class that combines in-person class sessions and on-line lessons. Students
are expected to meet for class as the syllabus outlines as well as complete all online activities.
Most in-class sessions will include class discussion of reading materials and critical content
issues/questions as well as skill-building activities.
2
Assignment Guidelines:
The School of Social Work faculty has adopted APA style as the preferred format for papers and
publications. The best reference is the Publication Manual of the American Psychological
Association, Sixth Edition (2010) that is available at most bookstores. The following web sites
provide additional information:
 http://juno.concordia.ca/help/howto/apa.php
 http://www.apastyle.org/apa-style-help.aspx
 http://owl.english.purdue.edu/owl/
 http://ssw.unc.edu/files/web/pdf/APA_Quick_Reference_Guide.pdf
Course Assignments:
1. Online Quizzes:
Similar to 510, quizzes will be used to assess your understanding of online content and assigned
readings. There are 2 online quizzes, one over the quantitative analysis content and the second
over the qualitative data collection and analysis content of the course. These quizzes comprise
20% of your course grade (10% each), and cannot be made up after the due dates which are
listed in the course outline.
 Students will have 1 hour to complete each quiz (10 questions)
 Quiz questions will be randomly assigned from a pool of questions
 You may re-take the quiz one additional time (your highest score will be recorded in
Sakai)
2. In Class Lab Assignments:
There will be 2 in-class labs throughout the semester. These labs are worth 30% of your overall
course grade (15% each). These labs will provide an opportunity for hands on experiences with
multiple aspects/components of the research and evaluation process.
3. Group Assignment Paper and Presentation: From Evidence to Program Improvement
Purpose: This assignment will give you the opportunity to think about how you can use
evaluation findings to improve your practice.
Products: The assignment asks each team for (1) a 7-10 page paper and (2) a 15-minute
presentation. The following is an overview of the components of the assignment, along with an
example for clarification.
Component
1.
Students will divide into teams based
on areas of practice/field placements.
Each team will identify one practice
issue. The issue can be related to
underserved groups (needs
assessment), organizational
challenges (process evaluation), or
the impact of interventions on users
(outcome evaluation). Evidence that
supports the issue is summarized
into an evaluation statement.
Example
Your team consists of students who are doing their field placements in
agencies working with adults with mental illness. You are aware of the
recent federal lawsuit filed by Disability Rights NC, alleging that the
state is violating the Americans with Disabilities Act by housing
people with mental illness in nursing and long term care homes that are
inappropriately restrictive and inconsistent with best practice
guidelines. You are interested in better understanding the experiences
of your clients who live in nursing homes, and more effectively
addressing what may be their unmet needs. Evidence of these unmet
needs may be available from agency records or previous research on
the housing needs of older adults with mental illness.
3
Component
Example
2.
Each team will design a strategy for
studying and understanding this
issue. The strategy will specify the
research question, study design,
sampling, data collection
instruments, and data collection
procedures. A description of the
above components will be included
in the Methods Section. Copies or
descriptions of the instruments will
be included or attached as
Appendices.
You decide to conduct a needs assessment. The study will be
descriptive in nature, and will include the following quantitative and
qualitative data:
 Demographics and prevalence: Using agency records, determine
the number of clients residing in nursing homes, create a
demographic profile of the group, and identify the homes in
which they live
 Quantitative data: You will find standardized instruments that
measure (a) satisfaction with housing, and (b) overall quality of
life
 Qualitative data: You will conduct qualitative interviews with
(a) a demographically representative sample of agency clients
residing in nursing homes, and (b) care staff in those homes, in
order to better understand issues and challenges from both
resident and provider perspectives
Copies of the draft data and instruments are included in the paper or
attached as appendices.
3.
Teams will generate three possible
alternatives for what might be found
in their research. For each alternative,
they will summarize possible
implications for program
improvement. A description of the
alternatives and implications will
be included in the Results and
Implications Section.
Possible alternative findings:
 Alternative 1: There is widespread dissatisfaction with the
nursing home living arrangement. Clients report generally poor
quality of life, especially in the areas of social support and access
to recreational resources. Staff report feeling unprepared to
effectively address the needs of residents with mental illness.
Implications: Training with care home staff to improve
responsiveness to residents with mental illness; agency
attention to meeting the social and recreational needs of
clients; legislative advocacy to revise policies around housing
for people with mental illness.
 Alternative 2: There is a range of perspectives reported, and
findings seem to vary by care home. Interview responses suggest
that care staff and availability of transportation make the most
difference in determining how a given client experiences his/her
life in the nursing home.
Implications: Targeted work with specific care homes;
possible relocation of clients to homes that seem to provide
superior care to individuals with mental illness.
 Alternative 3: There is a range of perspectives reported, and
findings seem to vary by age of client. Younger clients are far
more distressed with living in a nursing home than are older
clients. Interview responses suggest that isolation from peers and
the restrictiveness of rules are the most problematic aspects of
nursing home life.
Implications: Increased agency focus on peer support and
interaction for young adults with mental illness; investigation
of supportive housing alternatives for this especially
vulnerable population
Attendance and Participation:
Prompt attendance is expected because we have a reduced number of in-class sessions. If you
will not be able to attend a class, let the instructor know as soon as possible. It is your
responsibility to obtain handouts, information about class content, and information about
4
announcements, etc., from your classmates if you are unable to attend a class. Students with
more than two absences will receive an “L” unless they have made prior arrangements with the
instructor. In order to fully participate in and benefit from each class session, students should
complete required readings before class and come to class prepared to discuss them.
Course Grading and Performance Assessment:
Grading:
Assignment
Quizzes
Labs
Evaluation Paper
Presentation
Course Total
%
20%
30%
40%
10%
100%
Scale:
Points
<69
70 - 79
80 - 93
94 - 100
Grad
F
L
P
H
Late Assignments:
Late assignments are strongly discouraged. In case of a dire, life-threatening emergency, a late
assignment may be accepted at the discretion of the instructor. If permission for late submission
is not granted before breaking a deadline, the grade will be reduced 10%, and another 10%
reduction will occur each day (including weekends) that the assignment is late.
Please note that technical difficulties are not an acceptable excuse for turning in an assignment or
quiz late. All technical inquiries should be directed to the staff of the SSW Computing
Information and Technology Unit (CITU). Please contact Manuel Garcia at 919.962.6416 or
megarcia@email.unc.edu for assistance.
Electronic Devices:
Please set your cell phones to vibrate. Use of laptops is permitted as a tool for small group
activities and in-class assignments or as an approved accommodation for students with
disabilities.
5
COURSE OUTLINE AND ASSIGNMENTS
Week 1 (Jan 10, 2012): Introduction to Course and Assignments (in-class session)
Goal: We will review the procedures for the online course lessons, principles of evidence-based
practice (EBP) and authority-based practice (ABP), and understand the course assignments.
Assignments:
 Review Sakai Lesson “Introduction to Evidence-Based Practice” (no quiz required)
 Email instructor your area of interest
Relevant Readings:
Rubin & Babbie: Chapter 2
Royse et al.: Chapter 1
Hatry et al.: Step 1
Week 2 (Jan 17, 2012): Evidence Based Practice Methods Review (in-class session)
Goal: By the end of this lesson, students will understand how to conduct a needs, process, and
outcomes assessment of their organization’s target population.
Assignments:
 Review Sakai Lessons:
1. Orientation and Strategies for Needs Assessments
2. Formative and Process Evaluation Assessment
3. Outcome and Impact Evaluation
4. Designing an Evaluation
 Group assignments will be made by the instructor
Relevant Readings to Review:
Royse et al.: Chapters 3, 5, 7, 9 and 10
Week 3 (Jan 24, 2012): Quantitative Data Analysis-1 (no in-class session)
Goal: By the end of the lesson students will be able to define how statistical significance impacts
research and practice. Students will be able to code data in a meaningful way, interpret
descriptive statistics, and describe univariate analysis.
Assignment:
 Review Sakai Lesson:
1. Measurement and Questionnaire Construction
2. Quantitative Data Analysis (Part I)
Readings:
Rubin & Babbie: Chapter 17 (pp. 290-293)
Royce et al. Chapter 14 (pp. 341-351 & 362-370)
6
Week 4 (Jan 31, 2012): Quantitative Data Analysis-2 (in-class session)
Goal: By the end of the session, students will be able to describe multivariate analysis. Students
will also understand how statistical results from research should impact social work practice.
Assignments:
 Review Sakai Lesson:
1. Quantitative Data Analysis (Part II)
 Due Friday, Feb 3 Online Quiz over Saki Lessons “Quantitative Data Analysis (Part I)”
AND “Quantitative Data Analysis (Part II)”
Readings:
Rubin & Babbie: Chapter 17 (pp. 293-301)
Royse et al. Chapter 14 (pp. 351-362)
Hatry et al.: Step 6 (pp. 113-124)
Week 5 (Feb 7, 2012): Quantitative Data Analysis-3 (Groups 1, 2, & 3 in-class session)
Goal: By the end of the lesson students will be able to run basic statistical tests of difference and
association using Microsoft Excel, graphically present findings, and interpret the results.
Assignment:
 In class lab using Excel to conduct descriptive statistical analyses
 Everyone submit Evaluation Outline Worksheet # 1: Introduction and Literature Review
 Due Fri, Feb 10th for Groups 1, 2, 3—In class lab to conduct inferential statistical analyses.
Readings:
Appropriate readings from weeks 3 & 4
Week 6 (Feb 14, 2012): Quantitative Data Analysis-4 (Groups 4, 5, & 6 in-class session)
Goal: By the end of the lesson students will be able to run basic statistical tests of difference and
association using Microsoft Excel, graphically present findings, and interpret the results.
Assignment:
 Due Fri, Feb 17th for Group 4, 5, 6—In class lab to conduct inferential statistical analyses.
Readings:
Appropriate readings from weeks 3 & 4
Week 7 (Feb 21, 2012): Qualitative Data Collection Methods-1 (no in-class session)
Goal: By the end of this lesson, students will be able to understand how to conduct an evaluation
using qualitative methods, including in-person interviews, focus groups and participant
observations.
Assignment:
 Review Sakai Lesson
7

1. Qualitative Evaluation – Methods and Data Collection
Email Lab Exercise worksheet
Readings:
Royse et al.: Chapter 4 (pp. 83-104)
Rubin & Babbie: Chapters 7 (pp. 102-110) and 13 (pp. 218-233)
National Science Foundation. (1997). Overview of qualitative methods and analytic techniques.
User-Friendly Handbook for Mixed Method Evaluations, Chapter 3. Retrieved November 4,
2009 from, http://www.nsf.gov/pubs/1997/nsf97153/chap_3.htm
Week 8 (Feb 28, 12-12): Qualitative Data Analysis-2 (in-class session)
Goal: By the end of this session, students will be able to design, conduct and analyze qualitative
interview data.
Assignments:
 Review Sakai Lesson:
1. Qualitative Analysis
 Due Friday, Mar 2 Online Quiz over Sakai lessons “Qualitative Evaluation – Methods
and Data Collection” AND “Qualitative Analysis”
Readings:
Rubin & Babbie: Chapter 18
Jackson, K. F. (2012). Living the multiracial experience: Shifting racial expressions, resisting
race, and seeking community. Qualitative Social Work: Research and Practice, 11, 42-60.
Charles, P., Orthner, D.K., Jones, A. & Mancini, D. (2006). Poverty and couple relationships:
Implications for welfare policy. Marriage and Family Review, 39, 27-52
National Science Foundation (1997). Analyzing qualitative data. User friendly handbook for
mixed-methods evaluations. Chapter 4. Retrieved November 4, 2009 from,
http://www.nsf.gov/pubs/1997/nsf97153/chap_4.htm
Mar 2 – Mar 9, 2012: Spring Break (no class)
Week 9 (Mar 13, 2012): Qualitative Data Analysis-3 (in-class session)
Goal: By the end of this session, students will be able to design questions and collect data using
qualitative interviewing.
Assignment:
 In class lab (Part I) using in-person interviews and recording procedures. Lab Parts I & II
due by 9:00am on Tuesday, March 27th. Please send via e-mail.
 Submit Evaluation Paper Outline Worksheet #2: Research Questions and Methods
Readings:
None
8
Week 10 (Mar 20, 2012): Qualitative Analysis-4 (in-class session)
Goal: By the end of the lesson, students will be able to code and interpret qualitative interview
data as well as summarize results for reporting.
Assignment:
 In class lab (Part II) using qualitative interview data for coding and data analysis. Lab
Parts I & II due by 9:00am on Tuesday, March 27th. Please send via e-mail.
Readings:
None
Week 11 (Mar 27, 2012) : Group Work Sessions (meet w/ group—consultation available)
Goal: By the end of this week, students will have summarized their data collection and analysis
strategies and begun to propose practice solutions based on their alternative analyses.
Assignment:
 Small group work during and outside of class
Readings:
Appropriate to the students papers
Week 12 (Apr 3, 2012): Group Work Sessions (meet w/ group—consultation available)
Goal: By the end of this week, students will proposed practice solutions based on their
alternative analyses, prepared a summary report, and begun work on their class presentations.
Assignment:
 Small group work during and outside of class
Readings:
Appropriate to the students papers
Week 13 (Apr 10, 2012): Student Presentations (in-class session)
Goal: By the end of this session students will effectively present program improvement
proposals to their peers and describe the potential impact on social practice. Students will also
practice providing feedback to peers on ideas for improving evaluation and practice designs.
Assignments:
 Final Due: Evaluation Paper and Presentation
 Student Presentations
 Peer feedback
 Course evaluation
9
Evaluation Paper Outline
Worksheet #1: Introduction and Literature Review
Introduction
State the problem that to be addressed by your evaluation.
What is the prevalence of the problem (e.g., the number of people in a population who have the
problem)? Identify 1-2 sources that discuss the prevalence of the problem
What are the costs (social and financial) of the problem? Identify 1-2 sources that discuss the
social and financial costs of the problem.
Literature Review
Identify three peer-reviewed journal articles that you will be using in your literature review and
briefly summarize the findings of each article.
Article #1 Reference & Summary:
Article #2 Reference & Summary:
Article #3 Reference & Summary:
10
Evaluation Paper Outline
Worksheet #2: Research Questions and Methods
Research Question(s)
List the specific and measurable research question(s) that will guide your evaluation. State your
hypothesis for each research question.
Methods
Evaluation Design: Identify the method (e.g., needs assessment, process evaluation, outcome
evaluation) you plan to use and briefly describe the design.
Study sample: Briefly describe the sample population and specific sampling strategy you plan to
use.
Participant recruitment: Briefly describe how you will identify and recruit your sample
participants (or obtain existing records).
Data collection procedures: Briefly describe the strategy you will use to collect the data (e.g.,
focus groups, interviews, record abstraction) and the procedures to be used:
Measures: Briefly describe the measures you plan to use.
11
Evaluation Paper and Presentation
For this paper, students will develop a proposal for an evaluation project related to a specific
social work interest. Each group of students (4 – 5 students per group) will present a summary of
their evaluation to the class in 15 minutes. This 7-10 page double-spaced paper and brief class
presentation is worth 50% of your overall grade.
The paper should include the following seven components:
1. Introduction: (1/2 – 1 page) The introduction should include a statement of the problem
to be addressed by your evaluation and information concerning the prevalence (e.g., the
number of people in a population who have the problem), and costs (social and financial)
of the problem. This section should conclude with a discussion of why the problem is
important to the field of social work.
2. Literature Review: (1 - 2 pages) Empirical research relating to relevant interventions,
evaluations, programs or policies and their outcomes specific to your population and
problem of interest should be reviewed concisely and critically in this section. Also, in
one or two sentences describe your plan for addressing the problem with an evaluation or
intervention.
3. Research Question(s): (1/2 page) List the specific and measurable research question(s)
that will guide your evaluation. In addition, please state your hypothesis for each
question.
4. Methods: (2 - 3 pages) This section will present the plan for how the research question(s)
will be addressed. Please provide separate subheadings for each of the following:





Evaluation design: Identify the method (e.g., needs assessment, process
evaluation, outcome evaluation) and describe the design.
Study sample: Describe the sample population and specific sampling strategy.
Participant recruitment: Describe how you will identify and recruit your sample
participants (or obtain existing records).
Data collection procedures: Describe the strategy you will use to collect the data
(e.g., focus groups, interviews, record abstraction) and the procedures to be used.
Measures: Describe the measures and be sure to include information about the
reliability and validity of each (if available). Please attach the actual survey(s) or
instrument(s) as an appendix if you have access to them, otherwise the types of
questions included.
5. Results and Implications: (2 - 3 pages) Discuss three possible alternatives for what
might be found from your evaluation results. For each alternative, summarize possible
implications for program improvement (see syllabus for examples).
12
6. Conclusion: (1 page) Discuss the strengths and limitations of your proposed evaluation.
Discuss suggestions for further evaluation (e.g., given the limitations of your initial
design, how the next evaluation can be improved).
7. References & Writing: Provide an accurate list of all cited sources in APA format
(minimum of 5 peer-reviewed journal articles). The paper should be clearly and concisely
written and checked for errors prior to submission.
Grading Rubric: Required Components and Points
Possible
Introduction
3
Literature Review
5
Research Question(s)
4
Methods
10
Results and Implications
10
References and Writing
5
Conclusion
3
Class Presentation
10
Total Possible Points
50
13