THE UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL SCHOOL OF SOCIAL WORK COURSE NUMBER: SOWO 810 COURSE TITLE: EVALUATION OF SOCIAL WORK INTERVENTIONS SEMESTER: Spring 2011 (Tuesday, 10:30 – 11:50 AM, RM TTK 135) PROFESSOR: Gary L. Bowen, Ph.D., ACSW Kenan Distinguished Professor School of Social Work 325 Pittsboro Street, Chapel Hill, NC 27599‐3550 Office Phone: 919‐962‐6542 Home Phone: 919‐967‐3196 Mobile: 919‐448‐4058 Email: glbowen@email.unc.edu OFFICE HOURS: Tuesday (12:00 – 1:00 p.m.)/By Appointment UNC‐CH Safe Zone COURSE DESCRIPTION: Students apply knowledge of research methods and evidence‐based practice to the evaluation of social work interventions by developing and conducting a detailed evaluation of a specific social work intervention. OBJECTIVES: Upon completion of this course, students will be able to draw from knowledge of research methods and evidence‐based practice, and work effectively with supervisors, colleagues, peers, and consumers, to design, conduct, and disseminate the results of an evaluation of an intervention, program, service or policy implemented by their placement agency or needed by current or prospective consumers or program or policy planners. Specifically, students will demonstrate: 1) Skills for reviewing and presenting information about the importance of a social problem addressed by their agency, including its relation to discrimination based on culture, economic status, gender, religion, age, or sexual orientation; 2) The ability to consider this social problem in the context of current service delivery, and critically determine a matter requiring evaluation related to client need, the process of service delivery, or service outcomes; 1
3) The ability to design an appropriate evaluation, including describing a logic model, identifying a sample, determining a suitable study approach; and selecting appropriate, reliable, and valid measures (including consideration of validity for the targeted population); 4) The ability to plan, conduct, and synthesize appropriate qualitative or quantitative analyses; 5) The ability to apply knowledge of social work ethics and values to the design of practice intervention or other types of evaluations; 6) The ability to consider the implications of results for practice and policy (including implications related to the agency’s success in working with clients from different groups); and 7) The ability to summarize all of this material in a concise written document. CLASSES: January 11, January 18 (no class), January 25, February 1, February 8, February 15, February 22, March 1, March 8 (no class, Spring Break), March 15, March 22, March 29, April 5, April 12, April 19, April 26 MAIN TEXTS:1 Royse, D.; Thyer, B. A.; Padgett, D. K. (2010) Program evaluation: An introduction (5th ed.). Belmont, CA: Wadsworth. Rubin, A. & Babbie, E. R. (2010). Essential research methods for social work (2nd ed.). Belmont, CA: Brooks/Cole. Hatry, H., van Houten, T., Plantz, M.C. & Greenway, M.T. (1996). Measuring program outcomes: A practical approach. Alexandria, VA: United Way of America. TEACHING METHODS: This course has been designed to maximize student involvement, and it will be facilitated using a transformative learning model. From this model, students work with the instructor as full partners in assuming responsibility for the success of the course. BAD WEATHER POLICY: Please check your email by 7:00 a.m. on the day of class in case of snow, ice or other threatening and/or unsafe conditions. Use your best judgment about travel safety if you are driving to Chapel Hill from surrounding areas with inclement weather or threatening and/or unsafe conditions. If you have any questions or concerns, please contact me. 1
These are SOWO 510 texts.
2
RULES OF ENGAGEMENT: Class Attendance Students are expected to attend all class sessions, and classes will begin and end on time. If there is some reason that you cannot attend a class, please contact the instructor or leave a message for the instructor at the School of Social Work (919‐962‐6542). Students who miss two class sessions will be penalized by one letter grade (special exceptions may apply). Students who miss three or more class sessions will receive an "F" as their final grade for the class. Email Accounts All students are required to have a valid UNC email account. A valid UNC email address has the following extension: @email.unc.edu or @ unc.edu. Required Reading To facilitate class involvement and discussion, students are expected to read all required materials prior to class. Class Participation Students are expected to contribute meaningfully to class discussion. At the beginning of each class session, time will be allocated to address questions about readings and assignments. APA Format All written assignments should conform to the style guidelines of the American Psychological Association (6th ed.). If you do not have a copy of this manual, I would suggest that you buy one. American Psychological Association (2009). Publication Manual of the American Psychological Association (6th ed.). Washington, DC. Author. [www.apa.org/books/] Softcover: $28.95) SSW Writing Support Team School’s Writing Resources Page: http://ssw.unc.edu/students/writing. This link provides access to an array of resources including the newly updated “APA Quick Reference Guide.” ASSIGNMENTS: The student’s primary assignment in this course is an evaluation of an intervention, program, service, or policy related to an important social problem that is addressed by the student’s practice setting. Students are expected to evaluate consumer/program needs or the process and outcomes of practice in their field placement, in the form of a needs, process, outcome, or other type of assessment. This semester‐long evaluation will include: • a focused literature review of the problem being addressed, • formulation of the practice or policy related question(s), • development of the evaluation design and sampling strategy, including selection of 3
measures, data collection, data analysis, writing a final evaluation report, and presenting the report. •
•
•
•
The final project will be a written evaluation report using the format presented by Royse, Thyer, & Padgett (Chapter 15); it will include an executive summary, introduction, review of the literature, methodology, results, discussion, references, and appendices. Phase Assignments To ensure that the project is started promptly and can be completed during the semester, students will complete five phased assignments. These assignments can be used to develop the final evaluation report. Please bring two copies of each phase assignment to class. One copy will be peer reviewed by another student in class. Phase I – Description of social problem In no more than two double‐spaced pages, briefly describe the practice or policy related problem you will address and its importance. Include at least three citations. If you plan to evaluate an intervention, provide a brief description of the intervention or reference a treatment manual. In addition, clearly state your draft evaluation questions. Length: Maximum 1‐2 pages Due: Week 4 Value: 6% Phase II –Justification of evaluation In no more than two double‐spaced pages, write your revised evaluation questions and a brief description and justification of the evaluation you plan to conduct, include ethical considerations as appropriate, and evidence that suggestions for changes in service delivery are possible if indicated by the results. Include at least three citations. Length: Maximum 2 pages Due: Week 5 Value: 6% Note for Phases I and II: The final paper will include a literature review with at least six scholarly citations that are pertinent to the evaluation. Pay special attention to the literature on relevant evidence‐based practices in the area of study. Phase III – Evaluation design In no more than one double‐spaced page, describe your evaluation design. Indicate the type of evaluation you will implement (e.g., needs assessment, formative evaluation, process evaluation, outcome evaluation), your research design, and the strategy you will use to collect the data (e.g., focus groups, interviews, chart abstraction). Make sure to include your targeted sample size and how you will identify and recruit your sample participants (or obtain existing records). Length: Maximum 1 page Due: Week 6 Value: 6% 4
Phase IV ‐ Draft instrument Obtain/develop a draft of the data collection instrument you propose to use; this may be an existing instrument, one your create, or a combination of the two. Length: As needed Due: Week 8 Value: 6% Phase V‐ Draft data analysis plan In no more than two double‐spaced pages, create a data analysis plan that includes your research question(s) and summarizes the data that will be available. Specify your analysis strategy and which technique(s) you will use to answer your research question(s) (e.g., t‐test for independent groups, ANOVA, correlations, interpretation strategy for qualitative data). Length: Maximum 2 pages Due: Week 10 Value: 6% Final Paper/Program Evaluation Report The primary written assignment for this course will be an evaluation of an intervention, program, service, or policy related to an important social problem that is addressed by the student’s practice setting. It will incorporate the five phased assignments as well as the results and interpretation of the findings. It will include an executive summary, introduction, review of the literature, methodology, results, discussion, references, and appendices. The paper is expected to require 10‐12 pages, excluding the executive summary, references, tables/figures, and appendices. Its organization is as follows, with approximate page allocations indicated in parentheses: Executive Summary: (1 page) – This one page summary will incorporate the introduction through the discussion/interpretation as described below. Introduction (1‐2 pages) – The introduction should include a statement of the problem to be addressed and its overall impact, such as evidence related to its prevalence (the number of people in a population who have the problem), incidence (the number of new cases over a period of time), and/or effect (the importance of the problem). This section must explain why it is an important issue for social work practitioners to address and ethical considerations for this evaluation, and end with an explication of the research questions. Relevance (Literature Review) (1‐2 pages) – Previous empirical research pertaining to relevant interventions, outcomes, or other matters specific to your population and problem of interest should be reviewed concisely and critically in this section. Methods (2 pages) – This section has the following four subsections: (1) Design, (2) Sample, (3) Measures, and (4) Data analysis. If you are conducting an outcome evaluation, describe the intervention in this section (see Thyer, 1991). Provide separate subheadings for each section. Results (2‐3 pages) – This section will describe the results of your evaluation. If quantitative methods are used, descriptive statistics should be presented first and inferential statistics presented last. If qualitative methods are used, a description of the analysis themes is to be provided with illustrative narrative comments included as 5
examples. Tables and figures should be used as appropriate, and are not included in the page count. Discussion/Interpretation (3 pages) – This section should include the following, in this order: (1) a summary of your findings (e.g., related to the success of treatment, the discovery of new client needs, or whether staff are operating with a consistent program model); (2) a discussion of the implications of these findings and related recommendations (e.g., to change service delivery in a focused way to improve outcomes, better reach new clients, or assure consistency in service provision); (3) plausible alternative explanations for your findings; (4) a discussion of the limitations of your evaluation; and (5) suggestions for further evaluation (e.g., given the limitations of your initial design, how the next evaluation can be improved). This section should conclude with a discussion of the importance of your findings to social work practice and research. References – Provide an accurate list of all cited sources. Tables, Figures – Tables and figures are suggested to display you results. Typically, the first table describes the sample. Appendix – Include copies of any scales or questionnaires used in your program evaluation and other relevant information. The complete evaluation report should follow APA formatting, and should follow the structure and suggestions presented in Royse, Thyer, & Padgett, Chapter 15. Length: Maximum 10‐12 pages Due: Exam Period Value: 50% Oral presentation: Most evaluation findings are presented orally to stakeholders. Throughout the course, students will present to their classmates and answer their questions. The final presentation will allow students to present their overall projects as they would to those within their agencies. Please prepare 4 summary evaluation slides: Description of Social Problem (Slide 1), Evaluation Design/Data Analysis Plan (Slide 2), Major Results (Slide 3), Discussion and Interpretation (Slide 4). Send your summary slides to Professor Bowen 12 hours in advance of your presentation. Order of presentation will be determined by the class. Length: 7‐8 min presentation Due: Weeks 14, 15, & 16 Value: 5% Important Note: Although the course follows a logical order from problem specification to summary and discussion of results, you need to get “out of the gate” fast—the steps in the research process need to be considered simultaneously rather than sequentially. Do not wait for the assignment to get started on a particular phase. You completed the essential reading for this course in SOWO 510. It is assumed that you already have the relevant content for success in this course. Please keep in mind that this is a research exercise. 6
GRADING SYSTEM: Component Proportion Phase Assignments 1‐5 30% Final Paper (Evaluation Report) 50% Student Presentation 05% Class Participation 15% COURSE OUTLINE AND ASSIGNMENTS Week 1: January 11 Introduction to Evaluation of Social Work Interventions Content Overview of the course SOWO 510 revisited Review of required assignments for the course Discussion of how to: 1. Identify issues to evaluate in a practice setting 2. Form evaluation problem statements Supplemental Readings Rubin & Babbie: Chapter 4 and Appendix A Royse et al.: Chapters 1 and 15 Hatry et al.: Step 1 Class Activity In one paragraph or less, briefly describe the practice or policy related problem you will address and its importance. You will have an opportunity to discuss this idea with one of your colleagues in class. Assignment (Due: Week 3, January 25) Meet with your field supervisor and your faculty liaison, if available, to share your syllabus for SOWO 810, and discuss your practice or policy related problem and the proposed evaluation. Please have your field supervisor sign below to acknowledge completion of this assignment. Field Supervisor’s Signature / Date 7
Week 2: January 18 Independent Assignment (No Class) Assignment Conduct a comprehensive review of SOWO 510, including Rubin & Babbie, Royse et al., and Hatry. Identify three major insights/recovered memories from your review that have implications for your success in SOWO 810. Resources SOWO 510 Website 1. Go to http://sakaipilot.unc.edu and click the “Oxyen Log In” button 2. Type your username (your Onyen) and password, and click Login. 3. You will “land” in your own personal My Workspace area. Week 3: January 25 Steps in Intervention Research Integrative Discussion Results/insights from the discussion with your supervisor/faculty liaison Insights from independent review Refinements to your practice or policy related problem Required Reading *Fraser, M. W., & Galinsky, M. J. (2010). Steps in intervention research: Designing and developing social programs. Research on Social Work Practice, 20, 459‐466. *Thyer, B. A., & Myers, L. L. (2011). The quest for evidence‐based practice: A view from the United States. Journal of Social Work, 11(1), 8‐25. Evaluation Example Related to Client Needs, the Process of Service Delivery, and Service Outcomes: Unit Services Strategy2 *Bowen, G. L., Martin, J. A., Liston, B. J., & Nelson, J. P. (2009). Building community capacity in the U.S. Air Force: The Community Readiness Consultant Model. In A. R. Roberts (Eds.), Social Workers’ Desk Reference (2nd ed., pp. 912‐917). New York: Oxford University Press. Orthner, D., Bowen, G., & Mancini, D. (2003, December 15). The community readiness unit service guide for Air Force Space Command Family Support Centers. Colorado Springs, CO: US Air Force Space Command Family Matters. 2
A small group, in-class, exercise will be conducted around the readings. One half of the class will be assigned the
readings associated with this evaluation example.
8
SRI URL and Sample Ticket Codes
I am pleased to provide you with URL and demo ticket codes to complete the SRI‐Member (SRI‐
M). The results are anonymous and you can take the inventories more than once by using the same ticket code. Follow these steps: 1. Copy the 10‐digit ticket code associated with the SRI‐M 2. Click on the respondent’s link (Hold the Ctrl button and click) 3. Paste the ticket code into the open field (Ctrl V) 4. Follow the directions to complete the Inventory 5. Click on F11 for full screen view after your enter the system 6. Send me an email using the [Contact CRC] button, if you wish 7. Please print your individual profile and bring to class. Respondent’s Link: http://sri‐m.communitiesinblue.org/ Demo ticket code: M08926W001 Valid from 06 January 2011 to 11 February 2011 Evaluation Example Related to Client Needs, the Process of Service Delivery, and Service Outcomes: The Child Care Recycling Fund Concept3 Neenan, P. A., & Bowen, G. L. (1991). Multi‐method assessment of a child care demonstration project for AFDC recipient families: The genesis of an evaluation. Evaluation Review, 15, 219‐
232. Bowen, G. L., & Neenan, P. A. (1992). Child care as an economic incentive for the working
poor. Families in Society, 73, 295--303.
Bowen, G. L., & Neenan, P. A. (1993). Child day care and employment of AFDC recipients with preschool children. Journal of Family and Economic Issues, 14, 49‐68. Bowen, G. L., & Neenan, P. A. (1993). Does subsidized childcare availability promote welfare independence of mothers on AFDC: An experimental analysis. Research on Social Work Practice, 3, 363‐384. 3
A small group, in-class, exercise will be conducted around the readings. One half of the class will be assigned the
readings associated with this evaluation example.
9
Phase I Assignment– Description of Social Problem In no more than one double‐spaced page, briefly describe the practice or policy related problem you will address and its importance. Include at least three citations. If you plan to evaluate an intervention, provide a brief description of the intervention or reference a treatment manual. In addition, clearly state your draft evaluation questions. Length: Maximum 1‐2 pages Due: Week 4 Value: 6% Week 4: February 1 Development of Evaluation Justification Phase 1 Papers Due Small Group Discussion Content Discussion of how to develop evaluation justification Discussion of ethical issues faced in evaluations Supplemental Readings: Rubin & Babbie: Chapters 15, 16 Phase II Assignment – Justification of Evaluation In no more than two double‐spaced pages, write your revised evaluation questions and a brief description and justification of the evaluation you plan to conduct, include ethical considerations as appropriate, and evidence that suggestions for changes in service delivery are possible if indicated by the results. Include at least three citations. Length: Maximum 2 pages Due: Week 5 Value: 6% Note for Phases I and II: The final paper will include a literature review with at least six scholarly citations that are pertinent to the evaluation. Pay special attention to the literature on relevant evidence‐based practices in the area of study. Week 5: February 8 Development of Evaluation Designs Phase II Papers Due Small Group Discussion Content Discussion of how to develop: • Proposal evaluation designs • Evaluation logic models 10
Supplemental Readings: Rubin & Babbie: Chapters 12, 13 Royse et al.: Chapters 5, 9 Hatry et al.: Step 2 Phase III Assignment – Evaluation Design In no more than one double‐spaced page, describe your evaluation design. Indicate the type of evaluation you will implement (e.g., needs assessment, formative evaluation, process evaluation, outcome evaluation), your research design, and the strategy you will use to collect the data (e.g., focus groups, interviews, chart abstraction). Make sure to include your targeted sample size and how you will identify and recruit your sample participants (or obtain existing records). Length: Maximum 1 page Due: Week 6 Value: 6% Week 6: February 15 Development of Data Collection Instruments Phase III Assignment Papers Due Small Group Discussion Content Discussion of how to develop and search for appropriate data collection instruments Supplemental Readings: Rubin & Babbie: Chapter 6, 7 Royse et al.: Chapters 11, 12 Hatry et al.: Steps 4, 5 and Appendix Phase IV Assignment ‐ Draft instrument Obtain/develop a draft of the data collection instrument you propose to use; this may be an existing instrument, one your create, or a combination of the two. Length: As needed Due: Week 8 Value: 6% Week 7 – February 22 Consultation on Evaluation Implementation Faculty appointments, as needed, to discuss evaluation implementation issues 11
Week 8 – March 1 Development of Evaluation Data Analysis Plans Phase IV Assignment Papers Due Small Group Discussion Content Presentation on statistical analysis strategies Small group discussion of proposal data analysis plans Supplemental Readings Rubin & Babbie: Chapters 17, 18 Royse et al.: Chapter 14 Hatry et al.: Step 6 Phase V‐ Draft data analysis plan In no more than two double‐spaced pages, create a data analysis plan that includes your research question(s) and summarizes the data that will be available. Specify your analysis strategy and which technique(s) you will use to answer your research question(s) (e.g., t‐test for independent groups, ANOVA, correlations, interpretation strategy for qualitative data). Length: Maximum 2 pages Due: Week 10 Value: 6% Week 9 – March 8 (No Class) Happy Spring Break! Week 10 – March 15 Development of Proposal Statements of Implications for Agency Practice Phase V Assignment Papers Due Small Group Discussion Content Discussion of how to interpret evaluation findings in a way that is relevant for service planning and delivery Supplemental Readings Royse et al.: Chapter 15 Hatry et al.: Step 8 12
Week 11 – March 22 Consultation on Evaluation Implementation Faculty appointments, as needed, to discuss evaluation implementation issues Week 12 – March 29 Progress Review Session Small group reviews and discussion of evaluation progress Discussion of how to write evaluation reports and funding proposals Supplemental Readings: Rubin & Babbie: Appendix A and B Royse et al.: Chapter 15 Week 13 – April 5 Consultation on Evaluation Implementation Faculty appointments, as needed, to discuss evaluation implementation issues Week 14 – April 12 Presentation of Student Projects Presentations on evaluation proposals may include PowerPoint slides, or posters. Weeks 15 – April 19 Presentations of Student Projects Presentations on evaluation proposals may include PowerPoint slides, or posters. Weeks 16 – April 26 Presentations of Student Projects Presentations on evaluation proposals may include PowerPoint slides, or posters. Course evaluation Final Exam Final Paper/Program Evaluation Report Due 13