last updated: 1/05/09 FCS 510 Program Development and Evaluation in Family Life Education Andrew O. Behnke, Ph.D. 218 Ricks Hall Phone: (919) 515-9156 andrew_behnke@ncsu.edu Tuesdays 5:30 – 8:00pm Online or in Class at Tompkins G-109 Office Hours By Appointment - Virtually or in person Overview: This course will examine family focused program and evaluation development concepts and practice. Topics include building community support and oversight for programming; conducting needs and asset assessment; developing research-based, ecological interventions; and implementing fully integrated program evaluation. Students will be introduced to program development and integrated evaluation theory and processes. Evaluation and programming are best developed and conducted at the same time as they go hand in hand. Therefore, this program of study will strongly advocate for the integration of developing a program and developing an evaluation plan into a single set of tasks we will call Program and Evaluation Development. Please note that these are dynamic and reciprocal processes. Through this course, these patterns will be studied, and then through a field project, applied and experienced by the students. Course Objectives: 1. 2. 3. 4. 5. By the end of this course, students will create a mini-grant proposal which will include the various key elements in a real grant proposal (e.g.,program description, budget narrative, etc.) By the end of this course, students will construct a logic model for grant and for a program evaluation. By the end of this course, students will determine key issues of interest to them, assess how to developing and implementing an ecological solution to the issue, and evaluate its impact. By the end of this course, students will develop an evaluation plan of a real program in North Carolina. By the end of this course, students will carry out their evaluation plan and present their findings to the program administration and to class members. Readings: This course readings will be drawn from the following texts and will be available online. Students may decide to purchase the following books for their future reference. Optional Texts Designing and managing programs: An effectiveness-based approach. $57 By Peter Kettner, Rob Moroney, & Larry Martin ISBN: 0761915494 Creating Small Scale Social Programs $43 By Barbara Schram ISBN: 0803974353 Program Evaluation: Methods and Case Studies (6th Ed) $61 By Emil J. Posavac & Raymond G. Carey ISBN: 0130409669 Planning, Program Development and Evaluation (2nd Ed) $42 By Thomas C. Timmreck ISBN: 0763700622 The Only Grant-Writing Book You'll Ever Need: Top Grant Writers and Grant Givers Share Their Secrets $9 By Ellen Karsh & Arlen Sue Fox ISBN: 0786717548 Class Schedule: (updated 1/05/09) Dates subject to change, though the order of assigned readings will remain. DATE PPT TOPIC and BOOK CHAPTERS OTHER READINGS & ACTIVITIES & DELIVERABLE Posavac Schram WEEK# DUE DATES 1/13 Introduction to Week 1 program and evaluation development (ppt) 1/20 Ecological http://parenthood.library.wisc.edu/Powell/Powell.html 1 (pp. 1- 1 & 2 Week 2 Prevention 22) SKIM Programming (pp. 1-43) Student Discussion Leader planning and evaluation (ppt) 1/27 Strategic planning, 2 (pp. 3&4 Distance Education librarians to present literature review session via Elluminate Literature review, 29-44) SKIM and Curriculum (pp. 47Week 3 selection (ppt) 98) No Student Discussion Leader 5 (pp. 98128) MINI APA EXERCISE DUE 2/2 2/3 Timelines and process charts Week 4 (ppt) PRE-PROPOSAL OF MINI-GRANT DUE 2/3 2/10 Creation of a 3 (pp. programming logic 47-70) Week 5 model (ppt) Timmreck Chapter 8 – (pp. 147-168) (pdf) 6 (pp. 131166) Lauffer - Getting it Right (pp.317-335) (pdf) [4.4MB download] Student Discussion Leader http://www1.uwex.edu/ces/lmcourse/ Student Discussion Leader http://www.learnerassociates.net/proposal/ 2/17 Grant Writing Week 6 Nuts and Bolts (ppt) http://foundationcenter.org/getstarted/tutorials/shortcourse/index.html ROUGH DRAFT PART “A” DUE 2/17 http://www.lib.msu.edu/harris23/grants/tenways.htm http://www.gpc.edu/~ebrown/infobr3.htm http://www.mcf.org/mcf/grant/writing.htm Student Discussion Leader http://srdc.msstate.edu/publications/227/227.htm 2/24 Basic Evaluation 6 Week 7 and Community (pp. 114Assessment (ppt) 130) Timmreck Chapter 5 – (pp. 87-118) (pdf) Student Discussion Leader 3/3 Week 8 Spring Break - no class - - - 3/10 Quantitative Week 9 evaluation (ppt) 9 & 10 (pp. 175213) ROUGH DRAFT PART “B” DUE 3/10 3/17 Qualitative Week methods (ppt) 10 Statistical tools for program evaluation (pdf) http://www.joe.org/joe/1989summer/a5.html FLE - Chap. 4 - (pdf) 12 (pp. 234251) Student Discussion Leader http://ag.arizona.edu/fcs/cyfernet/cyfar/Intervu5.htm http://ag.arizona.edu/fcs/cyfernet/cyfar/Portfo%7E3.htm http://ag.arizona.edu/fcs/cyfernet/cyfar/Exisrec5.htm http://ag.arizona.edu/fcs/cyfernet/cyfar/focus.htm 3/24 Budgets and cost 11 (pp. Week benefit analyses 215-232) 11 (ppt) Student Discussion Leader http://www.winona.edu/grants/Education/Budget.htm http://www.joe.org/joe/1999august/tt3.html Cost Benefit Analysis (pdf) ROUGH DRAFT PART “C” DUE 3/24 Kettner Chapter 9 – (pp. 173-204) Appendix – (pp.239-249) 3/31 Program Week implementation 12 and Process evaluation (ppt) Student Discussion Leader Week 12 PowerPoint 7 7 (pp. 132- (pp. 167152) 193) Student Discussion Leader ROUGH DRAFT OF GRANT DUE 3/31 4/7 No class - CATCH Week UP WEEK 13 4/14 Week 14 4/21 Week 15 4/28 Week Consultation with Dr. Behnke about grant write up and final project Program review & adaptation Outcome & impact evaluation (ppt) MINI GRANT DUE 4/14 Program sustainability and reporting program accomplishments (ppt) ROUGH DRAFT PART “D” DUE 4/21 "Dead Week" No readings and no formal class 8 (pp. 154174) Kettner Chapter 11 – (pp. 215-238) What Works (pdf) Student Discussion Leader 13 & 14 8 (pp. 253- (pp. 193286) 220) Email me with your questions if you have any. Bring all your rough drafts together for the final paper. 16 5/5 FINAL PAPER Week DUE & 17 Get together for presentation of Final Papers PRESENTATIONS DUE Copyrighted Materials: This course web site contains copyrighted materials and was developed for instructional purposes to be used by graduate students in FCS 512, Department of Family and Consumer Sciences at North Carolina State University. Students currently registered in this course are permitted to print or make copies of parts of this site for their own personal use in conjunction with completing the course. Text, audio files, images or design of this web site may not otherwise be distributed or modified in any manner without the prior written permission of the instructor. This site may also contain material by various authors that is covered under additional copyright protection. In all cases this material will be clearly cited and may not be further disseminated without the express and written permission of the legal copyright holder. The user must assume full responsibility for any use of the materials, including but not limited to, infringement of copyright and publication rights of reproduced materials. Instructional Delivery Format: This course is primarily delivered via Elluminate [http://www.elluminate.com/] a full-featured Web conferencing tool that is free to all NC State students and faculty. This delivery medium allows students to fully participate from a distance. Elluminate provides the option for two-way audio, video, and file and application sharing. Live sessions will be recorded for later viewing, as well. The course is completely synchronous, meaning each class will meet during a specified class time (which will be selected to meet the needs of most students). This method will allow for students to interact with one another and with the instructor in real time. Students use a headset/microphone attached to a computer to be heard by the instructor and fellow classmates, and will need to follow a few steps to properly configure their computers to function properly with Elluminate. Additional information is available to help students get started with Elluminate . Blackboard Vista Learning Management System While Elluminate is primarily a synchronous learning environment, the Blackboard Vista learning management system provides a set of instructional tools to facilitate learning, communication, and collaboration between an instructor and students that will most often be asynchronous in nature. Students and or the instructor do not have to be logged in at the same time. In your Vista FCS 510 course site students will find the course syllabus and links to most of the content for this class. Vista is a password-protected learning environment, and students login using their Unity ID and Passwords to access their courses. If students ever have a problem accessing this course, contact the Help Desk at919-515-HELP or help@ncsu.edu. Also, feel free to copy your email to the instructor so that we can be aware of any commonly occurring issues. Browser: To access a Blackboard course it is essential to have a properly configured browser. Students can find imformation about the recommended browsers for Blackboard Vista and run a routine that performs a Browser Check by selecting the link on the Vista login page (http://vista.ncsu.edu). Once you ensure that you are using an acceptable browser, this exercise will help you determine if you have the proper plug-ins and settings to enable all of the tools on the site to functional properly. Instructions are provided on the Browser Check page. Some of the course content may also require specific browser plug-ins (ie., Flash Player, Adobe Acrobat Reader) to operate. You will be advised of any content that requires a specific plug-in to be loaded on your computer and you will be directed to a site for a free download of that plugin. Links to many plug-ins are available at: http://vista.ncsu.edu/help/plugins.php Students with Disabilities: Reasonable accommodations will be made for students with verifiable disabilities. In order to take advantage of available accommodations, students must register with Disability Services for Students, 1900 Health Center, Box 7509, NCSU. Phone: 919-515-7653. http://www.ncsu.edu/policies/academic_affairs/courses_undergrad/REG02.20.1.php Academic Integrity: The NCSU Student Code of Conduct covers all work done in this course. Any suspected violations will be promptly reported. Academic dishonesty will result in an automatic failing grade for the course. Student plagiarism has become a problem and will not be tolerated even once. http://www2.ncsu.edu/prr/student_services/student_conduct/POL445.00.1.htm Attendance: Students who miss class due to excused absence should work with the instructor to make up missed work. Documentation for excused absences are required by the instructor during the following class period.http://www.ncsu.edu/policies/academic_affairs/pols_regs/REG205.00.4.php End-of-course Evaluations: Online class evaluations will be available for students to complete during the last two weeks of class. Students will receive an email message directing them to a website where they can login using their Unity ID and complete evaluations. All evaluations are confidential; instructors will never know how any one student responded to any question, and students will never know the ratings for any particular instructors. Evaluation website: https://classeval.ncsu.edu Student help desk: classeval@ncsu.edu More information about ClassEval: http://www2.acs.ncsu.edu/UPA/classeval/index.htm Grading: This course will be evaluated using an A to F plus/minus system as follows: A+ A AB+ B B- = 98-100 = 94-97 = 90-93 = 87-89 = 84-86 = 80-83 C+ = 77-79 C = 74-76 C= 70-73 D+ = 67-69 D = 64-66 D= 60-63 F = 59 and below Please note: Late assignments are not accepted unless students have a university recognized excuse (religious holiday, death in family, medically excused absence due to illness, inclement weather.) Assignments are due by 5:30PM on the due date. After that time, 3 points will be deducted for each day that it is late (including weekends). Papers will NOT be accepted after the third day. Students will be evaluated as follows: Participation Self introduction discussion posting Elluminate session discussions Elluminate session leader APA mini assignment - team activity Participation total 1% 10% 15% 1% 27% Mini-grant write-up Pre-proposal Proposal Mini-grant write-up total Final Project Report Creative final presentation Final project total 3% 30% 33% 30% 10% 40% _______ 100% Course Requirements: Participation Self-Introduction discussion posting: Getting to know your classmates is an important part of building a learning community and research has shown that a sense of community is an important factor in the creation of a rich learning experience for DE students. As part of the Getting Started exercises each student should post a self introduction that includes information about your experience and future goals. (1%) Elluminate session discussions: discussions of the readings and the ideas they generate are essential. To promote effective class learning, preparation for, and participation in class is imperative (10%). Elluminate session leading: Each student will have an opportunity to be the leader on one week’s assigned readings. They will be responsible to know the reading and prepare some questions to spark discussion in the class. They are not responsible for a ppt presentation. Each student should prepare to talk for 30 minutes to an hour, should mark up their readings, prepare questions for the class (send to Dr. Behnke and classmates before class), and be prepared to share their insights (15%). APA mini assignment team project: Students will work in pairs to create properly formatted examples of different works using APA style. The samples will be submitted as PPT slides via the Vista Assignment tool according to the course schedule. (1%) Mini-Grant Proposal Students will develop grant-writing skills through actually putting together a fictitious (or real if desired) minigrant proposal. Pre-proposal Grant Assignment: Each student should write and submit a preproposal for their program grant. Preproposals are due by 2-03-09 and must be submitted electronically as a Word document via the Assignment tool. (3%) The preproposal should contain: APA Cover page, which should give i) title of proposal, ii) list of investigators, titles, and contact information, iii) name of the foundation or institution to which the grant will be submitted; and iv) project duration (beginning and ending dates) A concise (2-page limit) description of the project, including its motivation, objectives, major program components, program evaluation, and importance to the field. Be sure to address both the "short term outcomes” as well as the "broader impacts" (e.g., learning new skills or changing behaviors vs. reducing teen pregnancy or increasing school readiness) A CV for the PI (2-page limit). Students may be shorter than the example. See attached. All preproposals should be written in a font no smaller than 10 pt and with margins of 1". Here are two examples of CVs. Mini-CV example 1 (.doc) Mini-CV example 2 (.doc) Grant Proposal Assignment: Each student will write a programmatic grant proposal in the form of an NIHstyle grant (although the full grant should be no more than 20 pages in length, not 30 like NIH grants). Each grant should be submitted as a rough draft via the Assignment tool by 3/31. He will return grade and return papers in one week. Each student will then have two weeks to make changes and turn in their final mini grant by 4/14/09. Dr. Behnke will then assign reviewer(s) for each grant. An example review sheet (Panel Summary) is found below. The grants should be written as if they were actually being submitted to NIH or some private foundation. The topic of the grant is completely up to the discretion of the student, but we suggest that they choose a topic central to their core interest/research area. We are hopeful that this exercise will benefit the development of students future carreers and/or subsequent grant proposals. (30%) Here are some examples of successful proposals from a previous year. Mini-Grant Proposal example 1 Mini-Grant Proposal example 2 The following links will be useful: http://foundationcenter.org/getstarted/tutorials/shortcourse/info.html http://grants.library.wisc.edu/organizations/proposalwebsites.html Each submission must include the following items, in the following order: APA Cover Page – make this real to the grant you plan to apply for (which should give i) title of proposal, ii) list of investigators, titles, and contact information, iii) name of the foundation or institution to which the grant will be submitted; iv) project duration (beginning and ending dates); and v) estimated cost for each year of the project (as well as a total). Project Summary – A type of abstract- 250 words or less Table of Contents – A brief table of contents for the proposal Project Description (limited to 6 pages single spaced, but excluding references, figures, and tables which should go in appendices) I. Background and Significance II. Specific Aims III. Program Design and Methods IV. Evaluation Design and Methods V. Implications For Future Studies/Programs Appendices Appendix A. References (APA and single spaced) Appendix B. Biographical Sketches (<2 pages – your revised CV placed here) Appendix C. Budget – I will provide a form to fill in (one for each year of the project, plus a cumulative budget; A project may last up to 3 yrs). For our purposes, there are no constraints on the size of your grant and you may request support for your own salary. Appendix D. Budget Justification – This is the budget narrative, which explains the numbers on the budget. Appendix E. Program Logic Model – Feel free to pattern yours after the logic model examples from the linked example papers above and below. Be sure to follow the Wisconsin Extension Model at: http://www1.uwex.edu/ces/lmcourse/ Appendix F: Timeline / Process Chart JUST MAKE ONE OF THESE THAT LOOKS DOABLE EXAMPLE PANEL SUMMARY GRADING FORM Principal Investigator: Program: Title: Positive and unique aspects of the proposal: Negative aspects of the proposal: Suggestions for future submission: INCLUDED PARTS: -APA Cover Page – i) title of proposal, ii) list of investigators, titles, and contact information, iii) name of the foundation or institution to which the grant will be submitted; iv) project duration (beginning and ending dates); and v) estimated cost for each year of the project (as well as a total). -Project Summary – A type of abstract- 250 words or less -Table of Contents – A brief table of contents for the proposal -Project Description – Limited to 6 pages single spaced, not including references, figures, and tables: I) Background and Significance; II) Specific Aims: III) Program Design and Methods; IV) Evaluation Design and Methods; V) Implications For Future Studies/Programs -Appendix A. References - APA and single spaced -Appendix B. Biographical Sketches - <2 pages – your revised CV placed here -Appendix C. Budget – For one for each year of the project -Appendix D. Budget Justification – Explanation of the budget. -Appendix E. Program Logic Model – Following the Wisconsin Extension Model -Appendix F: Timeline / Process Chart Panel Recommendation: Rating/Score 25 out 25 Fund Amount: $ Do not fund Final Project/ Program Evaluation To assure students understand material that has been presented, they will implement the course material in a real life program development and evaluation integration project. Program visits will occur at least 3 times and through the course of these visits a detailed program logic model will be developed for the program. Students will work with the program to assess its integration of programming components discussed in this course (including creating [or updating if they have one] a program logic model). The program’s current evaluation plan will be assessed and if minimal, a new plan will be designed by the individual. The final product will consist of a comprehensive descriptive report and a set of program recommendations (30%). This could be presented to the program leaders in a meeting following the completion of the course (not required). Students will also complete a class presentation (10%), which will be an overview of the program description, report, findings, recommendations, and program stakeholder reactions – with lessons learned. APA format should be used in all writing submitted for this project. The criteria for the grading the report, presentation, peer rating scale are below. This project will contribute 40% to the student’s final grade. Program Visits – Each student will need to write these up during the course of the semester (see below) Initial Visit: (EARLY IN THE SEMESTER) Students should get to know key players, how far the program is into implementation, key implementation (process) related difficulties for program administrator, key problems impeding significant impacts or outcomes. Students should suggest to program administrators that as part of a masters level class that they are taking they will need to find a way to be involved in the evaluation of a program. They should ask what key components of the program might be observed or evaluated. Students should explore ways in which they might help out in the evaluation or implementation of the program as to not be a burden to the organization but rather a real asset. Students should explain that they will be preparing an evaluation tool to meet the needs of the program administrator. Since this will likely be a type of Program Monitoring, students will want to explore the type of evaluation tools needed by the administrator (Process Evaluation, Efficiency Evaluation, Formative Implementation Evaluation). Program administrators may not be willing to let students actually use their measurement tools with their clients, but students should ask them if they would at least like a copy. Students will bring this instrument to the second visit with their program administrator. If conducting an evaluation is not an option, students should relate to the program administrator that they will create a “hypothetical evaluation tool” of “set of measurement tools” which they will leave with them as a reference for their future evaluation needs. Students should request copies of a program logic models, strategic plans, timelines, milestone charts, task-time flow charts, PERT charts, Gantt charts, budgets, and/or budget narratives. These may not be available, but if a student is given permission to use these as references they will make understanding the program much more straightforward. Students will write up their impressions and an outline of this first meeting to go in the appendices. Students will begin writing their program description and literature review. Second through Fourth Visits: (STARTED BY 5th WEEK) Students should present the evaluation tool (or hypothetical evaluation tools) to the program administrator, and get feedback to improve it. Discuss times and availability for actually implementing one such evaluation. If the evaluation tool has previously been accepted by the program administrator via email, students may choose to make the second or third visit an actual evaluation visit. If a student is unable to actually use their instrument(s) with clients, they will need to create made up data (see below). Students should ask any questions they may have that are needed to better understand the running of the program. Students should conduct two (2) evaluations or observations of the program, preferably with different groups or at least during different visits. This will assure the student at least a minimal understanding of the program which they are evaluating or observing. OR If helping in the evaluation of the program is not possible, students should explore the option of participating in some other capacity, such as an observer or volunteer in some part of the program to experience the delivery of the program. Students will participate in at least two (2) planning meetings and record their reactions to the process. Students will write up their impressions and an outline of each of these meeting to go in the appendices. Writing up the Report Students must turn in each section of this report by the due date listed below. They will get feedback on each section about a week after having turned it in. Each subsequent section should be added to the previous and changes made to the previous section as requested by Dr. Behnke. If you do not make changes as requested you will likely receive a poor grade. As Dr. Behnke will likely make many changes in the version of the report which you turn in, you will need to use this version (paste it into the new document) and accept or reject changes using the menus “->Tools -> Track changes -> Accept or reject changes.” Writing is an art and there is no best way to say something, but students are encouraged to use professional language and syntax. Here is an example of a successful proposal from a previous year. Program Evaluation Report example 1 (.doc) Program Evaluation Report example 2 (.doc) Required Section Rough Drafts Parts A-D will be submitted via the Vista Assignment tool according to the following schedule. Students will receive written feedback and a letter grade of E (Excellent), S (Satisfactory), or N (Needs Improvement). No points will be given, but consideration of student effort and performance as reflected in these interim submissions will be made when grade for the Final evaluation paper is awarded. A = Due by 2/17/2009 B = Due by 3/10/2009 C = Due by 3/24/2009 D = Due by 4/21/2009 FULL REPORT DUE BY = 05/05/2009 Each report must include the following items, in the following order: A Title page – APA format please D Abstract or executive summary – do this last when you’re done – 1-2 paragraphs A Project description – Describe the program briefly – 2-3 paragraphs A Brief literature review – make a case for doing work like this - see example below – 1-2 paragraphs B Data sources & Evaluation methodology – This is a very important and long section, just try to be simple and describe what you want to do. Don’t worry about statistical terminology or statistical wording you should discuss two or three methods – at least one qualitative and one quantitative. This where you should describe: 1. 2. 3. 4. 5. 6. 7. 8. who will do what to carry out the evaluation who will need informed consent who will be surveyed how they will be surveyed who will analyze the data when things will be evaluated what will be analyzed how you will analyze the data once you have the data. It is also in this section where you should describe the measures you will be using. How many items, what the questionnaire is trying to get at through its questions, etc. For instance, you might say that you will use a pre and post survey to assess coping with divorce among participants. [You can then use a parenthetic statement like (See Questionnaire 1 in Appendix E) to help the reader know where to find the measure you are referring to measure coping]. – 2-3 paragraphs one for each type of method used AND 2-3 paragraphs describing the list above. C Results – Make up charts or graphs to demonstrate what the project has done Short term and midterm; If qualitative components, provide quotes or examples that make your case for program success and areas for program improvement - YOU CAN MAKE UP HYPOTHETICAL DATA IF YOU WOULD LIKE – YOU ARE NOT REQUIRED TO GATHER DATA UNLESS YOU WANT TO AND HAVE PERMISSION TO DO SO - Use charts, diagrams, a made up table, etc. – 1-2 pages C Program accomplishment report – Some programs like to list some of their accomplishments in table with bullets – 2 paragraphs and maybe a bulleted list D Recommendations and reactions – The most important part of the report - what you gained from the experience and your recommendations to the program to make it run smoothly. Include the reactions of those you worked with at the agency or organization you reviewed. Make sure most of this is helpful feedback that you could take to the program stakeholders. – 1-3 pages D Conclusion – 1 paragraph Appendicies D Appendix A: References (APA format) - a page or two single spaced C Appendix B: Program logic model – use your mini grant as a template to improve on – 1 page or so D Appendix C: Program visit descriptions MAKE SURE YOU DO THESE ALL ALONG – 1-2 paragraphs per visit (label each visit say one visit and tell about it) B Appendix D: Evaluation measures/ instruments (Surveys, focus groups, etc) – Length depends on how many measures you have Project Grading Criteria 1. Students will write a 10-20 page report (double spaced) which will be graded on: Writing Clarity (3pts) Use of APA style (2pts) Organization of materials (3pts) Content Choice of what to include – abstract, outline, report components (5pts) Quality of analysis, instruments (4pts) Site visit explanations (5pts) Recommendations and reactions (5pts) Appendices (3pts) 2. Students will prepare and present a 7-10 minute PowerPoint presentation of their findings to the class. Presentation to include: Quality of visual aids and props (2pt) Presentation style (1pt) Presentation agenda (2pt) Presentation content (5pts) © 2009 Department of 4-H Youth Development & Family and Consumer Sciences NC State University