Outcome-Based Management Program Training Presentation September 8-12, 2008 Introductions and Icebreaker Please say the following about yourself: Name Club Position One positive thing you’ve seen with a member or the club while working at BGCC -2- Agenda Introductions 10 min Part 1: Outcome-Based Management Program Overview 15 min Part 2: Data Collection Requirements 20 min Part 3: Immediate Action Items and Resources 10 min Break 10 min Part 4: KidTrax Training and Refresher 30 min Q&A 10 min -3- Part 1: Outcome-Based Management Program Overview -4- What Are Outcomes? Outcomes are the desired results, both short term and long term, that our members will experience by attending our clubs and programs Intermediate Outcomes Long Term Outcomes Short term program results that are small contributions towards long term outcomes Program end results, often difficult to measure, that target core assumptions and behavior, and are built from short term outcomes Examples Examples Academics: Increased likelihood of being promoted grade levels Academics: Academic achievement and attitudes that lasts a lifetime Character: Greater knowledge of positive vs. negative behaviors and attitudes Character: Avoidance of detrimental and dangerous behavior Health: Higher participation in sports activities outside of the club Health: Maintaining a healthy lifestyle through smart nutrition and behavior These outcomes, while somewhat abstract, all contribute to the core mission of the Boys & Girls Clubs of Chicago -5- How Are Outcomes Achieved? Outcomes are achieved by using various inputs and processes, and can be visualized by using an outcome logic model Outcome Logic Model ` Outcomes Input Resources required to create the outcome Two Program Instructors Two volunteers One Program Instructor Three volunteers Four baseball bats Four baseballs Process Output Intermediate Term Long Term How the inputs are What the club produces Short term program used to deliver the by running the program, results that are small programs and activities expressed using contributions towards measurable figures long term outcomes Daily Power Hour homework help Weekly softball team games 60 unduplicated youth participate weekly 15 youth participate daily 20 unduplicated youth participate weekly 50% of Juniors attend each week -6- Increased likelihood to complete assignments on time Increased likelihood to understand the value of homework Increased awareness of healthy exercise habits Program end results, often difficult to measure, that target core assumptions and behavior, and are built from short term outcomes Increased likelihood for higher academic performance and school promotion Improved ability to identify and utilize available resources Increased likelihood of maintaining balanced diet, weight, and overall health What Is The Outcome-Based Management Program? The Outcome-Based Management (OBM) program will allow us to collect information to see how club attendance impacts member outcomes in terms of academics, health, and character …to We create willOBM use data reports fromthat KidTrax… can be communicated to …and stakeholders match it with and help data improve providedprograms! by CPS… CPS grades by club ADA / length of membership CPS attendance by club ADA / length of membership Character development by club ADA / program attendance Leadership development by club ADA / program attendance Health attitudes by club ADA / program attendance Outcome-Based Management Reports KidTrax Database CPS IMPACT Data -7- What Are The Benefits Of Creating The Outcome-Based Management Program? OBM program will have wide ranging benefits at both the club level and the executive level Improve Programs and Operations Earn More Money Improve Our Members Outcome reports lead to… Outcome reports lead to… Member-level data and tracking lead to… …program assessments and benchmarks… …quicker, easier prep for grant proposals… …more information to work individually with members… …and allow us to share best practices and improve programs! …and allow us to get more money! …and improve their academics, character, and health! Remember, this is a long term initiative and benefits won’t be recognized overnight -8- How Can We Ensure This Will Be Successful? Directors and staff members need to understand the following factors that are key to making this program a success Understand the benefits of OBM Ensure accurate and timely data entry Directors and staff first need to understand why we’re developing this program to make sure it’s done right! The outcomes are only as good as the data that’s entered! Understand KidTrax and its functionality Regularly use the metrics and KidTrax To take full advantage of the OBM program, directors and staff need to know how to input the KidTrax data and run new reports! Reviewing data on a regular basis will help you understand the power of OBM! This program is NOT intended to make your job more difficult. Instead, it will be designed to be simple, and result in more accurate data and better results -9- Test Your Knowledge – OBM Program 1. Which of the following is NOT an example of an outcome? A. Members improve academic performance B. Members attend Power Hour twice per week C. Members have greater self esteem and self confidence D. Members have a more active lifestyle Answer: B - 10 - Test Your Knowledge – OBM Program 2. The Outcome-Based Management (OMB) program A. Utilizes data from a number of sources, including KidTrax and CPS B. Will generate reports that will help improve our programs C. Is the responsibility of everyone to be successful D. All of the above Answer: D - 11 - Test Your Knowledge – OBM Program 3. Which of the following is NOT a benefit of the OBM program? A. Improve programs and operations B. Earn more money C. Increase staff support D. Improve our members Answer: C - 12 - Test Your Knowledge – OBM Program 4. For the OBM program to be successful, everyone will be responsible for A. Making sure the members are attending all programs B. Relying on their Club Directors to perform the majority of activities C. Understanding the benefits and ensuring good use of KidTrax D. Holding more programming than in previous years Answer: C - 13 - Part 2: Data Collection Requirements - 14 - Data Collection Processes The data collection requirements will cover four processes related to attendance tracking and survey / test administration Attendance Club Attendance Tracking Description Usage Surveys / Tests Selected Program Attendance Tracking Outcome Measurement Toolkit Administration Selected Survey and Test Administration Tracking member attendance at clubs, including entry and exit times Tracking member attendance at selected programs Administering survey tool developed by BGCA on a regular basis Administering surveys / tests developed by BGCA / clubs on a regular basis Used with other metrics to see how club attendance impacts members Used with other metrics to see how program attendance impacts members Used with club and program attendance to analyze changes in attitudes and behaviors Used with program attendance to analyze changes in attitudes, behaviors, and knowledge - 15 - Club Attendance Tracking Requirements and Details Club Attendance Tracking Selected Program Attendance Tracking Outcome Measurement Toolkit Administration Selected Survey and Test Administration Key Requirements 1. Every member must check in and out at front desk to participate in club activities 2. Club staff must scan members into KidTrax on a daily basis 3. Club staff must ensure that member entry and exit times are properly inputted into KidTrax Other Details Responsibility: Front Door Volunteer / Assigned Club Member / Staff Member – Please identify a person who is trained / familiar w/ KidTrax and able to assist others entering data Frequency: Daily Batch Scanning: Only permitted for school-based clubs by using sign-in sheet, unless there are emergency or special circumstances – Members who enter 30 minutes after the club opens or leave 30 minutes before the club closes must have their times recorded on the sign-in sheet and be entered into the system separately – Front door resource at stand-alone clubs should use ID scanning or manual scanning in Trax Attendance Manager Best Practices: Many clubs have created binders containing backup IDs, printed out member bar codes, or created new sign-in sheets with Time In / Out to use as backups - 16 - Choose Your Own Adventure – Club Attendance Tracking Club Attendance Tracking Selected Program Attendance Tracking Outcome Measurement Toolkit Administration Selected Survey and Test Administration Front Desk, Stand-Alone Club You’re sitting at the front desk, texting w/ your friends, when a member walks in. You ask for his ID card, but the member forgot it! You panic and start breathing heavily. Do you… Uh oh, guess we won’t be able to track that member! Shoo him into the club without checking in? Stop, breathe, then perform a manual KidTrax scan? Good choice! Back to texting, until… Front Desk, School-Based Club We’re one team – you should be helping out! Decide to wait a couple days to enter this data – this isn’t my club, right? You’re acting quick, but you need to include exit times too! Take the initiative and batch scan everyone in w/ the same entry / exit time? Even though it’s not done the same night, you’re doing the right thing! Wait until tomorrow morning to batch everyone except the member that left early? - 17 - …you’re mysteriously transported to a school-based club a couple hours later. All the members have left and they’ve put their exit times on the sign-in sheet. One member had to leave 45 mins early, but other than that, everyone left when the club closed. Do you… Program Attendance Tracking Requirements and Details Club Attendance Tracking Selected Program Attendance Tracking Outcome Measurement Toolkit Administration Selected Survey and Test Administration Key Requirements 1. Every child must check in and out with the Program Instructor to participate in selected programs and activities 2. Club staff must enter program attendance within two days of program session 3. Batch scanning is acceptable for data entry Other Details Responsibility: Program Instructor (work w/ Directors to get appropriate KidTrax training / assistance) Frequency: Up to 2 days after program session Attendance Sheets: Instructors should utilize attendance sheets provided in program planning kits Participating Programs: On-Going Programs Quarterly Programs – SMART Programs – SMART Moves – Career Launch – Power Hour – SMART Girls – Youth for Unity – College Bound – Street SMART – Triple Play – Keystone Club – Passport to Manhood – Skill Tech / Digital Suite – Torch Club – Money Matters - 18 - Choose Your Own Adventure – Program Attendance Tracking As the Program Instructor, you’ve started your lesson for the day and have all the members pumped! Suddenly, 10 minutes into it, one of the participants runs out of the room to take a phone call and doesn’t come back. Do you… Make note of it on the attendance sheet and not to give her credit for attending? If he knows how to enter data, he’ll be happy to help you out up to 2 days after the session Shrug it off and check her in normally– she got the gist of the lesson, right? Ignore it until your Program Director catches you? Ignoring it only makes it worse! The quick, sensible answer – good choice! Club Attendance Tracking Talk with your Club Director about learning to use KidTrax? Batch scan it at the end of the program with the Membership Clerk? - 19 - Selected Program Attendance Tracking Outcome Measurement Toolkit Administration Selected Survey and Test Administration She’s not here for long, so she doesn’t get credit If her phone call’s more important, she shouldn’t be included in attendance Program’s over and you’re wiped out. You know you have two days to put the attendance into KidTrax, but you never learned how to use the system – spent too much time playing Guitar Hero w/ the members. Do you… Outcome Measurement Toolkit Administration Requirements and Details Club Attendance Tracking Selected Program Attendance Tracking Outcome Measurement Toolkit Administration Selected Survey and Test Administration Key Requirements 1. Toolkit must be completed by the same members 3x per year during time period in October, January, and May 2. A set number of members must use toolkit over this period 3. All answers must be inputted into KidTrax Other Details Responsibility: Club Director / Program Director / Club Manager / TEEN Reach Coordinator Frequency / Timeline : 3 times per year (allot ~2 week; must be completed by October 24, January 18, and May 16) – Clubs may allot extra time to input the toolkits into KidTrax, but must be uploaded by dates above – Suggested administration dates: October 13-24, January 7-18, and May 5-16 Administration: Members must complete the appropriate survey toolkit for their grade level and should be tested at least twice per year TEEN Reach: Participants are required to use the toolkit only once per year (suggested month: January) and must complete the separate Responsible Youth section Expectations: Clubs are expected to survey 70% of their projected ADA goals for that year - 20 - Club Attendance Tracking Choose Your Own Adventure – OMTK Administration As the Club Manager, it’s the first week in January and you’ve been busy planning for the new year. You look at your to-do list and *BAM* it hits you that you haven’t done anything to plan for the OMTK. Uh oh. Do you… Your TEEN Reach coordinator will appreciate it! You don’t want to lose funding, do you? Schedule computer time over 3 weeks (2 won’t be enough) and schedule testing by age groups? Ignore it and feign ignorance – it’s not like anybody’s checking Work with your TEEN Reach coordinator to ensure all participants have taken the OMTK? Pray that everyone takes it and the Illinois Alliance doesn’t notice? - 21 - Selected Program Attendance Tracking Outcome Measurement Toolkit Administration Selected Survey and Test Administration The earlier you can plan to administer the toolkit, the easier it will be! Ignoring it will only get you comments on your annual review You’ve scheduled the OMTK administration, but know there are also TEEN Reach requirements that you’ll need to meet. Do you… Selected Survey and Test Administration Requirements and Details Club Attendance Tracking Selected Program Attendance Tracking Outcome Measurement Toolkit Administration Selected Survey and Test Administration Key Requirements 1. Selected pre / post surveys and tests must be administered to all members 2x per program at the beginning and end of program 2. Surveys / tests must be administered each time the program is run 3. All answers must be inputted into KidTrax Other Details Responsibility: Program Instructor (work w/ Directors to get appropriate KidTrax training / assistance) Frequency: 2 times per program (during first and last day of instruction) Administration: Staff may administer tests either electronically or by paper, but they are responsible for putting the information into KidTrax Participating Programs*: – SMART Moves – SMART Girls – Street SMART *These requirements do not change any guidelines for other programs (e.g., 21st Century) - 22 - Choose Your Own Adventure – Selected Survey and Test Administration As a Program Instructor, you’re preparing materials for your first class and you realize that you completely forgot about the pretest. You don’t have access to computers, so you have to do it by paper. Do you… Call up the service center to have them send it to you? Access the BGCC intranet and print it out from the OBM section? Don’t despair – your participants will do great! But first get those tests into KidTrax… There’s still plenty of time for planning – entering data in KidTrax only takes a few minutes! Make sure that you enter the test data in KidTrax 1-2 days after it’s given, then work on the lesson plans - 23 - Selected Program Attendance Tracking Outcome Measurement Toolkit Administration Selected Survey and Test Administration You don’t want to bother them with that, do you? Sulk for a day, then spend the next two days curriculum planning, trying to put the test out of mind Club Attendance Tracking The quick, easy, sensible solution! You’ve collected all of the tests and it looks like you have some work ahead of you (average score: 4 out of 10). Frustrated, you go home for the night and… Enforcement and Staff Impact To make sure we meet these new requirements, the service center will monitor clubs’ data on a regular basis and take appropriate actions to ensure data is entered accurately and timely Data Collection Process Club Attendance Tracking Selected Program Attendance Tracking Outcome Measurement Toolkit Administration Selected Survey and Test Administration Enforcement Reviewed by service center on an on-going basis to ensure data entered properly Reviewed by service center on an on-going basis to ensure data entered properly Reviewed by service center after toolkit administration period to ensure clubs are capturing appropriate % of members Reviewed by service center on a quarterly basis to ensure that program survey being administered to all participants - 24 - Staff Impact • These requirements are expected to be met by all staff members and will be a part of the performance review process • Clubs that are not in compliance will have a letter sent to the Club Director and Board President • Staff disciplinary action will be taken for non-compliance in accordance to existing policies and practices Requirements Summary Key Takeaways 1. The key to this program is data – make sure it’s captured accurately and timely! 2. Everything must be entered into KidTrax – no exceptions! 3. The Data Collection Requirement documents provide additional details, process flows, and best practices – review them for more information! Suggested Practices Make data collection a priority and try to get into a routine (i.e., enter data right as you get in or right as you leave Club Directors / Managers, review requirements with the staff and put it on the weekly agenda – Plan to schedule a staff meeting within the next week to review these requirements Many of these requirements and processes should be familiar, helping to make this a smooth transition - 25 - Test Your Knowledge – OBM Requirements 1. Batch attendance entry is permitted under all of the following normal scenarios EXCEPT A. Entering club attendance at a school-based club B. Entering club attendance at a stand-alone club C. Entering program attendance at a school-based club D. Entering program attendance at a stand-alone club Answer: B - 26 - Test Your Knowledge – OBM Requirements 2. Which of the following statements about program attendance is true? A. Program Instructors may enter attendance information up to a week after the session B. Batch scanning of program attendance is NOT permitted C. Program attendance must be entered into KidTrax for all club programs D. Instructors should utilize attendance sheets that are part of the program planning kit Answer: D - 27 - Test Your Knowledge – OBM Requirements 3. It is required that the Outcome Measurement Toolkit be administered to A. All members B. 50% of projected ADA C. 70% of projected ADA D. As many members as possible Answer: C - 28 - Test Your Knowledge – OBM Requirements 4. Compliance with these requirements may result in all of the following EXCEPT A. Increase in the number of club members B. Exemption from disciplinary actions related to OBM C. Annual incentives for staff members and clubs D. Positive marks on performance reviews Answer: A - 29 - Part 3: Immediate Action Items and Resources - 30 - CPS Member and Parent Consent Forms Each member who attends CPS will be required to fill out consent forms based on their age for us to obtain their CPS information Age <12 Ages 12 - 17 Age >18 Parent Consent Form Parent Consent Form Member Consent Form Member Assent Form Once completed, send forms to the service center with membership applications Please encourage members to complete and submit forms to the club within ten (10) days of their distribution - 31 - Student ID Number Collection Process Clubs must use school rosters and report cards from local CPS schools to allow staff to enter student ID numbers on the consent forms if they are not provided Student ID Number Collection Student ID Number Entry CPS student ID numbers can be found in three places: – Student Rosters – Student ID Cards – Report Cards Staff will need to assist members by entering their CPS student ID number on the consent form if they are not able to provide it Club Directors should work with school principals and administrators to collect student rosters – A letter will be provided by the service center to help explain the importance of the program and how it can assist the school in improving its students’ performance If available, clubs may also utilize report cards that were collected last year Note that this will be a one-time process and the service center will follow-up later in the fall to obtain ID numbers that are still missing - 32 - Survey Best Practices There are a number of best practices you can use for administering surveys and tests to members Prepare all materials prior to the scheduled survey. If using hard copies, keep in mind kids often respond better to surveys printed on colored paper Leave enough time to set up and arrange the room appropriately prior to the members’ arrival Introduce yourself to the members as they arrive and give a brief description of your purpose for being there and administering the survey. Sample script: – I am here to ask you to help make our clubs better. The survey your are about to take is being used to get your ideas about our club. It is not a test and there are no right or wrong answers — we want to know what your experiences and opinions are. Take your time and be sure to answer each question based on what you really think. If you don’t understand a question, please raise your hand or come find me and I can help you. The survey should take you only 8 to 10 minutes. (May differ depending on survey length).” Request that members attempt to answer every question so their opinions and experiences can be included Let members know you are here to help them if they have any questions While members are completing the survey, quietly walk around the room (do not hover), answer questions and make sure things are running smoothly When youth finish the survey, thank them for their time and provide incentives, if appropriate Source: The Colorado Trust, ” Toolkit for Evaluating Positive Youth Development” - 33 - BGCC Intranet Resources The BGCC Intranet will have many resources available, including BGCC Intranet Pathway: Downloadable Documents > Outcome-Based Management Program Resources Available Program Surveys / Tests (should be used if administering via paper copy) Outcome Management Toolkit Parental / Student Consent Forms Requirement Documents KidTrax Instruction Guides Presentations Other Forms and Guides This will be your one-stop shop for all things OBM! - 34 - Conference Call and Meeting Schedule To support the training you have today, we will hold a conference call and be present at the club manager meeting next week to answer any additional questions and concerns Conference Call Club Manager Meeting Tuesday, September 16 Wednesday, September 17 11:30 am – 12:30 pm 12:00 pm – 2:00 pm Dial In: 888-998-2663 Location TBD Passcode: 6235253 We will also distribute a Frequently Asked Questions (FAQ) document based on your feedback! - 35 - Contact Information Please contact the following people if you have any comments or questions regarding the program Primary Contact (Until September 19) Louis Levine (Deloitte), (312) 513-8166, llevine@deloitte.com Additional Trainers / Contacts George Colone (McCormick): (773) 271-8400 x 4307, gcolone@bgcc.org Khari Humphries (Holy Angels): (773) 268-4819, khumphries@bgcc.org Dawn Jimenez (Park Manor): (312) 520-6731, djimenez@bgcc.org Sidney Johnson (King): (773) 638-5464, sjohnson@bgcc.org Jim Pottinger (Cotter): (773) 348-1866, jpottinger@bgcc.org Erin Rochford (Valentine): (773) 927-7373 x4713, erochford@bgcc.org Juan Saenz (Logan Square): (773) 342-8800 x4210, jsaenz@bgcc.org LaTrice Smith (GSC): (312) 235-8007, lsmith@bgcc.org Katie Huckaby (GSC): (312) 235-8008, khuckaby@bgcc.org Ezohn Smith (GSC): (312) 235-8006, esmith@bgcc.org - 36 - Test Your Knowledge – Action Items 1. Members between the ages of 12 and 17 should receive which of the following forms? A. Student Consent Only B. Parental Consent Only C. Student Assent Only D. Parental Consent and Student Assent Answer: D - 37 - Test Your Knowledge – Action Items 2. Staff should administer surveys and tests related to the OBM program A. Printed out from the KidTrax Survey Manager B. Found on the BGCC intranet C. Provided by TEEN Reach coordinators D. Part of the Program Planning Kit Answer: B - 38 - Test Your Knowledge – Action Items 3. Which of the following is NOT a best practice for administering a survey? A. While participants are completing the survey, quietly walk around the room to make sure things are running smoothly B. Provide incentives for members to complete the survey C. Do not require members to complete all questions in the survey D. Prepare all materials prior to the scheduled survey Answer: C - 39 - Break - 40 - Part 4: KidTrax Training and Refresher - 41 - Manual Scanning (Trax Attendance Manager) Purpose: Track club attendance as soon as member enters / exits stand-alone club Situation: When member forgets his or her card and there is immediate access to Trax Attendance Manager at the front desk Responsibility: Front Door Volunteer / Assigned Club Member / Staff Member Refer to Manual Scanning Instruction Guide - 42 - Batch Attendance Tracking (KidTrax) Purpose: Track club attendance at school-based sites and program attendance after session completed by Program Instructor Situation: After Program Instructor runs session, will need to record attendance for the following programs: On-Going Programs Quarterly Programs – SMART Moves – SMART Girls – Career Launch – Power Hour – Street SMARTS – Youth for Unity – College Bound – Triple Play – Keystone Club – Passport to Manhood – Torch Club – Money Matters – Skill Tech / Digital Suite Responsibility: Program Instructor Refer to Batch Attendance Instruction Guide - 43 - Survey Setup (KidTrax Survey Manager) Purpose: Setup Outcome Measurement Toolkit, survey, or test to collect results from members in KidTrax Situation: Staff needs to setup survey / test for member to either take directly on the computer or be entered into KidTrax by a staff member Responsibility: Club Director / Program Director / Club Manager / TEEN Reach Coordinator / Program Instructor Refer to Survey Setup Instruction Guide - 44 - Survey Administration (KidTrax Survey Kiosk) Purpose: Administer Outcome Measurement Toolkit, survey, or test in KidTrax to collect results from members Situation: Staff needs to administer survey / test to members directly through the computer or access surveys and enter data on behalf of the members Responsibility: Club Director / Program Director / Club Manager / TEEN Reach Coordinator / Program Instructor Refer to Survey Administration Instruction Guide - 45 - Survey Results (KidTrax Survey Manager) Purpose: Track which members have completed a survey and review results of the assessment Situation: Staff needs to review this information during the testing period to ensure all members have completed the assessment and review results after testing completed Responsibility: Club Director / Program Director / Club Manager / TEEN Reach Coordinator / Program Instructor Refer to Survey Results Instruction Guide - 46 - Q&A - 47 -