Balancing Evaluation and Program Needs REAP Symposium 2009 Agenda • • • • • • Introduction EYA Organizational Overview Performance and Quality Improvement Division PQI Charge from the EYA Leadership Team Method: Cross-functional Teams Areas of Focus • Balanced Scorecard/ Key Performance Indicators • Master Quality Plan and Toolkit • Totally Integrated Electronic Record • Report Functionality • Integrating Processes • Q and A “It is one of the most beautiful compensations of this life that no person can sincerely try to help another without helping oneself.” ~Ralph Waldo Emerson A Proven History Pioneering “Firsts” 1994: One of First Private Providers of High Risk Programming for Girls in Florida. 2008: EYA Becomes Lead Agency Provider. 1968: First provider of an outdoor therapeutic program in Florida. 2001: Largest private provider of conditional release programming in Florida. 1982: First Private Provider of State Training School in Southeast. 1996: External Review of Data and Programs and Outcomes by Clemson and Harvard. 2007: Achieves agency-wide COA accreditation. 1991: Funded and Implemented Early Intervention Demonstration Project (Hi-Five). 1969: First provider of an outdoor therapeutic program for Girls in the Southeast. 1996: First Outdoor Therapeutic Programs to be Accredited by COA. 1985: Funded/Implemented Demonstration Project for ReEntry in Florida. January 27, 2009 3 Eckerd Youth Alternatives Mission: Develop and share programs that promote the well-being of children and serve at-risk youth and their families. Vision: Ensure that each child has the opportunity to succeed. EYA Values • • • • • • • • • • We base our concepts upon a belief in God and the uniqueness and inherent worth of each individual. We believe that youth always come first. We strive to assist youth in developing their potential to achieve balanced physical, intellectual, emotional, and spiritual strength. We believe that youth and their families are our partners in treatment. We seek to build a culturally diverse organization that honors the cultural diversity of the youth, families and communities we serve. We embrace innovation, flexibility, reflection, and continuous improvement. We thrive in a culture of learning, personal development and professional growth. We value all members of our staff and strive to provide the highest level of care and support to each. We hold ourselves accountable for achieving superior outcomes. We strive for excellence in all of our endeavors. Continuum of Services for Youth and Families Areas of Focus QUALITY Safety STAFF Quality Assurance Recruitment Targeted Performance FINANCIAL Professional Improvement Fiscal Strength Development & Planning EXTERNAL RELATIONS & Discipline Retention Evidence-Based Donor Cultivation GROWTH AND Accurate Analysis Practices & Outcomes TRANSFORMATION & Communications Forecasting & Mission-Driven Reinvestment Publications Capital Allocation & Business Viability Assessment Cash Alumni Management Relations Marketplace Analysis Targeted Retooling & Expansion January 27, 2009 8 Performance & Quality Improvement As a performance-based organization, we clearly define, collect, validate, analyze and communicate our outcomes to make informed decisions based on evidence Cross-functional Teams • • • Balanced Scorecard Master Quality Improvement Plan with Toolkit Data Collection (Totally Integrated Electronic Record) and Report Functionality Balanced Scorecard Team Summary Benefit Tasks The Balance Scorecard™ is a performance management tool that tracks the most important goals and initiatives at an organization. It measures financial and operational activities to ensure that they are aligned with the organizational mission, vision and strategies. Ensures consistency and thoughtfullness in the methodologies used to measure outcomes and provides feedback to all levels of the organization. • Prepare list of the different scorecards • Select the measurement criteria • Identify data source, instrument, collection, analysis, reporting for each measurement • Create communication tool Effort: Medium Impact: High Balanced Scorecard Basics • • • • • Developed by Robert Kaplan and David Norton (1990) Based on research exploring organizational performance measurement Communication Tool Measurement System Strategic Management System Balanced Scorecard Perspectives • Customer: Who are our customers? What do our customers expect or demand from us? What is our value proposition in serving them? • Internal Process: What must we excel at in order to continue to add value for our customers? What are the processes we have/need to best execute our strategies? • Employee Learning: What is the level/content of employee skills needed to meet our mission/value? What information systems needs to support these processes? What organizational climate (culture) supports this success? • Financial: What are the financial perspectives that would help us understand the effectiveness and efficiency of our financial processes? Balanced Scorecard Perspectives Balanced Scorecard Current Departments Customer/Client Operations Quality External Relations Employee Learning Staff Human Resources Internal Process Growth Development PQI Education Facilities Services Admissions Financial Financial Treasury/Risk Mgt Accounting Performance Measure Criteria • • • • • • • Linked to strategy Easily understood Link in a chain of cause and effect Updated frequently Accessible Measurable (averages, avoiding initiatives) Is it driving the “right” behavior? Evaluation Plan Sample Evaluation Plan Balanced Scorecard Examples • Customer: 1) Percentage of youth completing an EYA program increasing mental health functioning by at least 10% 2) Percentage of youth completing an EYA program reporting continued positive wellbeing impact (defined by education, mental health, behavioral, including recidivism, employment, and living status.) • Internal Process: 1) Number of instances of TIER being “off-line” during normal working hours 2) Cost of fundraising expenses incurred by EYA in a fiscal year will be at least $1/$1. • Employee Learning: 1) Voluntary employee turnover will be maintained at 50% or less. 2) Percentage of participants’ managers finding an increase in productivity, initiative, confidence, or measure directly related to the subject of the training. • Financial: 1) Overall administrative overhead is less than 10%. 2) Total revenue minus total expense is at least 3% of total revenue. Master QI Plan Team Summary A comprehensive document that establishes organizational strategies for identifying critical areas, assessing, developing and implementing quality assurance objectives and procedures in a consistent and methodical way. Benefit Allows us to deliver top quality care to our clients, provide excellent, reliable data and information to our internal and external customers and measures outcomes for continuous quality improvement. Tasks • Assess current systems and tools • Cross functional needs assessment • Define requirements, resources and deliverables • Implementation strategies that include training and ongoing support. Effort: High Impact: High Data and Reporting Team Summary Benefit Tasks Several key tools are used to track and report the quality monitoring and response information, including PBViews and TIER computer systems. This initiative will re-deploy these systems and train the individuals who will be using the systems. Appropriate tools to capture and report large amounts of data is critical in the monitoring of a diverse group of programs in multiple locations. Having effective tools will minimize the time required by staff and leadership to report and analyze quality data. • Deploy software with clear expectations on who and when data will be entered • Engage vendor to re-deploy TIER system • Create a training plan for all users of the different systems to ensure they are comfortable using the system Effort: Medium Impact: Medium Summary Information Management Transactional Details Business Users Portal/Intranet Everyone PBViews Sample 18 Canned Reports Performance “View” BIRT/e.Spreadsheet BIRT Report Studio Briefing Books Internet Publishing Maps Annual Descriptive Summary Youth Outcomes • • • • • • Satisfaction Survey measures perceptions of accountability, consequences, education, empathy, group, home/transition, staff 35% youth and 14% parent respondent rate – not in parity with EYA demographics Respondents are only from residential programs – primarily NC 64% of youth completing an EYA program reported positive satisfaction of services 65% of parents of youth completing an EYA program reported that their child made positive life changes 62% of parents reported that their relationship with their child improved Youth Outcomes • • • • • • • WJII: Measures math (calculation, math fluency, applied problems) and reading (letter-word identification, reading fluency, passage comprehension) Average length of stay ranges from 6.2-11.3 months Youth receive a minimum of 250 instructional minutes of math and 250 instructional minutes of reading/week 90% youth respondent rate (Pre-Post Test) At intake, 53% were below grade level for reading and 79% were below grade level for math 82% of youth completing an EYA program improved reading levels an average of 1.6 grade levels 78% of youth completing an EYA program improved math levels an average of 1.2 grade levels Youth Outcomes • • • • • • • Matson’s Evaluation of Social Skills for Youngsters 78% youth respondent rate (Pre-Post Test) Average length of stay ranges from 2.7-11.3 months Demographics are in parity with EYA population 42% of youth completing an EYA program improved social skills by an average of 15%. Less than 20% of youth in ReEntry or community supervision programs scored in the normal range for “appropriate” social skills. In this population, aggression and overconfidence scored high as well. More than 80% of you in private academies and day schools scored in the normal range for “appropriate” social skills. In this population, impulsivity scored high as well. Youth Outcomes • ASEBA Youth Self Report measures mental health functioning • Average length of stay ranges from 2.7-11.3 months • 82% youth respondent rate (Pre-Post Test) • Demographics are in parity with EYA population • 47% of youth completing an EYA program scored in the normal range at the beginning of the program and 72% scored in the normal range at the end of the program • Other than the commonality of clinical scores in rule breaking and externalizing behaviors, youth varied greatly in other constructs Integrating into Practice • Safety: evaluating frequency and areas of vulnerability, incident cost metrics, promoting effective strategies & impact, cross-functional teams, organizational safety program • QI: educating organization, formalizing processes and systems, creating communication strategies with accountability, updating policies/procedures, refining Internal Program Reviews • PI: fine tuning performance outcomes, evaluating current methodologies and solidifying data collection, web-enabling instruments, analyzing data, interpreting and reporting results, collaborative research and data sharing, Internal Program Reviews Reporting Findings Tool Development Scheduling & Team Selection Evaluation of Findings Enhancement Plan/Follow-UP Written Report January 27, 2009 Internal Review Process Post Review Conference Pre-review Conference Program Review 29 Next Steps • • • • • • Collaboration/Partnering with other providers and researchers Sharing methodologies and approaches Understanding the impact of outcomes and how it influences current issues and training Disseminating information about programs effectiveness and innovations Designing protocols based on best evidence Identify model programs and replicating success Q and A Opportunity is missed by most because it is dressed in overalls and looks like work. ~Thomas Edison Maurie Lung, LMHC, LMFT Eckerd Youth Alternatives, Inc./Life Adventures Counseling & Consulting PO Box 10907, St. Petersburg, FL 33733 727-481-4577 dmlung@lifeadventurescc.org Greg Holeman, Chief Quality Officer Eckerd Youth Alternatives 100 N. Starcrest Dr., Clearwater, FL 33765 727-461-1236 X437 gholeman@eckerd.org