Balancing Evaluation and
Program Needs
REAP Symposium 2009
Agenda
•
•
•
•
•
•
Introduction
EYA Organizational Overview
Performance and Quality Improvement Division
PQI Charge from the EYA Leadership Team
Method: Cross-functional Teams
Areas of Focus
• Balanced Scorecard/ Key Performance Indicators
• Master Quality Plan and Toolkit
• Totally Integrated Electronic Record
• Report Functionality
• Integrating Processes
• Q and A
“It is one of the most beautiful compensations of this life that no
person can sincerely try to help another without helping oneself.”
~Ralph Waldo Emerson
A Proven History
Pioneering “Firsts”
1994: One of First Private Providers of High Risk Programming for Girls in Florida.
2008: EYA Becomes Lead Agency Provider.
1968: First provider of an outdoor therapeutic program in Florida.
2001: Largest private provider of conditional release programming in Florida.
1982: First Private Provider of State Training School in Southeast.
1996: External Review of Data and Programs and Outcomes by Clemson and Harvard.
2007: Achieves agency-wide COA accreditation.
1991: Funded and Implemented Early Intervention Demonstration Project (Hi-Five).
1969: First provider of an outdoor therapeutic program for Girls in the Southeast.
1996: First Outdoor Therapeutic Programs to be Accredited by COA.
1985: Funded/Implemented Demonstration Project for ReEntry in Florida.
January 27, 2009
3
Eckerd Youth Alternatives
Mission: Develop and share programs that promote the well-being of
children and serve at-risk youth and their families.
Vision: Ensure that each child has the opportunity to succeed.
EYA Values
•
•
•
•
•
•
•
•
•
•
We base our concepts upon a belief in God and the uniqueness
and inherent worth of each individual.
We believe that youth always come first.
We strive to assist youth in developing their potential to achieve
balanced physical, intellectual, emotional, and spiritual strength.
We believe that youth and their families are our partners in
treatment.
We seek to build a culturally diverse organization that honors the
cultural diversity of the youth, families and communities we serve.
We embrace innovation, flexibility, reflection, and continuous
improvement.
We thrive in a culture of learning, personal development and
professional growth.
We value all members of our staff and strive to provide the highest
level of care and support to each.
We hold ourselves accountable for achieving superior outcomes.
We strive for excellence in all of our endeavors.
Continuum of Services for Youth and Families
Areas of Focus
QUALITY
 Safety
STAFF
 Quality Assurance

Recruitment
 Targeted
Performance
FINANCIAL
 Professional
Improvement
 Fiscal
Strength
Development
& Planning
EXTERNAL
RELATIONS
&
Discipline

Retention

Evidence-Based

Donor
Cultivation
GROWTH
AND
 Accurate
Analysis
Practices & Outcomes
TRANSFORMATION
&
Communications
Forecasting
&
 Mission-Driven
Reinvestment
Publications
 Capital
Allocation &
 Business Viability Assessment
 Cash
Alumni
Management
Relations
 Marketplace Analysis
 Targeted Retooling
& Expansion
January 27, 2009
8
Performance & Quality Improvement
As a performance-based organization, we clearly
define, collect, validate, analyze and communicate
our outcomes to make informed decisions based on
evidence
Cross-functional Teams
•
•
•
Balanced Scorecard
Master Quality Improvement Plan with Toolkit
Data Collection (Totally Integrated Electronic
Record) and Report Functionality
Balanced Scorecard Team
Summary
Benefit
Tasks
The Balance Scorecard™ is a performance management tool that tracks the
most important goals and initiatives at an organization. It measures financial
and operational activities to ensure that they are aligned with the organizational
mission, vision and strategies.
Ensures consistency and thoughtfullness in the methodologies used to measure
outcomes and provides feedback to all levels of the organization.
• Prepare list of the different scorecards
• Select the measurement criteria
• Identify data source, instrument, collection,
analysis, reporting for each measurement
• Create communication tool
Effort: Medium
Impact: High
Balanced Scorecard Basics
•
•
•
•
•
Developed by Robert Kaplan and David Norton (1990)
Based on research exploring organizational performance measurement
Communication Tool
Measurement System
Strategic Management System
Balanced Scorecard Perspectives
•
Customer: Who are our customers? What do our customers
expect or demand from us? What is our value proposition in
serving them?
•
Internal Process: What must we excel at in order to continue to
add value for our customers? What are the processes we
have/need to best execute our strategies?
•
Employee Learning: What is the level/content of employee skills
needed to meet our mission/value? What information systems
needs to support these processes? What organizational climate
(culture) supports this success?
•
Financial: What are the financial perspectives that would help us
understand the effectiveness and efficiency of our financial
processes?
Balanced Scorecard Perspectives
Balanced Scorecard Current
Departments
Customer/Client
Operations
Quality
External Relations
Employee Learning
Staff
Human Resources
Internal Process
Growth
Development
PQI
Education
Facilities Services
Admissions
Financial
Financial
Treasury/Risk Mgt
Accounting
Performance Measure Criteria
•
•
•
•
•
•
•
Linked to strategy
Easily understood
Link in a chain of cause and effect
Updated frequently
Accessible
Measurable (averages, avoiding initiatives)
Is it driving the “right” behavior?
Evaluation Plan
Sample Evaluation Plan
Balanced Scorecard Examples
•
Customer: 1) Percentage of youth completing an EYA program
increasing mental health functioning by at least 10% 2)
Percentage of youth completing an EYA program reporting
continued positive wellbeing impact (defined by education,
mental health, behavioral, including recidivism, employment, and
living status.)
•
Internal Process: 1) Number of instances of TIER being “off-line”
during normal working hours 2) Cost of fundraising expenses
incurred by EYA in a fiscal year will be at least $1/$1.
•
Employee Learning: 1) Voluntary employee turnover will be
maintained at 50% or less. 2) Percentage of participants’
managers finding an increase in productivity, initiative,
confidence, or measure directly related to the subject of the
training.
•
Financial: 1) Overall administrative overhead is less than 10%. 2)
Total revenue minus total expense is at least 3% of total revenue.
Master QI Plan Team
Summary
A comprehensive document that establishes organizational strategies for
identifying critical areas, assessing, developing and implementing quality
assurance objectives and procedures in a consistent and methodical way.
Benefit
Allows us to deliver top quality care to our clients, provide excellent, reliable
data and information to our internal and external customers and measures
outcomes for continuous quality improvement.
Tasks
• Assess current systems and tools
• Cross functional needs assessment
• Define requirements, resources and
deliverables
• Implementation strategies that include
training and ongoing support.
Effort: High
Impact: High
Data and Reporting Team
Summary
Benefit
Tasks
Several key tools are used to track and report the quality monitoring and
response information, including PBViews and TIER computer systems. This
initiative will re-deploy these systems and train the individuals who will be using
the systems.
Appropriate tools to capture and report large amounts of data is critical in the
monitoring of a diverse group of programs in multiple locations. Having effective
tools will minimize the time required by staff and leadership to report and
analyze quality data.
• Deploy software with clear expectations on
who and when data will be entered
• Engage vendor to re-deploy TIER system
• Create a training plan for all users of the
different systems to ensure they are
comfortable using the system
Effort: Medium
Impact: Medium
Summary Information
Management
Transactional
Details
Business Users
Portal/Intranet
Everyone
PBViews Sample
18 Canned Reports
Performance “View”
BIRT/e.Spreadsheet
BIRT Report Studio
Briefing Books
Internet Publishing
Maps
Annual Descriptive Summary
Youth Outcomes
•
•
•
•
•
•
Satisfaction Survey measures perceptions of
accountability, consequences, education, empathy,
group, home/transition, staff
35% youth and 14% parent respondent rate – not in
parity with EYA demographics
Respondents are only from residential programs –
primarily NC
64% of youth completing an EYA program reported
positive satisfaction of services
65% of parents of youth completing an EYA program
reported that their child made positive life changes
62% of parents reported that their relationship with
their child improved
Youth Outcomes
•
•
•
•
•
•
•
WJII: Measures math (calculation, math fluency, applied
problems) and reading (letter-word identification, reading
fluency, passage comprehension)
Average length of stay ranges from 6.2-11.3 months
Youth receive a minimum of 250 instructional minutes of
math and 250 instructional minutes of reading/week
90% youth respondent rate (Pre-Post Test)
At intake, 53% were below grade level for reading and
79% were below grade level for math
82% of youth completing an EYA program improved
reading levels an average of 1.6 grade levels
78% of youth completing an EYA program improved
math levels an average of 1.2 grade levels
Youth Outcomes
•
•
•
•
•
•
•
Matson’s Evaluation of Social Skills for Youngsters
78% youth respondent rate (Pre-Post Test)
Average length of stay ranges from 2.7-11.3 months
Demographics are in parity with EYA population
42% of youth completing an EYA program improved
social skills by an average of 15%.
Less than 20% of youth in ReEntry or community
supervision programs scored in the normal range for
“appropriate” social skills. In this population,
aggression and overconfidence scored high as well.
More than 80% of you in private academies and day
schools scored in the normal range for “appropriate”
social skills. In this population, impulsivity scored high
as well.
Youth Outcomes
•
ASEBA Youth Self Report measures mental health
functioning
•
Average length of stay ranges from 2.7-11.3 months
•
82% youth respondent rate (Pre-Post Test)
•
Demographics are in parity with EYA population
•
47% of youth completing an EYA program scored in the
normal range at the beginning of the program and 72%
scored in the normal range at the end of the program
•
Other than the commonality of clinical scores in rule
breaking and externalizing behaviors, youth varied
greatly in other constructs
Integrating into Practice
•
Safety: evaluating frequency and areas of
vulnerability, incident cost metrics, promoting
effective strategies & impact, cross-functional
teams, organizational safety program
•
QI: educating organization, formalizing processes
and systems, creating communication strategies
with accountability, updating policies/procedures,
refining Internal Program Reviews
•
PI: fine tuning performance outcomes, evaluating
current methodologies and solidifying data
collection, web-enabling instruments, analyzing
data, interpreting and reporting results, collaborative
research and data sharing,
Internal Program Reviews
Reporting
Findings
Tool
Development
Scheduling
& Team
Selection
Evaluation of
Findings
Enhancement
Plan/Follow-UP
Written
Report
January 27, 2009
Internal
Review
Process
Post
Review
Conference
Pre-review
Conference
Program
Review
29
Next Steps
•
•
•
•
•
•
Collaboration/Partnering with other providers and researchers
Sharing methodologies and approaches
Understanding the impact of outcomes and how it influences
current issues and training
Disseminating information about programs effectiveness and
innovations
Designing protocols based on best evidence
Identify model programs and replicating success
Q and A
Opportunity is missed by most because it is dressed in
overalls and looks like work. ~Thomas Edison
Maurie Lung, LMHC, LMFT
Eckerd Youth Alternatives, Inc./Life Adventures Counseling &
Consulting
PO Box 10907, St. Petersburg, FL 33733
727-481-4577
dmlung@lifeadventurescc.org
Greg Holeman, Chief Quality Officer
Eckerd Youth Alternatives
100 N. Starcrest Dr., Clearwater, FL 33765
727-461-1236 X437
gholeman@eckerd.org