Assessing & Improving
Youth Program Quality
Nicole Yohalem,
Forum for Youth Investment
Quality Youth Development Practice
Webinar
April 2, 2009
Why focus on quality?
• Quality matters
• Quality is measureable (we can “know it”)
• Quality is malleable (we can “grow it”)
Quality Matters
While improving youth outcomes is the goal we are all working
toward, we believe that to get there at scale, our focus needs to
be on helping leaders improve the quality and reach of
programs and services available in the community.
Quality Matters
There is basic agreement that young people
need structured, voluntary activities to fill their
out-of-school hours.
– The good news: There is now strong evidence
that these programs can produce positive change
in things we all care about – academic
achievement, social and emotional development.
– The bad news: Many programs do not.
Quality Matters
Why do only some programs make a
difference?
– Controlling for participation, young people who participate
in high quality programs achieve greater gains than those
who do not.
– In 2007 meta-analysis, Durlak and Weissberg grouped 73
programs into 2 clusters based on quality features.
Programs with those features showed positive effects on
almost every outcome – school performance, social
behavior, attitudes and beliefs. Programs that did not have
the SAFE features showed no effect on any outcome.
Quality Matters
In programs that score well on the YPQA, young
people are more likely to have:
•
•
•
•
Improved program attendance
Improved school-day reading
Reduced school-day suspension
Self-reported interest in program, sense of challenge and
sense of growth
Key Point: Programs can improve outcomes by
improving the quality of their services.
Quality is Measurable
•
Assessing Afterschool Program Practices Tool (APT)
National Institute on Out-of-School Time and the MA Department of Education
•
CORAL Observation Tool (CORAL)
Public/Private Ventures
•
Out-of-School Time Observation Instrument (OST)
Policy Studies Associates
•
Program Observation Tool (POT)
National Afterschool Association
•
Program Quality Observation (PQO)
Deborah Vandell and Kim Pierce
•
Promising Practices Rating Scale (PPRS)
Measuring Youth Program Quality
A Guide to Quality Assessment Tools
Updated January 2009
WI Center for Education Research and Policy Studies Associates, Inc.
•
Quality Assurance System (QAS)
Foundations Inc.
•
Program Quality Self-Assessment Tool (QSA)
New York State Afterschool Network
•
School-Age Care Environment Rating Scale (SACERS)
Frank Porter Graham Child Development Center, UNC
•
Youth Program Quality Assessment (YPQA)
High/Scope Educational Research Foundation
Quality is Measurable
There is a lot of similarity across
definitions. Common elements include:
–
–
–
–
–
–
Relationships
Environment
Engagement
Social Norms
Skill Building Opportunities
Routine/Structure
Quality is Measurable
There are some differences among tools,
however:
– Emphasis on the point-of-service vs. program
administration.
– The extent to which measures actually teach
– The extent to which measures are clear and
unambiguous
Quality is Malleable
Can these aspects of quality be improved?
Once we know it, can we grow it?
YES
Youth Program Quality Intervention
Systemic quality improvement systems anchored by the YPQA are being developed in:
–States: MI, ME, RI, KY, NM, AR, MN, IA, WA, NY
–Cities/Counties: Austin, Chicago, Rochester, Detroit, Grand Rapids, Palm Beach County,
Baltimore, Nashville, St. Louis, Louisville, Georgetown Divide/Sacramento, Columbus IN,
Indianapolis IN, Tulsa OK
Seattle
Rochester
Minnesota
Grand
Rapids
Minneapolis
Washington*
Chicago
New York
etroit
Iowa
Indianapolis
Sacramento/
Georgetown
Divide
l
mbus
Columbus
New
Mexico
Oklahoma
Maine
St. Louis
Kentucky
Arkansas
Nashville
Rhode
` Island
`
`
Baltimore
Austin
West Palm Beach
County
YPQI Focus: POS quality in context
Youth PQA Form A
POS
Point-of-Service
PLC
Professional
Learning Community
SAE
System
Accountability
Environment
Engagement
Interaction
Support
Safety
Youth PQA Form B
•Org policies/practices
•Management values
•Performance feedback
•Continuity/staffing
•Standards and metrics
•Staff development
Quality is Malleable
Components of effective quality improvement
systems:
– Quality standards that address what should happen at the
point of service
– Ongoing assessment of how well services compare to the
standards
– Targeted plans for how to improve
– Training and technical assistance (including on-site
coaching) that fits those plans
Quality Improvement: Lessons Learned
1.
2.
3.
4.
5.
Have well defined purposes for the system.
Focus on the point-of-service.
Anchor improvement efforts with data about POS.
Create incentives for continuous improvement.
Build in on-site, ongoing technical
assistance/coaching.
6. Be intentional about pilot participation.
7. Build learning communities.
8. Recognize that management is a key lever.