Quality in Afterschool
April 28, 2015
Agenda
1.
2.
3.
4.
Welcome
Context setting
Research
Believe It – Build It: MN’s Guide to Effective
Practices in Afterschool
5. Quality Assessment and Process Examples
6. Funder Table Talks
7. Funder Table Talk Report Back, Commitments
and Questions
Context Setting
Sheila Oehrlein
Minnesota Department of Education
Research on Quality
Deborah Moore & Kari Denissen Cunnien
University of Minnesota & Ignite Afterschool
The Youth Work Learning Lab Presents…


The Youth Work Learning Lab - School Of
Social Work

Hard for youth workers to measure
and collect (time, resources and
expertise)

Can’t help determine changes
needed in program or by staff

Hard to use many tools across
diverse program environments

Difficult for programs to evaluate
causal information

Different funding sources require
different measures

Using poorly designed tools

Tools are too expensive

Tools don’t help staff be more
clear about what to “do”

Not enough learning and training
support

Other accountability “burdens”
never reduced – keep piling on
The Youth Work Learning Lab - School Of
Social Work

Data should help us understand something we do not know,
but it is better if it relates to our practical experience

Data should not be a full time job for
youth programs

Data should be easy to use for change

Data is only one way of understanding the world, others
matter too…
The Youth Work Learning Lab - School Of
Social Work

Based on solid, consistent and
compiling research.

Looks at settings and/or behaviors vs.
youth internal states (i.e. harder to
measure youth outcomes).

Based on things where we have higher levels of control (our
practices).

Based where we have the most commonality across the diverse
ways we do our work (how we work with youth, not content).
The Youth Work Learning Lab - School Of
Social Work
Researchers Agree on
Program Quality Elements
The National Research Council & Institute for Medicine list
the following key features for quality programs:
 Physical
and psychological safety
 Appropriate structure
 Supportive relationships
 Opportunities to belong
 Positive social norms
 Support for efficacy and mattering
 Opportunities for skill-building
 Integration of family, school and community efforts
- Community Programs to Promote Youth Development, 2002
• Empirical evidence linking quality
with positive impact
(Durlak, 2007; Vandell, 2005; 2007; 2009; 2011;
Sambrano, 2005 Catalano, 2005, Eccles & Gootman,
2002; Roth, Brooks-Gunn, Murray &
Foster, 1998).
• Negative consequences of poor quality programs
(U.S. Dept. of Health, 2006; Belle, 1999).
• If we assess program quality, we can improve it.
(Smith, 2012).
Creating the space for young people to make decisions
and have a choice about things that matter to them.
http://www.youtube.com/watch?v=Z2Z17g-2rIc
The Youth Work Learning Lab - School Of
Social Work
BELIEVE IT. BUILD IT.
Kari Denissen Cunnien
Ignite Afterschool
We Believe Statements
Building Blocks
Continuous Program
Improvement Cycle
Intentional
Program
Design
Supportive
Relationships
&Environment
Youth Voice
& Leadership
Responsiveness
to Culture &
Identity
Community
& Family
Engagement
Organizational
Management,
Staff Support &
Youth Safety
Safe & Active, Focused & Explicit
Are these effects “meaningful”?
• Aspirin on heart disease, d = .03
• Early child care & math achievement, d = .09-.12
• Class size reductions on math achievement, d = .23
Durlak, J.A., & Weissberg, R.P. (2010). American Journal of Community Psychology
Above charts modified from a presentation by Deborah Vandell
A community approach
Gambone, M.A., Klem, A.M. & Connell, J.P. (2002). Finding Out What Matters for Youth: Testing Key Links in a Community Action Framework
for Youth Development. Philadelphia: Youth Development Strategies, Inc., and Institute for Research and Reform in Education.
Human Development Approach
Human development is defined as the process of enlarging people’s freedoms and
opportunities and improving their well-being. Central to the human development
approach is the concept of capabilities. Capabilities—what people can do and
what they can become— are the equipment one has to pursue a life of value.*
“the quality of a human life involves
multiple elements whose
relationship to one another needs
close study….one of the appealing
features…is its complexity: it
appears well equipped to respond
to the complexities of human life
and human striving.”**
*Measure of America, A Project of the Social Science Research Council. Retrieved from: http://www.measureofamerica.org/humandevelopment/
**Nussbaum, Martha C. (2011) Creating Capabilities: The Human Development Approach.
Continuous Program Improvement
Youth Program Quality Assessment
(YPQA)
Deborah Moore
University of Minnesota
1.
A validated instrument designed to
assess the quality of youth programs
at the point of service (POS--where
youth and adults interact) and identify
volunteer/staff training needs.
2.
A set of items that measures youth
access to key developmental
experiences.
3.
A tool which produces scores that can
be used for comparison and
assessment of progress over time.
4.
Identifies staff training needs and
encourages improvement and
creation of a professional learning
community
 Youth
self-reports of:
o Belonging
o Interest
o Challenge
o Learning
o Efficacy
o Communication
o Empathy
o Problem Solving
Pick a
card and
have a
chat
about
why that
practice
matters
to young
people…
o
Observational assessment is scary AND affirming for youth workers, tread with care.
o
The YPQA is useful as a self-assessment process where staff observe their own
program or when external assessors observe (but these are not the same).
o
Assessment is the Beginning – for the data to have power, it must be placed in the
context of continuous quality improvement (emphasis on improvement) and data
must get in the hands of those that work directly with youth.
o
There are advantages for using one tool (creates common language across staff and
programs, allows for community to aggregate data, focuses shared training on mutual
improvement goals across programs, etc.)…but some choice can also be good.
o
Quality assessment works BEST if it is about learning and improvement. Funders
help when they focus on encouraging participation and improvement.
o
It takes time- to change habits, to build organizational capacity
The Youth Work Learning Lab - School Of
Social Work
To download the YPQA free of charge go to:
Weikart Center for Youth Program Quality- Download the YouthPQA or
http://www.cypq.org/downloadpqa
For info on classes and consulting using the YPQA go to:
Deborah Moore - Director
Youth Work Learning Lab
University of MN – Peters Hall
1404 Gortner
St. Paul, MN 55108
612-625-7813 OR
ddm2@umn.edu
Academic Skills PQA
Kara Bixby & Nora Robinson
Saint Paul Public Schools Foundation
Youth Program Scenario
Extended day learning class for 6th
graders.
 Conducting an architecture project to
promote learning of STEM concepts.
 Objective is to learn about angles and
proportions.
 Group discussion: What will you have the
class do to learn these skills in a high-quality
way?

31
 2014 Saint Paul Public Schools Foundation
Key Questions
How do we help youth programs provide
high-quality academic opportunities?
 What do they look like in out-of-school
time?
 What practices are most important?

32
 2014 Saint Paul Public Schools Foundation
Defining Academic Skill-Building
Academic
Support
Academic
=
LEARNING
SkillBuilding
Intentionality
Overview of the Tool
Measures the quality of academic skillbuilding opportunities.
 Expansion of skill-building scale in Youth
Program Quality Assessment (YPQA).
 Alignment with Youth Program Quality
Intervention (YPQI).
 Research-based and validated.
 Assists in identifying areas where staff
practice can be improved.

34
 2014 Saint Paul Public Schools Foundation
Settings for Use
Programs that provide academic supports
for youth
 21st Century Community Learning
Centers
 Schools
 Extended day learning opportunities
 Camps and museum programs
 Collective impact models with academic
goals

35
 2014 Saint Paul Public Schools Foundation
Why Use?
36

Geared toward program improvement
and performance management.

Focused on practices shown to be
effective for academic skill-building.

Offers a higher standard of performance.

Helps programs show the research-based
practices they are using.
 2014 Saint Paul Public Schools Foundation
Why Use? (cont.)
37

Includes items specific to social and
emotional learning.

Focused on skill development.

Additional point of information and data
for academics.

Prepares young people to benefit from
school/core instruction.
 2014 Saint Paul Public Schools Foundation
Additional Information

Download tool at Weikart Center
website:
 http://www.cypq.org/downloadpqa

38
Twin Cities info session to be scheduled
Fall 2015
 2014 Saint Paul Public Schools Foundation
Kara Bixby
Research & Evaluation Director
kara.bixby@sppsfoundation.org
651-325-4210
Nora Robinson
Tutoring Partnership Director
nora.robinson@sppsfoundation.org
Questions?
39
651-325-4244
 2014 Saint Paul Public Schools Foundation
Quality Mentoring Assessment Path
(QMAP)
Mai-Anh Kapanke
Mentoring Partnership of Minnesota
Quality Mentoring
Assessment Path
(QMAP)
MILESTONES…
2007
Conceptual work with a Quality
Advisory Committee (QAC)
2008-’09
Tool Development and QAC beta
testers
2009-’10
36 programs piloted - officially
launched October 2010
2013
QMAP expansion - Connecticut,
Nebraska and Washington
2014
QMAP expansion - Maryland
2015:
Ohio enters agreement to use
QMAP
WHAT IS QMAP?
QMAP is:
QMAP is not:
 Continuous improvement
 Accreditation process
 Built around a cycle
 One-time evaluation
 Accommodates varying
program models
 Used with start-up
programs
 Guided by quality
assurance goals
 A way to rank or score
programs
 Confidential
 Public reporting system
QMAP ONLINE TOOL
Divided into three sections
 Program Design & Management
 Program Operations
 Program Evaluation
RESOURCES
 Links to free
resources for
every item
 Includes research
and templates
ON-SITE REVIEW
 2 - 4 hours
 Develop priorities for
Improvement &
Innovation Plan with
program staff, not just a
score or ratings
 Plan becomes the
platform for technical
assistance with individual
programs
FOLLOW-UP
Summary Packet sent:
 Cover letter
 Final Improvement &
Innovation Plan
 Statement of
Accuracy
 QMAP Benefits
Overview
QUALITY PARTNER RECOGNITION
 Dedicated Partners
Complete the
QMAP process
 Expert Partners
100% of outlined best
practices in place
“MESSAGES FROM THE FIELD”
 Smaller programs are honored to receive the
same level of recognition as their peers from
larger programs
 It’s a great way to share program information
with staff, Boards and top leadership of multiservice organizations
 QMAP helps programs leverage funding
 QMAP builds the field toward quality
 When programs have best practices in
place, our families, kids and
volunteers benefit
Expanded Learning Solution Plan
Jaimee Bohning
Northside Achievement Zone
NAZ SEAL
OF EFFECTIVENESS
• Partners following evidencebased Solution Plans
• Track real-time results for
programs and individuals
• Blueprint for replication
RESULTS NAZ
ROUNDTABLE ANALYSIS OF DATA
Collective learning and performance accountability
• Monthly convening
• Includes senior NAZ staff, Board members, Parent Advisory Board
members, and rotation of partner leaders
• Analysis to find technical and adaptive solutions
• Building culture of accountability: partners are accountable to each
other and to families
HIGH TOUCH,
HIGH TECH
PARTNERSHIP
EXPANDED LEARNING
SOLUTION PLAN
• AGREED UPON RESULT:
“Expanded Learning enrolled scholars
demonstrate accelerated progress to achieve
grade-level performance in math and reading.”
EXPANDED LEARNING
SOLUTION PLAN
Solution Plan “buckets”
• Organization Commitment
• Reading Skills
• Math Skills
• Supporting Scholar Engagement
• Transition Preparation and Support
• Continuous Quality Improvement
• Partnerships with Schools and Community
Organizations
• Family Engagement
EXPANDED LEARNING
SOLUTION PLAN
• We aim to answer three key questions:
– How much?
– How well?
– Is anybody better off?
What is working and how can we do more of it?
Focus on scalability and replication
What isn’t working?
Why not?
Is it still important? If so, how do we improve?
If not, how do we cease resource allocation?
Table Talk
1.RFP Application Process
2.Site Visits
3.Communicating quality to your Stakeholders
Table Talk Report Back
1.What were key highlights from each table?
2.What will you take away from today and bring
back to your organization?