Youth Program Quality Assessment
Youth PQA Basics
Center For Youth Program Quality
1
Agenda
•
•
•
•
•
•
Welcome
Opening: Intro to the Youth PQA
Observational Note Taking
Fitting & Scoring
Building Your Pyramid
Self Assessment Q&A
The Quality Counts
Initiative
Why Quality Counts, Who We Are, & Where We’re Going
WHY? Because Rhetoric and Reality Don’t Match
The American Dream
All youth ready, every family
and community supportive,
each leader effective.
The American Reality
Only 4 in 10 youth ready, only 1 in 3 youth
supported, too few leaders effective.
The American Dilemma
Fragmentation, complacency, and low expectations of
youth, communities and leaders
The Ready by 21™ Challenge
Change the odds for youth by changing the way we do business
4
…There is Increasing Agreement on Skills Needed for the 21st Century…
Specific Vocational
Knowledge
& Skills
21st Century
Skills & Content
Information & Media Literacy * Creativity *
Intellectual Curiosity * Critical & Systems
Thinking * Accountability and Adaptability *
Communication * Problem Solving *
Interpersonal Skills * Social Responsibility *
Financial Literacy * Global Awareness * Civic
Literacy * Self-Direction
Cultural,
Physical & Behavioral
Health Knowledge
& Skills
Ready for Life
Youth Development Outcomes
Subject
Matter
Knowledge
Researchers Agree on What It Takes to
Support Development
The National Research Council & Institute for
Medicine list the following key features of
positive youth development settings:
•
•
•
•
•
•
•
•
Physical and psychological safety
Appropriate structure
Supportive relationships
Opportunities to belong
Positive social norms
Support for efficacy and mattering
Opportunities for skill-building
Integration of family, school and community efforts
- Community Programs to Promote Youth Development, 20026
Longitudinal Studies Confirm that these
Supports Make a Difference
7
Providing these Supports Can Change the Odds
Gambone/Connell’s research suggests that if all young people got the
supports they needed in early adolescence, the picture could change…
from 4 in 10 doing well
to 7 in 10 doing well*
.
8
Systems for Quality
Accountability Policies in Places
• YPQA is part of state and county accountability policies:
–Cross sector (DHS& DOE) snapshots: Iowa, Washington, Arkansas
–Statewide 21st Century: Michigan, Maine, Minnesota, Rhode Island, New Mexico,
–Cities and Counties: Rochester, Detroit, Grand Rapids, Palm Beach
Rochester
Grand
Rapids
Minneapolis
Washington*
Chicago
New York
etroit
Iowa
Indianapolis
mbus
Georgetown
Divide
Columbus
St. Louis
Oklahoma
Rhode
` Island
`
`
Kentucky
Nashville
l
Austin
West Palm Beach
County
9
WHO WE ARE - 3 Nationals, 5+ States, 7 Localities
STATES (w/ participating localities)
• Iowa
Linn County, N. Central Iowa, Polk
County/Des Moines
• Kentucky
Lexington, Louisville
• New York
Broome, Onondaga, Orange &
Rockland Counties
LOCALITIES
•
•
•
•
•
•
•
Austin, TX
Georgetown Divide (Black Oak Mine), CA
Columbus (Bartholomew County), IN
Grand Rapids, MI
Indianapolis, IN
Nashville, TN
St. Louis, MO
• Oklahoma
NATIONAL PARTNERS
Norman, Tulsa
• Rhode Island
Central Falls, Newport, Pawtucket
• Washington State
(Honorary QC member)
• The Forum for Youth Investment
•
The Center For Youth Program Quality
(formerly High/Scope Youth Development Group)
• AED National Training Institute for
Community Youth Work
10
WHERE WE’RE GOING - Four Domains
The Quality Counts initiative is designed to help you maximize your capacity to
improve the quality and reach of youth programs by taking concrete steps in
four areas (domains) to ensure that you have:.
Capacity to Assess & Improve Programs
Strong Policy / Leadership Horsepower
A Strong & Stable Program Base
Capacity to Recruit, Train, Retain Workforce
11
Why bother paying attention to youth
program quality?
• Network leaders build awareness and
communication
• Programs see their strengths and areas to
improve, receive critical feedback, plus
improve cmty networking and collaboration
• Funders gain an important metric
• Youth can have better program
experiences
12
What is program quality?
inputs
inputs
???
outcomes
inputs
youth program
Another way to say it:
What do we want to see in high quality youth programs?
13
High POS Quality
Several ways to organize:
Readin’ + ‘Ritin + ‘Rithmatic (old-school)
Affect + Active Learning + Metacognition (Education)
Relationships + Relevance + Rigor (Education 2.0)
Relatedness + Autonomy + Competence (Psychology SDT)
Safety + Belonging + Esteem (Psychology old-school)
8 Features of Positive Youth Development Settings (PYD)
Relationship + Task + Increasing Complexity
Content
Therapeutic
process
The Pyramid of Program Quality
Plan
Make choices
Engagement
Reflect
Lead and mentor
Be in small groups
Partner with adults
Experience belonging
Interaction
Encouragement Reframing conflict
Supportive
Skill building
Session flow
Active engagement Welcoming atmosphere
Environment
Psychological and emotional safety
Program space and furniture Emergency procedures
Safe
Healthy food and drinks
Physically safe environment
Environment
Youth Voice and Governance
Professional Learning Community
15
What is the Youth PQA?
1. A validated instrument
designed to assess the quality
of youth programs and identify
staff training needs.
Engagement
Interaction
Supportive Environment
2. A set of items that
Safe Environment
measures youth access
to key developmental experiences.
3. A tool which produces scores that can be used for
comparison and assessment of progress over
16
time.
The Youth PQA and Maslow
B-Needs
Actualization
D-Needs
Esteem
Needs
Love/
Belonging
Needs
Engagement
Interaction
(youth-youth; peer community)
Supportive Environment
(adult-youth relationships)
Safety
Needs
Maslow
categories
Safe Environment
PQA Subscales (within pyramid)
17
Scores From Validation Studies
Validation Study:
Outside Observer
(N=140 offerings)
Self-Assessment
Pilot
(N=24 orgs)
Safe environment
4.35
4.39
Supportive environment
3.75
4.16
Opportunities for interaction
3.11
3.73
Engagement
2.83
3.37
(N=51 orgs)
(N=24 orgs)
Youth centered policies and
practices
3.92
3.20
High expectations for
students and staff
3.86
3.90
Access
3.86
4.18
Program Offering Level
Organization Level
18
Youth PQA Domain Scores
N= 735 offerings (all unique staff) in 180 organizations
5
4.38
4
3.72
2.97
3
2.53
2
1
I. Safe
Environment
II. Supportive
Environment
III. Interaction
Opportunities
IV. Engaged
Learning
Contrasting Pedagogy Profiles
N=599 offerings in 120 organizations
welcome
5
belong
Quality
4
learning
group
3
choices
2
planning
reflect
1
PYD I
5 = occurred for everyone
3 = occurred for some
1 = did not occur
Staff Cent I
Low Qual II
How we think about DDCI
- People change not programs
Prochaska, J.O., & DiClemente, C.C. (1982). Transtheoretical therapy toward a more integrative model of change. Psychotherapy:
Theory, Research and Practice, 19(3), 276-287.
Maintenance
Repeat
cycle
Action
POS
Point Of Service
Implementation
& coaching
PLC
Professional
Learning Community
Preparation
Planning with data
Contemplation
Quality assessment
Individual Change
Model
8/13/2008
SAE
System Accountability
Environment
Organizational
Context
The Center for Youth Program
Quality
21
Designing Quality Improvement Systems (QIS)
Mostly Mangers
System
External quality
Capacity
assessment (a)
Managers and direct staff
Use of on-line
dashboards and
training (d)
External Quality Report with Norms
Quality
Advisor (e)
TOTs for quality
assess, coaching,
and youth work
methods (f,g)
Coaching & Training
Self-Assessed Quality Report
Program
SelfStaff Skill assessment
& Knowl
of Quality (b)
Use of on-line
dashboards and
training (d)
Planning
with Data (c)
Phase 1: Readiness &
Capacity
Targeted youth
work methods
training for
direct staff (h)
Quality
coaching by
managers (i)
Phase 2: Impact &
Sustainability
22
Defining the Purpose of Your QIS
Lower Stakes
Higher Stakes
(the creative middle)
Program
Self-Assessment
External
Assessment
Rough data to
get staff thinking and
discussing program
quality in the context
of best practice
Precise data for
internal and external
audiences for evaluation,
monitoring, accountability,
improvement, reporting
Less time
Less money
Impact internal
audiences
More time
More money
Impact internal and
external audiences
23
Columbus Indiana
Phase 1: Building Local Capacity
POS
Point
Of Service
SAE
System
Accountability
Environment
STEP 1
Decide to
build system
PLC
STEP 2a
Selfassessment
STEP 2b
Professional
Learning
Community
STEP 3
STEP 4
STEP 5
Plan for
improvement
Carry out
plan
Measure
change
External
assessment
August 25
Quality Matters
Presentation
August 26
Annually
Youth PQA Basics
October 8
Opt Phase 2
January 27-28
Planning with Data
Improvement Plan
Method Workshops
Quality Coaching
Ext Assessment
Program SA
Ext Assessment
Observe-Reflection
Planning with Data
Step 1: Pilot Sites Attend TrainingYouth PQA Basics, External Assessor
Step 2: Pilots Select Self-Assessment
Teams and Develop Schedule
26
Step 3: Self Assessment
27
Step 4: Submit Scores
28
High/Scope’s eTools Web Site
etools.highscope.org
Your one stop shop for:
– Online training
• Intro to Youth PQA
• Intro to Scores Reporter
– Online assessment
• Submitting Youth PQA Scores
• Generate reports
Online PQA Scores Reporter
etools.highscope.org
3 levels of users
-Youth Program Facilitators
-Youth Program Directors
-Network Users
Online PQA Scores Reporter
Enter Data with mouse or keyboard
Online PQA Scores Reporter
View & Print Reports
Your programs
scores can
automatically be
compared to
national norms
Online PQA Scores Reporter
View & Print Reports
Step 5: Teams attend Planning with Data
and create Program Improvement Plans
34
Step 6: Staff members work the plan!
Repeat Youth PQA annually…
35
Does it Work? Findings from Several
Samples
• POS quality-outcomes findings:
– Supportive environment related to: Attendance
– Interaction related to: Interest in program
– Engagement related to: Sense of challenge, sense of growth,
school-day reading, school-day suspension
– Note: No offerings get to high engagement without high support
and high interaction
• Quality Improvement (YPQI) Findings
– Scores increase from pre to post
– Scores increase in the targeted areas more
– Management practices are related to quality change (Vision,
Feedback, Continuity)
Questions about…
Purpose?
Process?
Pilot?
Next Steps…
37
Youth Program Quality Assessment
38
Structure of the Youth PQA
Form B
Organizational Interview
Form A
Observation
Ask questions, write, score
(2 hours)
Watch, write, score (3 hours)
Program Offering 1
Program Offering 2
Organization
Program Offering 3
Program Offering 4
39
“subscale”
Sample indicator
III. Interaction
“item”
III-L. Youth have opportunities to develop a sense of belonging.
Note: Structured refers to the quality of being intentional, planned, and/or named; it does not refer to informal conversation.
Indicators
1 Youth have no
opportunities
to get to know
each other
(beyond selfselected pairs
or small
cliques).
Supporting Evidence
3 Youth have
informal
opportunities to
get to know each
other (e.g., youth
engage in
informal
conversations
before, during, or
after session.
5 Youth have structured
opportunities to get to
know each other (e.g.,
there are teambuilding activities,
introductions, personal
updates, welcomes of
new group members,
icebreakers, and a
variety of groupings
for activities)
“indicator row”
The Youth PQA consists of
2 scales (Form A & Form B); 7 subscales (4 in A, 3 in B)
30 items (18 in A, 12 in B); 103 indicator rows (60 in A, 43 in B)
40
Indicator Scramble
What are the key words?
What are the key quantities?
What are we counting?
41
Sample Anecdotes
• Youth sort of identified with program
offering. III-L (3rd row)
• Kids sometimes taunt each other and try
to annoy each other, but they seem to get
along, too. I-A (1st row)
• Kevin dominated committee meetings he
sat in on. Did better job of listening and
acknowledging comments in the large
group. III-O (1st row)
42
Observational Note Taking
Effective
Ineffective
• Objective as possible
• Rich detail in snapshot form
• Focus on interactions between
• Subjective terms such as ‘good’ or
‘bad’
• Rater’s opinions
• Assumptions about internal states:
she felt angry; he did not get it
• Anecdotes are too vague; lack detail
• Lacking facts: what you see and hear
• Summary in the place of quotes
• Raters repeat what the indicator says
• For a 3 involving some positive and
some negative, raters have one but
not the other
• Anecdote does not fit the indicator
• Anecdote could support more than
one score
– Staff and youth
– Youth and youth
– Youth and environment
• Allow time for interactions to reach
completion
• State the outcome of interactions
• Who, what, when, where
• Quotes: what youth and staff say
• Lists of materials
• What you see in the room
• Anecdote can stand alone
43
44
Follow-up questions
• Only ask for items with question(s) in the
evidence column.
• Ask the questions as they are printed.
Form A items with at least one “Ask” indicators (24 total)
I-C
I-D
II-K
III-L
indicator rows 1-6
indicator row 4
indicator rows 1-4
indicator row 1
III-N
IV-P
IV-Q
IV-R
indicator rows
indicator rows
indicator rows
indicator rows
2-3
1-2
1-2
2-4
45
Fit and Score (label practice)
1. Try to place each label in the correct row for
each item. Look over the anecdotal evidence
for each item and see how the anecdotes are a
“fit” with the indicator rows.
2. Score each indicator row. For each row, read
through the levels of the indicator row and
select the level - 1, 3, or 5 - that most closely
agrees with your evidence. Pay close attention
to words like some, most, and always. Start at
level 5 to see the full description of necessary
evidence and check agreement with the
anecdotal evidence by looking to the lower
46
levels.
More on Scoring the YPQA
• Fitting data to the indicator rows can
eventually happen in your head while you
are observing.
• Always try to see multiple items in every
interaction and cross-reference constantly.
• Look to a preponderance of evidence but
favor higher scores.
• If you lack evidence with “fit,” collect more
data.
47
Assessing POS Quality using Youth PQA
Let’s Try It! Watch, Write, Score
• Collecting objective anecdotal evidence
• Scoring rubrics
• Reliability
Instructions
– In this activity you will watch a short (4 minute) video clip, take notes,
and score a few indicators based on the video.
– Before you watch the video, get ready:
• Make sure you have your hardcopy of Form A of the Youth PQA in
front of you for this activity.
• Make sure you have scrap paper and a pen or pencil handy
• Review items II-J, III-M, and IV-Q (you will be scoring one indicator
from each)
• While you watch the video take notes by hand. Make sure your
notes are as objective as possible. Try to capture actual quotes.
48
More on Self-Assessment
• Self-assessment only works if program (direct)
staff are involved.
• Try to get a good mix of program times and
offerings.
• Assessment is different from evaluation.
Assessment gives you the opportunity to see
where you are, whereas evaluation suggests
judgment.
• Remember, the conversation is the most
important part of self-assessment.
49
Youth PQA Step by Step
SelfAssessment
STEP
External Assessment
Team that includes as many
frontline staff as possible
1. Select
Assessor(s)
Use anchored rater if
possible
When will data be collected?
Who watches whom?
2. Plan the Data
Collection
Check time & schedule
Make sure frontline staff
knows you’re coming
3. Collect Data
1-2 hr visit
Take notes
Ask follow-up questions
4. Fit & Score
Fit & score each indicator
5. Generate Report
Enter indicator scores into
Scores Reporter
6. Plan for
Improvement
Use the report to make plans
for items to work on
50
Staff observe each other in 1015 minute chunks & take
notes
Have a meeting. Go indicatorby-indicator, sharing
evidence and coming to
consensus on each score
Enter indicator scores into
Scores Reporter
Use the report to make plans
for items to work on
How will you use this data to promote the right kinds of
Accountability Behaviors?
Engage
Survive
Me
Comply Avoid/Resist
Focus
Us
Low
High
Commitment
51