CAPP Evaluation
August 2011 Update
Jane Powers, Amanda Purington, Jenny Parise
ACT for Youth Center of Excellence
Overview
• Review Evaluation Plan:
Documenting EBP Implementation
• Ways to improve documentation
• Revisions to the Tools
• Next Steps
CAPP Initiative
Goal 1: Promote healthy sexual behaviors
and reduce the practice of risky sexual
behaviors among adolescents
Core Strategy 1: Provide
comprehensive, age appropriate,
evidence-based and medically accurate
sexuality education to promote healthy
sexual behaviors
We know a lot about what works to
prevent teen pregnancy
•Increase age of first
intercourse
EBP
•Increase use of condoms
•Decrease # sexual partners
•Decrease frequency of sex
Decrease
Teen
Pregnancy
Promote
Adol Sexual
Health
What does it take to implement
EBPs in real world settings???
Research to Practice
While many EBPs have yielded positive outcomes
in research settings, the record at the local level
of “practice” is mixed (Wandersman, 2009;
Lesesne et al, 2008; Fixsen, 2005).
What do we know about
Implementation?
• Implementation influences program outcomes
• If EBPs are not implemented with fidelity and quality,
not likely to result in outcomes observed in research
• Achieving good implementation increases chances of
program success and stronger benefits for participants
(Durlak and DuPre, 2008)
Factors Affecting Implementation
•
•
•
•
•
Community Level
Organizational Capacity
Program Characteristics
Training and TA
Facilitator Characteristics
CAPP
An Opportunity to Learn!!!
Opportunity to learn:
•
How do EBPs work in different settings?
• How do EBPs work with diverse youth populations?
• What adaptations are needed to make EBPs work
more effectively?
• Are EBPs successful in getting to outcomes?
Documenting Implementation is critical!
• Helps us learn how the intervention was
delivered
• Helps us understand challenges faced
• Helps us learn about what does and does
not work re implementing EBPs
Tools to evaluate Implementation
• Fidelity Checklist: document what you did,
adaptations made, successes/challenges
• Attendance Record: who, where, when, how
much
Role of the Fidelity Checklists
Role of the Fidelity Checklists
• Document:
–
–
–
–
What you did
Timing
Adaptations made and why
Challenges and successes
Role of the Fidelity Checklists
Help you Prepare!!
Using Fidelity Checklist to Plan
• Review activities
• What equipment is needed?
• Timing
• Anticipating issues
• Questions that might come up
• Prepare for next cycle
What have we learned?
Ways to Improve Documentation
1. Provide More Details!
– WHAT was changed?
– WHY was it changed?
GREAT example
“We took the entire class period to implement and
process this activity. Although it is timed at 15 minutes,
it took ten minutes alone to break the class of 32
students into groups, hand out cards and explain the
activity. We also needed more time to allow groups to
respond as we physically had more groups than if we
were working in a small group setting.”
Good example
“Due
to the age of the participants and the material
in the Robert Townsend video clip, the teacher did
not find it to be appropriate for the classroom.
Instead we discussed the information in the clip
and used a short role-playing exercise to make the
information clear.”
– Why wasn’t it appropriate?
– What was the role-playing exercise?
Need more details
“ran out of time”
• Tells us WHY but not WHAT was changed
• Was the activity shortened (if so, how?) or skipped
altogether?
“switched the order of these activities”
• Tells us WHAT the change was but not WHY it was
made
2. Document ALL adaptations
Was Activity Carried Out According to Direction
in the Facilitator’s Curriculum?
•Leave something out
•Change order
•Add new content
•Shorten activity
•Substitute activity
•Modify activity
2. Document ALL adaptations cont.
– Include info on pre-approved adaptations
– Were NO adaptations at all made??
– Example
• NO adaptations made
• General Comment: “It would have been nice to
have more time for each activity/module”
– Were activities completed according to the curriculum?
– Were some activities omitted or shortened?
3. Complete the Fidelity Checklist
right away!
– Capture this information when the
experience of the module and the activities
are fresh in your mind
4. Complete the right number of tools
One set of tools per cycle per set of participants
Class 1
Fidelity Checklist
Attendance Record
Class 2
Fidelity Checklist
Attendance Record
Class 3
Fidelity Checklist
Attendance Record
Class 4
Fidelity Checklist
Attendance Record
Class 5
Fidelity Checklist
Attendance Record
Teacher 1
Teacher 2
Teacher 3
Fidelity Checklist
Attendance Record
Common Adaptations
• Common adaptations include:
– Updating videos, or using something more relevant in
place of videos
– Changes to accommodate timing issues
– Changing language/names to be more inclusive
• How can the COE support these common
adaptations?
– Providing/linking to updates on medical information
– Identifying alternative videos (updated)
– Developing alternatives to the condom demonstration
Documentation Details
• Block names of individual participants
• Do not use acronyms – spell these out
• Write legibly
• Review your documentation for typos
•
beforehand
Bundle all evaluation materials for each cycle
(fidelity checklists and attendance records)
and mail to:
Amy Breese
Cornell University
ACT for Youth Center of Excellence
Beebe Hall
Ithaca, NY 14853
Submit tools electronically
• Save the fidelity check list and attendance
record under a different name and input
the information
• Preserve the template for future use
• Send the documents to Amanda Purington
(ald17@cornell.edu)
Revisions to the Fidelity Checklists
1. Clarified the adaptations description field:
“If Changed, WHAT was changed and WHY? Please be
specific: describe things you left out, added, or
changed and WHY.”
2. Added “Were changes (if any) pre-approved?”
3. Added a comments section after each module:
“Please use this space if you have comments on this
module or any of its activities:”
Revisions to the Attendance Record
1. Added “Age” column for participants
Next Steps
• Additional tools:
– Participant Feedback Form
– Facilitator Feedback Tools
– Pre/Post Tests
– Other needs?
• Tools and presentations on
www.actforyouth.net - Search for “CAPP”
in the site search box at the top of the
home page.
• Revise Tools for Year 2
Contact Information
Amanda Purington: ald17@cornell.edu
607-255-1861
or
Jane Powers: jlp5@cornell.edu
607-255-3993