Peering into the black box
between pre and post:
A theory of change
related to interactive/
experiential delivery
methods
Susan Carter
Evaluation Coordinator
National Indian Youth Leadership Project
susanleecarter@comcast.net
with
J. Fred Springer, EMT Associates
fred@emt.org
A presentation for the 6th Annual Research
and Evaluation in Adventure Programming
Symposium
Gaithersburg, MD
March 17-19, 2010








48 site quasi-experimental design
10,500 youth
4 point longitudinal study
Common instrument
Comprehensive process data
Hierarchical statistical design
American Evaluation Association Study of the
Year award, 2000
Largest direct service ATOD prevention
evaluation ever implemented
The Multi-Site High Risk Youth Study
funded by SAMHSA
Fred Springer, EMT, Co-PI
Common Factors Reducing
30-day Drug Use
0 .1 8
.18
0 .1 3 5
Effect Size
.135
0 .0 9
.09
0 .0 4 5
.045
0
C o nne ct io n B uilding
B e ha viora l L ife- S kills F ocu se d
Life-skills
Focused
Intro sp ec tive Le arnin g
P rogra m C ohe ren ce
Connection
Building
Coherent
Inte ns e C ont ac t
Introspective
Orientation
Intense
Contact
4





Promoting introspection / connectedness are
critical common factors
Out of school more effective
Manualized programs less effective
Intensity counts
Promoting behavioral skills more effective
than focus on knowledge and attitude change
HRY Common Factor Findings

Project Venture identified as one of eight
best programs in HRY study
◦ PV included all of the common factors
Incorporates connectedness building and
introspection
 Uses interactive experiential program
delivery strategy

Application: Project Venture

Population

Program Components
◦ Middle school aged American Indian youth
◦ High school aged AI service staff
◦ Mix of risk levels (at risk + high risk)
◦
◦
◦
◦
◦
◦
Experiential education
Connecting to the natural world
Physical and social-emotional challenge
Service learning
Focus on developing positive relationships
Positive Youth Development approach
Essential Elements of Project
Venture





Experiential, adventure-based positive
youth development program for American
Indian youth
Recognized by NREPP as evidence-based
The only American Indian ATOD
prevention program currently in NREPP
Widely disseminated throughout the US
and Canada
Twenty year implementation and
evaluation history
Project Venture: Overview

Outdoor, adventure activities:
ropes course, rappelling, canoeing,
backpacking, camping, mountain biking, etc.

Service Learning
Meaningful projects that are youth centered
and designed, address cultural,
environmental and other community needs.
In a research-supported, year long sequential
program.
Project Venture: Strategies
Positive substance abuse prevention
findings (prevent, reduce, delay onset)
especially related to alcohol abuse
 Increased internal assets
 Increased external assets, especially
community and peer domains

Project Venture: Evaluation
Findings

We propose to advance knowledge
concerning how adventure based
strategies and activities can be
strengthened, promoted, and replicated.
Purpose of proposed study

Interactive /experiential activities that
clearly incorporate reflective learning
skills and promote connectedness to
positive people and environments will
promote intended behavioral change more
effectively than interactive/experiential
activities that have less focused emphasis
on these interim outcomes.
Hypothesis
Quasi-experiment
 Baseline, exit, six and twelve month
follow up
 Assess similar participants in Project
Venture programs:

◦ 6 programs using an enhanced version of PV
◦ 6 “regular” PV programs
◦ (perhaps 6 groups with no PV)
Method notes
Behavioral outcomes (ATOD, etc.)
 Connectedness (e.g., Michael Karcher’s
Hemmingway scales)
 Reflective learning (perhaps adapted from
education literature)

Measures
Rigorous development and testing
program to develop the measures
 Rigorous statistical methodology including
propensity scores for improving
comparison equivalence
 Hierarchical modeling to account for
nesting effects, treatment interactions,
treatment settings
 Strong fidelity/coherence measurements

Analysis notes
Characteristics of Most
Successful Prevention
Programs
How Project Venture Incorporates these Characteristics through
Interactive Experiential Delivery Strategies
Engagement
Cultural content, leadership, values / experiential learning / connecting to
the natural world / physical and emotional challenge
Intensity
150 + hours year (3 + hours per week)
68 sessions / In & after school, weekends, summer
Reflective Orientation
Experiential learning cycle, debriefing; Outdoor adventure: problem
solving; Service learning: student led, problem solving; Camp, treks:
journaling, values, norms, natural world, spiritual awareness; Native
cultural values: spiritual traditions; In school: Problem solving, skill
building, socialization
Connectedness Building
Outdoor adventure: team building, trust building; Service learning: youth
led, meaningful, relationship emphasis; service ethic; Camp, treks: build
caring, cooperative community; Native cultural values; family events,
cultural traditions
Life-skills Focused
Outdoor adventure: cooperation, leadership, efficacy ; Service learning:
problem solving, leadership, cooperation; Camp, treks: leadership, efficacy;
In school: problem solving, skill building
Positive Environment
Cultural content; adult participation, interaction, trust, support;
Coherence
Clear purpose and process; training; collaboration; implementation project
Community Support
Guided by Native Elders; Native cultural values; positive youth
development approach; collaboration.
High Risk Youth Study, SAMHSA mapped onto Project Venture
Secondary data analysis of 5 years of
pre/post survey data from Project Venture 7th
grade participants (600) and matched
comparison group (400) collected between
2005-2010
AND
 Longitudinal follow-up study of PV program
participants who were in 7th grade in 2006 –
including original program survey (adapted
for older youth cohort) AND life history
interviews

Other study ideas