Measuring Results
by Barbara Schneider and Nicole Cheslock
A project of The Coevolution Institute
Prepared for the EE Learning Community
July 14, 2003
Purpose of Measuring Results
„
To gain insights into how to best affect and
measure behavior change through a cross
field analysis of current evaluation practice
„
To provide program stakeholders with
information on how different fields have
approached the challenge of affecting and
assessing behavior change
Methods behind Measuring Results
„
Cross field study
„
„
„
„
Health
Social Marketing
Museums
Environmental Education
„
Academic literature review
„
Peer review
Health - Overview
„
„
„
„
Strong theoretical base
Evaluation dominated by experiments and
field studies
Successful programs are tailored to the
participants and use multiple strategies
Successful programs target and teach
specific behaviors
Health - Theoretical Frameworks
„
Health Belief Model
„
Social Learning Theory
„
CDC Evaluation Framework
Health - Methods
„
Case studies
„
Experiment and quasi-experimental designs
„
Listening techniques
„
Tracking
Health - Findings
„
Individual characteristics impact intervention
outcomes
„
Same sex and similar role models are salient
(SLT, research on edu-tainment)
„
Behavioral skills training is effective
„
Multiple strategies have positive results
Health - Implications
EE Learning Community Implications
„
Tailor to your audience
„
Use multiple techniques
„
Teach skills: Build awareness, knowledge, and
experience in targeted behaviors
Health - Resources
Online Resources
„
„
Centers for Disease Control and Prevention
http://www.cdc.gov/eval/
EC/UNFPA Initiative for Reproductive Health in
Asia
http://www.asia-initiative.org/m_and_e_2002.html
Social Marketing - Overview
„
„
„
„
Social marketing aim to benefit society at large
Behavior modification theories and cost benefit
models shape social marketing work
Evaluation methods include case studies,
authentic assessments, SROI
Effectiveness based on opinion leaders, credible
message providers, support/technical
assistance, communication
Social Marketing - Theoretical
Frameworks
„
Antecedents, Moderators, Consequences
„
Cost Benefit Analysis
Social Marketing - Methods
„
Case studies
„
Media and Authentic assessments
„
Social Return On Investment (SROI)
Social Marketing - Findings
„
Opinion leader
„
Multiple credible sources
„
Intensive personal support
„
Means of communication - Spreading the
message(s)
Social Marketing - Implications
EE Learning Community Implications
„
„
„
„
Strong leadership/modeling
Clear and credible communication
Ongoing support/communication of
message(s)
Building evaluation/assessment components
into programming
Social Marketing - Resources
Online Resources
„
„
„
Agency for Healthcare Research and Quality
http://www.ahcpr.gov/chip/content/outreach_enrollm
ent/outreach7.htm
Roberts Enterprise Development Fund
http://www.redf.org/
Weinreich Communications
http://www.social-marketing.com/
Museums - Overview
„
„
„
„
„
Theoretical move toward contextual models
of learning
Observation / time and tracking studies
Move toward accounting for visitor variables
Learning does take place!
Each individual / group has a unique
experience within the same setting
Museums - Theoretical Frameworks
Much of what is known about museum experiences has
been influenced by a view of learning that is determined by
the degree to which the intended curriculum is understood.
This view of learning has expanded to include sociocultural models.
„
Contextual Model of Learning
(Falk and Dierking, 2000)
„
Conversational Elaboration
(Leinhardt & Crowley, 1998)
Museums - Methods
„
Observation
„
Listening
„
Experimental designs
„
Longitudinal
„
Challenges: Duration/frequency of visit(s),
determining causal links, likelihood that future
experience will trigger full impact of experience,
access to visitors
Museum - Findings
„
Visitor & exhibit characteristics
„
„
„
„
Family behavior
„
„
„
Each visitor has a unique experience
Demographic characteristics: years of education
Interactive exhibits builds intrinsic motivation, promotes learning
Predictable: fairly equal amounts of time at each exhibit, distinct
phases including orientation, viewing, cruising, departure
Children interact more frequently with hands-on exhibits
Field Trips
„
„
Most effective when there is preparation/novelty reduction
Are educational/professional development experiences for classroom
teachers
Museums - Implications
EE Learning Community Implications
„
„
„
Each person and group has a unique learning
experience
Agenda and expectations shape learning
experiences
Orientation to setting, clear learning objectives,
and follow-up have been influential in fostering
learning experience for youth
Museums - Resources
Online Resources
„
„
Association of Science Technology Centers
http://www.astc.org/resource/index.htm
Museum Learning Collaborative
http://museumlearning.com
Environmental Education Overview
„
„
„
„
Move from “Knowledge-Attitude-Behavior” (KAB)
model to more complex frameworks
Traditional emphasis on quantitative data
collection tools moving toward use of mixed methods designs
Weak link between theory and practice
Multiple factors lead to responsible
environmental actions
QuickTime™ and a
Photo - JPEG decompressor
are needed to see this picture.
Environmental Education Theoretical Frameworks
Knowledge
„
„
Attitude
Behavior
Knowledge (or understanding of environmental issues as a result of
becoming educated about them) leads to greater awareness (and to
a change in attitude), which leads to more responsible behavior with
respect to the environment.
The empirical links between knowledge, attitudes, and behavior are
tenuous: Some researchers believe there is a strong correlation;
others suggest weak connections.
Environmental Education Theoretical Frameworks
Move from “K-A-B” model to more complex frameworks
„
Reasoned Action and Planned Behavior (Ajzen & Fishbein, 1986)
Emphasizes intention
„
Activator–Behavior–Consequence (Geller, 1992)
Emphasizes activators
„
Responsible Environmental Behavior (Hines et al 1986)
Emphasizes affective/personality, cognitive, and
situational characteristics
Environmental Education - Methods
Evaluation Methods
„
„
Self-reported information (through interviews,
questionnaires, surveys)
Observation, consumption habits, and authentic
assessments
„
Multiple methods
„
Inconsistencies and weaknesses are pervasive
Environmental Education - Findings
There are many studies that indicate the
importance of individual variables but on the
whole there is no systematic process across
the studies to make definitive statements
about the variables.
It is clear, though, that individual
characteristics are important and play a role
in people’s experiences and in how they learn.
Environmental Education - Findings
Findings - What shapes and impacts
responsible environmental behavior (REB)?
„
„
„
„
„
„
Environmental sensitivity
Locus of control / self-efficacy
Knowledge of behaviors
Skill in behavior
Prompts
Demographic factors: gender and years of formal
education
(Chawla, DeYoung 1996, Hines et al., 1986/87; Hwang et al., 2000; Marcinkowski,1998)
Environmental Education-Implications
EE Learning Community Implication(s)
„
Fostering REB includes targeting specific behaviors.
„
Fostering REB includes teaching those behaviors in
a context that is appropriate to your audience.
Tailor!
„
Fostering REB includes building audience’s self
efficacy.
Environmental Education - Resources
Online Resources
„
Interpreting Biodiversity: A manual for environmental educators in
the tropics. (Domroese & Sterling. 1999)
http://research.amnh.org/biodiversity/center/publ/pubdwnld.html
„
Nature Conservancy Handbook, Framework for Site Conservation
http://www.consci.org/scp/download_the_handbook.htm
„
North American Association for Environmental Education
http://www.naaee.org/
„
City of Toronto Assessment & Policy Development Office Individual ecological footprint http://www.city.toronto.on.ca/eia/index.htm
Key Findings and Implications of
Measuring Results
Effective Strategies for Changing Behavior
„
Theories of Change
„
Target Specific Behaviors
„
Tailor Strategies
„
Promote Self Efficacy
Key Findings and Implications of
Measuring Results
Effective Evaluation Strategies
„
Create Measurable Objectives
„
Use Multiple Methods Design and
Authentic Assessments
„
Systemize and Disseminate
For more information about
The Coevolution Institute,
the Measuring Results report, and
upcoming Measuring Results Symposia,
Please contact
info@coevolution.org
or call 415-362-1137