Collaborative Evaluation
Rita O'Sullivan
John O’Sullivan
Evaluation, Assessment, &
North Carolina A & T State Univ
Policy Connections (EvAP)
School of Education
Univ. of NC - Chapel Hill
ritao@unc.edu (919/843-7878)
NC Cooperative Extension
Greensboro, NC
johno@ncat.edu-(336/285-4683)
Evaluation Steps
Overview
• Introduction to Collaborative Evaluation
• Collaborative Evaluation Techniques:
Collaborative Proposal Development
Clarifying Evaluation Requests
Designing Collaborative Evaluations
Evaluation Planning
Collaborative Data Collection
Collaborative Data Analysis
Convincing Levels of Evidence
Collaborative Reporting: Evaluation Fairs
Collaborative Evaluation Cycle
• Final Questions/Discussion
Starting Definition: COLLABORATIVE
EVALUATION
•
is an approach that engages program
stakeholders actively in the evaluation
process. When stakeholders
collaborate with evaluators, their
understanding increases and the utility
of the evaluation is often enhanced.
Collaborative Evaluation as an
Evaluation Approach
•
Participatory Evaluation
(e.g., Cousins, 1996; Patton, 1997)
•
•
•
Empowerment Evaluation
(e.g., Fetterman, 2001; 2005)
Collaborative Evaluation
(e.g., O’Sullivan, 2004; O’Sullivan
& O’Sullivan, 1998; Liliana
Rodriguez-Campos; 2005)
Characteristics of a
Collaborative Evaluation
•
Stakeholders share responsibility for the
evaluation
•
Evaluation questions are developed together
•
Evaluation expertise grows among board
members, parents, teachers, service
providers and other key stakeholders
•
Evaluation evidence is used for program
improvement
Multi-Site Annual Sequence
•
•
•
•
•
•
•
Individual Site Evaluation Planning
Feedback on Evaluation Plans
Eval Plan Implementation with Technical
Assistance
Interim Sharing of progress
Technical Assistance as Needed
Evaluation Fair Sharing
Cross-Site Summaries
Advantages to Collaborative
Evaluation
•
•
•
•
•
Enhances the dynamics of eval, changing
paradigm shift (interactions of ownership)
External evaluators learn program
dynamics
Stakeholders understand the purposes of
the evaluation
Data collection and reporting are of
better quality
Access to additional evaluation resources
Characteristics of a
Collaborative Evaluator
•
•
•
•
Good listener
Flexibility
Understanding of evaluation
Knowledge of the program
Collaborative Evaluation Techniques:
Developing an Evaluation Proposal
•
•
•
•
•
•
Program Background Statement
Evaluation Purpose
Evaluation Questions with
Proposed Information Gathering
Strategies
Timeline of Evaluation Activities
Capability Statement
Budget
Collaborative Evaluation Techniques:
Clarifying an Evaluation Plan
•
Gathering information about the
programs nature and scope
•
Determining the purpose of the
evaluation
•
Probing the resources available to
conduct the evaluation
Mentoring Project
• Your are about to meet with the
Director of the Mentoring Project
• You are an external evaluation
consulting group, and she has asked for
your help.
• Get together in small groups to decide
on three questions to ask her that will
clarify her evaluation needs.
Collaborative Evaluation Techniques:
Evaluation Planning - Logic Models
GOALS &
OBJECTIVES
1. To
educate
teen
parents
about good
parenting.
ACTIVITIES
150 teen
parents in
parenting
classes
Provide
health clinics
RESULTS
(Outcomes)
SMART:
Specific
Measurable
Attainable
Realistic
Time-bound
Improve
parenting
practices
(90% of
participants)
MONITORING
EVIDENCE
(Outputs)
(Service Stats)
+ Quality of
Activities
Attendance
lists
Class Evals/
Observations
OUTCOME
EVIDENCE:
Immediate
Intermediate
Long-Term
RETRO
Pre/Post
Parenting
Practices
Surveys
Retrospective Pre/Post
• Ask respondents to tell you at the
end of the program, where they were
when they started and where they
are now.
Program Theorizing/Theory of
Change (Identify Indicators)
• Long Term: Healthy children ready for school
• Intermediate Outcome 1: Parents able to assist
children with pre-school skills
• Intermediate Outcome 2: Program participants
interact appropriately with children
• Immediate Outcome 2: Participants enhance
understanding of good parenting
• Immediate Outcome 1: Develop and recruit
program participants
Web Sites/References
• www.eval.org (AEA website - Guiding Principles +…
• www.wisc.ex (Ellen Taylor Powell)
• www.wkkf.org (WK Kellogg Foundation - logic models + eval
manual)
• gi.unc.edu/research/k-12-toolkit (EvAP Evaluation Tool Kit for
International Presenter Programs)
• www.gseis.ucla.edu/~srmevaluationgroup/reports/evap.PDF
(Meta-evalaution of Collaborative Evaluation of First 5 LA
Childcare Initiative)
• O’Sullivan, R. G., & D’Agostino, A. (2002). Promoting evaluation
through collaboration with community-based programs for young
children and their families. Evaluation, 3, 372-387.
• Campos-Rodriguez, L. Collaborative Evalaution. Tamarac, FL:
Lumina Press.
• O’Sullivan, R. G., Jay, M., & Powers Costello, B. (2006).
Transcribing Focus Group Interviews: Is There a Viable
Alternative for Analysis? Paper presented at the annual
meeting of the American Evalaution Association, Portland, OR.
Collaborative Evaluation Techniques:
Designing Collaborative Evaluations
Evaluation Crosswalk:
EVALUATION QUESTIONS
I. 1. To what extent have
parenting classes promoted
better parenting?
II. 2. How have project
strengthened the capacity of
rural communities?
IB. How can the program be
improved?
1 2 3 4 5 6
X X X X
X X
Collaborative Data Collection
•
Explore information gathering
alternatives with program staff
•
Validate instruments by consulting with
program experts, including staff
•
Share summaries of information
gathered with data sources and staff
Convincing Levels of Evidence
1 - NOT Convincing
2 = SOMEWHAT Convincing
___ Newspaper story
___ Comparison with group
___ A friend
___ Pre-test/Post-test
___ Testimonial
___ Celebrity endorsement
___
3 = VERY Convincing
___ Double-blind controlled
___ A friend of a friend
___ Description of events
___ Salesperson
___ Treatment/control group
___ Expert critique
___
Collaborative Data Analysis
•
Engage program staff and/or
participants in data analysis
•
Share data analyses with program
staff
•
Ask program staff for their
interpretation of data findings
Collaborative Reporting:
Evaluation Fairs & Conferences
promote evaluation
• enhance networking
•
suggest alternative program ideas
• allow programs to celebrate
achievements
•
Collaborative Evaluation and
Evaluation Ethics
AEA Guiding Principles:
•
•
•
•
•
Systematic Inquiry
Competence
Integrity/Honesty
Respect for People
Responsibilities for General & Public
Welfare
Collaborative Evaluation Cycle
Clarify Evaluation Request
Conduct Evaluation
Fair
COMMUNICATION
Implement the
Evaluation
Design
Evaluation
Discussion Items
•
•
•
•
•
•
•
Evaluator as technical assistance consultant
Staff changes create the need for training
Reluctance of program to take ownership
Reporting responsibility can become muddled.
Differences between internal and external
evaluators
Objectivity of collaborative evaluators can
become suspect.
Some may expect distanced evaluation
Thank you!
• Please complete evaluation forms
and
• Contact us if we can be of assistance.