Upward Bound
Internal & External Program
Evaluation
Southeastern Association of Educational Opportunity Program Personnel
38th Annual Conference
January 30 – February 3, 2010
Presenter
Elizabeth C. Retamozo, Executive Director
NOSOTROS Education Center
www.nosotrosedu.org
Phone: 469-450-7137
Email: eretamozo@nosotrosedu.org
2
Presentation Materials
• PowerPoint Slide Presentation & Handout
• Located at: www.nosotrosedu.org
– Go to Resources Page
– Register Your Email Address
– Click the Presentations & Links Tab
– Click SAEOPP 2010 Folder
– UB Internal & External Program Evaluation
3
Types of TRIO Evaluation Plans
 Outcome-Based Evaluation
 Goals-Based Evaluation
 Process-Based Evaluation
4
Common Evaluation Plan Terminology
Qualitative Values
 Expressed by impressions, feelings & perceptions
Ex. Surveys, Interviews, Focus Groups, etc.
Quantitative Values
 Expressed in numerical terms, counted,
compared on a scale
Ex. Grade Point Averages, Test Scores, etc.
or
5
Common Evaluation Plan Terminology
Formative Methods
 Continuous (daily, weekly, monthly, quarterly,
semester) assessment to track progress
Ex. Monthly Reports
Summative Methods
 End of year assessment that measures the
results or outcomes of a Project
Ex. Annual Performance Report
6
Sample TRIO Evaluation Plans
 Outcomes Based Evaluation
• External & Summative
Looks
at
impacts/benefits/changes
to
your
students as a result of your project’s efforts after
their
participation
in
your
Project
(PE
OBJECTIVES).
7
Sample TRIO Evaluation Plans
 Goals Based Evaluation
• Internal & Formative
Evaluates the extent to which projects are
meeting
predetermined
internal
goals
or
objectives.
8
Sample TRIO Evaluation Plans
 Process Based Evaluation
• External & Summative)
Geared to fully understanding how a Project works
How does it produce that results that it gets?
9
Outcome Based Evaluation
 Outcomes Based Evaluation (External &
Summative)
 Measures performance on the four (4) established
UB Program objectives by the end of each year
(summative).
 DE Mandated Objectives = Prior Experience
10
Outcome Based Evaluation Plan
1) Identify Major Outcomes (Mandatory Objectives)
2) What Types Of Data Will Be Collected?
3) When Will Various Types Of Data & Reports Be
Collected?
4) What Instruments Will Be Developed?
5) How Will The Data Be Analyzed?
6) How Will The Data Collected Determine
Accomplishment?
11
Outcome Based Evaluation
UB Outcome Based Evaluation Plan
12
Goals Based Evaluation
 Goals Based Evaluation (Internal & Formative)
 Evaluates the extent to which Project is meeting
predetermined internal goals
 Project Developed INTERNAL GOALS based on
DE Objectives
 Includes quantifiable & qualitative measures to track
progress during the Project year (formative)
 Monitored and reviewed weekly/monthly
13
Goals Based Evaluation
1) What Project goals were established?
2) What is the status of the project’s progress
towards achieving the goals?
3) Will the goals be achieved according to the
timelines specified in the program implementation
or operations plan?
4) Do personnel have adequate resources (supplies,
facilities, training, etc)?
14
Goals Based Evaluation
5) How should priorities be changed to focus more
on achieving the goals?
6) How should timelines be changed?
7) How should goals be established in the future?
15
Goals Based Evaluation Plan
1) Develop Internal Goals based on External
Objectives
2) Activities & Services Needed to Achieve Each
Goal
3) Project Staff Responsible to Carry Out Activities
& Services
4) Data that Needs to be Collected
5) Instruments (Forms) that Need to be Developed
and Collected
16
Goals Based Evaluation Plan
6) When is Data Collected?
7) How is Data Collected Analyzed?
8) Was the Goal Met? Yes/No
9) Needed or Recommended Revisions
10) Unanticipated Outcomes
17
Outcome Based Evaluation
Sample UB Internal Goals
UB Goals Based Evaluation Plan
18
Process Based Evaluation
 Process Based Evaluation (External &
Summative)
 Geared to fully understanding how a project works
 How does it produce the results that it gets?
 Usually External Conducted by Program Evaluator
19
External Evaluation
• Recommended in last grant proposal competitions
• Must be conducted if included in grant proposals
• Allowable cost for the project
• May be paid by the grantee agency/institution
• May be provided free of charge – in-kind/donation
20
External Evaluation
• May be conducted by person working/employed
by agency/institution BUT not for the Project
• Must have experience conducting evaluations for
educational opportunity programs & TRIO like
programs
• Must state what type of evaluation will be
conducted by the evaluator.
21
Process Based Evaluation Plan
1) How does the Project identify, recruit, & select
participants?
2) How does the Project decide what specific
services & activities to provide to its participants?
3) How does the Project monitor participants’
academic progress?
4) How does the Project deliver services to its
participants?
22
Process Based Evaluation Plan
5) How does the Project
provided to participants?
document
services
6) What is the Project’s goals based evaluation
plan?
7) What is the Project’s outcome based evaluation
plan?
8) How does the Project plan for the next program
year?
23
Process Based Evaluation
UB Process Based Evaluation Checklist
External Process Based Evaluation Report
24
Bibliography & Resources
Basic Guide to Program Evaluation
Written by Carter McNamara, MBA, PhD, Authenticity
Consulting, LLC. Copyright 1997-2007.
Adapted from the Field Guide to Nonprofit Program Design,
Marketing and Evaluation.
Online Guide:
A Basic Guide to Program Evaluation.pdf
25
Bibliography & Resources
Basic Guide to Goals-Based Evaluation for Nonprofit
Organizations with Very Limited Resources
Written by Carter McNamara, MBA, PhD, Authenticity
Consulting, LLC. Copyright 1997-2007.
Online Guide: Basic Guide to Outcomes-Based Evaluation
for Nonprofit Organizations with Very Limited Resources.mht
26
Bibliography & Resources
US Department of Health & Human Services Administration for
Children & Families - Office of Planning, Research & Development
The Program Manager's Guide to Evaluation
ACF OPRE The Program Manager’s Guide to Evaluation website.mht
This guide explains program evaluation -what it is, how to
understand it, and how to do it. It answers your questions
about evaluation and explains how to use evaluation to
improve programs and benefit students, families, and staff.
27
Questions & Answers
28