The Formative & Summative
Evaluation Process
TS & EOC
SAEOPP Priority 1 Training
PY 2009
Presenter
Elizabeth C. Retamozo, Executive Director
NOSOTROS Education Center
www.nosotrosedu.org
Phone: 469-450-7137
Email:
eretamozo@nosotrosedu.org
eretamozo@tx.rr.com
2
Presentation Materials
• PowerPoint Slide Handouts
• Additional Documents & Materials located at:
• www.nosotrosedu.org
– Go to Resources Page
– Register Your Email Address
– Click the Presentations & Links Tab
– Click SAEOPP Priority 1 Training
3
Session Summary
This session will focus on providing Talent Search
staff with an overview of:
• Program Evaluation
• Types of Program Evaluations
• Methods & Types of Data Collection
• Quantitative & Qualitative Data
• Formative & Summative Methods
4
Evaluation
• Evaluation - Systematic method for collecting,
analyzing, and using information to answer basic
questions about your program.
• Helps to identify effective & ineffective services,
practices, & approaches.
• Informs & assists program staff to make changes
to improve program effectiveness.
5
Evaluation Plan
• Evaluation Plan - A written & comprehensive
plan to determine the effectiveness of the
program in meeting stated objectives & goals.
• Includes what you plan to do, how you plan to do
it, who will do it, and when it will be done.
• Provides a mechanism for evaluating &
implementing programmatic changes to improve
program effectiveness
• Relates your project to GPRA & the TRIO
Programs Performance Indicators
6
The GPRA Connection
1) Government Performance and Results Act of 1993
(GPRA)
2) DOE Response to GPRA Requirements: Goals 1-6
3) TRIO Programs Performance Indicator: Goal 5
4) TRIO Programs’ Specific Program Measures:
 Upward Bound, Talent Search, Educational Opportunity Centers
 Student Support Services & Ronald E. McNair
5) Prior Experience Criteria & Mandatory Objectives – APR
6) Project Evaluation Plan
7) Activities & Services Specific to Each Objective
7
Types of Evaluation Plans
 Goals-Based Evaluation
 Outcome-Based Evaluation
Process-Based Evaluation
8
Evaluation Plan “Buzz” Words

Qualitative Values

Quantitative Values

Formative Methods

Summative Methods
9
Qualitative Values
• Measurement
perception
of
“feelings”,
impressions,
&
 Ways to describe effectiveness that are
measurable, but may not aggregate in traditional,
quantifiable methods
 Information that is difficult to measure, count, or
express in numerical terms
10
Collection of Qualitative Data
• Interviews & Focus Groups
Used to fully understand someone's impressions or
experiences, or learn more about their answers to
questionnaires
• Observations
Used to gather accurate information about how a program
actually operates, particularly about processes
• Questionnaires, Surveys, Checklists
Used to quickly and/or easily get info in a non-threatening
way
11
Quantitative Values
• Information that can be expressed in numerical
terms, counted, or compared on a scale
 Promotion to Next Grade Level Rates
 Graduation From High School Rates
 # of College Admissions Forms Completed
 # of Financial Aid Applications Completed
 College Enrollment Rates
12
Collection of Quantitative Data
• Report Cards
• Transcripts
• Test Scores (NCLB, PSAT, SAT, ACT)
• National Student Clearinghouse Reports
13
Formative Methods
• Continuous (daily, weekly, monthly, quarterly,
semester) assessment of services/activities &
progress towards achieving objectives
• Allows for changes in activities during the year so
final objectives/outcomes will be met (changes in
delivery of services)
14
Formative Methods
• Weekly/monthly student service reports
# of continuing student with 0 services
# students with less than 2 services
# of students with equal to or more than # services
# students that have received a specific services
• Student evaluation of workshops & field trips
• Grading period report cards (6/9 weeks)
15
Progress Towards Achieving Goals & Objectives
During the Year -
Formative Methods:
• Project staff document all services provided to
students as they occur on Individual or Group
Service Logs
• Project staff enter all data on service logs into the
project database on a weekly basis
• Project database generates monthly/quarterly
contact reports documenting which students
have been served, the types of services
provided, & the frequency of services provided.
16
Progress Towards Achieving Goals & Objectives
During the Year -
Formative Methods:
• Project Director tracks progress towards the
accomplishment of all objectives via reports
(quantitative) generated by the project database
on a weekly/ monthly basis
• Participants rate specific group activities and
workshops to rate their level of satisfaction with
the quality of services on evaluation forms
(qualitative).
17
Summative Methods
• End of year assessment that measures the
results or outcomes of a project.
 Concerned with a program's overall effectiveness
 Quantitative data is analyzed for a given period
 Qualitative evaluation supplements the quantitative data
18
Achieving Goals & Objectives at End of the Year -
Summative Methods:
• Project database combines all monthly reports to
produce a report documenting & confirming the
achievement of project objectives at the end of
the year (quantitative).
• Students
complete
end-of-year
surveys
(qualitative) to measure level of satisfaction with
the overall quality of project services.
19
Achieving Objectives at the End of the Project Period -
Summative Methods:
• Target school/agency/project personnel complete
end of year evaluation forms (qualitative) to
measure level of satisfaction with the overall
quality of project services.
• Project Director reviews evaluation results to
implement any corrective measures to eliminate
potential problems by appropriately modifying &
improving activities & services for the next
program.
20
TRIO Evaluation Plans
• Goals Based Evaluation (Internal & Formative)
Evaluates the extent to which projects are meeting
predetermined goals or objectives. INTERNAL GOALS
• Outcomes Based Evaluation (Internal & Summative)
Looks at impacts/benefits/changes to your students as a
result of your project’s efforts during and/or after their
participation in your project. PE OBJECTIVES
• Process Based Evaluation (External & Summative)
Geared to fully understanding how a program works -- how
does it produce that results that it does.
21
Goals Based Evaluation
1) What project goals were established?
2) What is the status of the project’s progress towards
achieving the goals?
3) Will the goals be achieved according to the timelines
specified in the program implementation or operations
plan?
4) Do personnel have adequate
facilities, training, etc)?
resources
(supplies,
22
Goals Based Evaluation
5) How should priorities be changed to more focus on
achieving the goals?
6) How should timelines be changed?
7) How should goals be established in the future?
23
Sample TS Internal Goals
1) Identification, Recruitment, & Selection of
Participants:
To Identify, recruit, & select new students to have
a total of 600 participants with 67% classified as
both low-income & potential first generation
college students within 90 days of the beginning
of the program year.
24
Sample TS Internal Goals
2) Development of Needs
Individual Service Plans:
Assessment
&
100% of all participants will have their needs
assessed and an Annual Individual Service Plan
developed within 30 days of their initial selection
or renewal into the project each program year.
25
Sample TS Internal Goals
3) Delivery of Promotion Services To 9th - 11th
Participants:
70% of all 9th - 11th grade participants will
receive a minimum of three (3) college, career,
and financial aid exploration services each
semester for a total of six (6) services by the end
of the program year.
26
Sample TS Internal Goals
4) Delivery of Individual College Admissions &
Financial Aid Application Assistance to all
College Ready Participants:
80% of all college ready participants will receive
a minimum of five (5) individual advising sessions
each semester for a total of ten (10) sessions by
the end of the program year.
27
Sample TS Internal Goals
5) Increased Awareness & Knowledge of
College, Financial Aid, & Career Information:
80% of participants will demonstrate an increase
in knowledge of college, financial aid & career
information as demonstrated by a comparison
pre/post tests by the end of the program year.
28
Sample TS Internal Goals
6) Perception & Level of Satisfaction with
Project Services:
85% of all participants and target school
personnel surveyed will rate the project as
"Excellent" on the Project Evaluation Form by the
end of each project year.
29
Outcome Based Evaluation
TS Goals Based Evaluation Plan
30
Sample EOC Internal Goals
1) Identification, Recruitment, & Selection of
Clients:
To Identify, recruit, & select new students to have
a total of 1000 clients with 67% classified as both
low-income & potential first-generation college
students within 180 days of the beginning of the
program year.
31
Sample EOC Internal Goals
2) Development of Needs
Individual Service Plans:
Assessment
&
100% of all clients will have their needs assessed
and an Individual Service Plan developed within
30 days of their initial selection into the project
each program year.
32
Sample EOC Internal Goals
3) Delivery
of
GED/Continuing
Education
Services to Non-College Ready Clients:
100% of all Non-College Clients will receive a
minimum of three (3) GED/Continuing Education
services
(individual
advising,
referrals,
information, etc.) by the end of the program year.
33
Sample EOC Internal Goals
4) Delivery of Individual College Admissions &
Financial Aid Application Assistance to all
College Ready Clients:
100% of all college ready clients will receive a
minimum of three (3) individual advising sessions
by the end of the program year.
34
Sample EOC Internal Goals
5) Perception & Level of Satisfaction with
Project Services:
85% of all Clients and target agency personnel
surveyed will rate the project as "Excellent" on
the Project Evaluation Form by the end of each
project year.
35
Outcome Based Evaluation
EOC Goals Based Evaluation Plan
36
Outcome Based Evaluation
1) Identify Major Outcomes (Mandatory Objectives)
2) Specify Observable Measures & Indicators
3) Identify what Info is Needed to Show these
Indicators
37
Outcome Based Evaluation
4) Decide How this Info Can be Realistically &
Efficiently Gathered
5) Assign Staff Responsibility & State Timelines
6) Analyze & Report Findings
38
Outcome Based Evaluation
TS Outcome Based Evaluation Plan
EOC Outcome Based Evaluation Plan
39
External Evaluation
• Recommended in last grant proposal competitions
• Must be conducted if included in grant proposals
• Allowable cost for the project
• May be paid by the grantee agency/institution
• May be provided free of charge – in-kind/donation
40
External Evaluation
• May be conducted by person working/employed
by agency/institution BUT not for the project
• Must have experience conducting evaluations for
educational opportunity programs & TRIO like
programs
• Must state what type of evaluation will be
conducted by the evaluator.
41
External Evaluation
• Process-Based Evaluation
Process-based evaluations are geared to fully
understanding how a program works -- how
does it produce that results that it does.
• Useful if programs are long-standing and have
changed over the years.
• Useful for accurately portraying to outside
parties how a program truly operates.
42
Process-Based Evaluation
• Examines the extent to which a program is
operating as intended by assessing ongoing
program operations & whether the targeted
population is being served.
• Helps
program
staff
identify
needed
interventions
and/or
change
program
components to improve service delivery.
43
Process Based Evaluation
1) How does the project identify, recruit, & select
students?
2) How does the project decide what specific
services & activities to provide to its students?
3) How does the project monitor students’ academic
progress?
44
Process Based Evaluation
4) How does the project deliver services to its students?
5) How does the project document services provided to
student?
6) What is the project’s goals based evaluation plan?
7) What is the project’s outcome based evaluation plan?
8) How does the project plan for the next program year?
45
Process Based Evaluation
Process Based Evaluation Plan
46
Bibliography & Resources
Basic Guide to Program Evaluation
Written by Carter McNamara, MBA, PhD, Authenticity
Consulting, LLC. Copyright 1997-2007.
Adapted from the Field Guide to Nonprofit Program Design,
Marketing and Evaluation.
Online Guide:
A Basic Guide to Program Evaluation.pdf
47
Bibliography & Resources
Basic Guide to Goals-Based Evaluation for Nonprofit
Organizations with Very Limited Resources
Written by Carter McNamara, MBA, PhD, Authenticity
Consulting, LLC. Copyright 1997-2007.
Online Guide: Basic Guide to Outcomes-Based Evaluation
for Nonprofit Organizations with Very Limited Resources.mht
48
Bibliography & Resources
US Department of Health & Human Services Administration for
Children & Families - Office of Planning, Research & Development
The Program Manager's Guide to Evaluation
http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval/reports/p
mguide/pmguide_toc.html
This guide explains program evaluation -what it is, how to
understand it, and how to do it. It answers your questions
about evaluation and explains how to use evaluation to
improve programs and benefit students, families, and staff.
49
Questions & Answers
50