Basic
Program Evaluation
NTSC Training Materials
Purpose/Objectives
• Increase your knowledge of
processes involved in
program evaluation
• Provide information and
resources to help you design
and conduct your own
program evaluation
Program Evaluation
Training
This training presentation is
in 16 modules,
encompassing 9 steps in the
evaluation process
Program Evaluation
Training – Modules
• Module 1 – Introduction
• Module 2 – Overview
• Module 3 – Defining the Purpose
• Module 4 – Specifying the Questions
• Module 5 – Identifying Evidence
Needed
• Module 6 – Specifying the Design
Program Evaluation
Training – Modules
• Module 7 – Data Collection Plan
• Module 8 – How to Collect Data
• Module 9 – Using Commercial
Instruments
• Module 10 – Using Self-Constructed
Instruments
• Module 11 – Collecting Data
Program Evaluation
Training – Modules
• Module 12 – Analyzing Data
• Module 13 – Drawing Conclusions
and Documenting Findings
• Module 14 – Disseminating Information
• Module 15 – Feedback for Program
Improvement
• Module 16 – Conclusion
Program Evaluation
Training – Modules
• Module 1 – Introduction
• Module 2 – Overview
• Module 3 – Defining the Purpose
• Module 4 – Specifying the Questions
• Module 5 – Identifying Evidence
Needed
• Module 6 – Specifying the Design
Module 1 –
Introduction
Module 1 – Introduction
• Why evaluate?
• What is evaluation?
• What does evaluation do?
• Kinds of evaluation
Why Evaluate?
• Determine program outcomes
• Identify program strengths
• Identify and improve weaknesses
• Justify use of resources
• Increased emphasis on
accountability
• Professional responsibility to show
effectiveness of program
What is Program
Evaluation?
• Purposeful, systematic, and careful
collection and analysis of information
used for the purpose of documenting the
effectiveness and impact of programs,
establishing accountability, and
identifying areas needing change and
improvement
What Evaluation
Does
• Looks at the results of your
investment of time, expertise,
and energy, and compares those
results with what you said you
wanted to achieve
Kinds of
Evaluation
• Outcome
• Implementation
• Formative
• Summative
Outcome Evaluation
What: Identifies the results or effects of
a program
When: You want to measure students’ or
clients’ knowledge, attitudes, and
behaviors as a result of a program
Examples: Did program increase
achievement, reduce truancy, create
better decision-making?
Implementation
Evaluation
What: Documents what the program is
and to what extent it has been
implemented
When: A new program is being
introduced; identifies and defines the
program; identifies what you are actually
evaluating
Examples: Who receives program,
where is program operating; is it being
implemented the same way at each site?
Timing of
Evaluation
• Formative
– as the program is happening to make
changes as program is being implemented
• Summative
– at the end of a program to document
results
Module 2 – Overview
• Module 1 – Introduction
• Module 2 – Overview
• Module 3 – Defining the Purpose
• Module 4 – Specifying the Questions
• Module 5 – Identifying Evidence
Needed
• Module 6 – Specifying the Design
Overview – The 9step Process
• Planning
• Development
• Implementation
• Feedback
Overview – The 9step Process
Overview – The 9step Process
Overview – The 9step Process
Module 3 – Defining
the Purpose
• Module 1 – Introduction
• Module 2 – Overview
• Module 3 – Defining the Purpose
• Module 4 – Specifying the Questions
• Module 5 – Identifying Evidence
Needed
• Module 6 – Specifying the Design
9-step Evaluation
Process
Step 1: Define
Purpose and Scope
Step 1: Scope/Purpose
of Evaluation
• Why are you doing the evaluation?
– mandatory? program outcomes? program
improvement?
• What is the scope? How large will the effort
be?
– large/small; broad/narrow
• How complex is the proposed evaluation?
– many variables, many questions?
• What can you realistically accomplish?
Resource
Considerations
• Resources
–$$
–Staff
•who can assist?
•need to bring in expertise?
•do it yourself?
•advisory team?
–Time
• Set priorities
• How you will use the information
Module 4 – Specifying
the Questions
• Module 1 – Introduction
• Module 2 – Overview
• Module 3 – Defining the Purpose
• Module 4 – Specifying the Questions
• Module 5 – Identifying Evidence Needed
• Module 6 – Specifying the Design
9-step Evaluation
Process
Step 2: Specify
Evaluation
Questions
Evaluation
Questions
What is it that you want to know about
your program?
operationalize it (make it measurable)
Do not move forward
if you cannot answer
this question.
Sources of
Questions
• Strategic plans
• Mission statements
• Policies
• Needs assessment
• Goals and objectives
• National standards and guidelines
Broad Questions
• Broad Scope
– Do our students contribute positively to society after
graduation?
– Do students in our new mentoring program have a
more positive self-concept and better decision-making
skills than students without access to the mentoring
program?
– To what extent does the state’s career development
program contribute to student readiness for further
education and training and success in the workforce?
Narrow Questions
• Narrow Scope
– Can our 6th grade students identify appropriate
and inappropriate social behaviors?
– How many of our 10th grade students have
identified their work-related interests using an
interest inventory?
– Have 100% of our 10th grade students
identified at least 3 occupations to explore
further based on their interests, abilities, and
knowledge of education and training
requirements?
Exercise 1 –
Scope (p. 2 of Workbook)
• From the list of questions, identify
those that might be considered broad
and those that might be considered
narrow
• How large will the resources need to
be to answer the question
Exercise 2 – Scope
Write (p. 3 of Workbook)
• List one broad evaluation question
and one narrow evaluation question
Module 5 –
Identifying Evidence
Needed
• Module 1 – Introduction
• Module 2 – Overview
• Module 3 – Defining the Purpose
• Module 4 – Specifying the Questions
• Module 5 – Identifying Evidence
Needed
• Module 6 – Specifying the Design
Identifying Evidence
Needed to Answer Your
Questions
• What evidence do you have to
answer your question?
Identifying Evidence
Needed to Answer Your
Questions
• Need to think about what
information you need in order to
answer your evaluation questions
Example Evidence
– Broad Scope
Do our students contribute positively
to society after graduation?
• Percent of our students that are employed, in
education or training programs, in the military,
are supporting a family by working at home,
and/or are volunteering for charitable causes 3
years after high school graduation
• Percent of our students that vote in local and
national elections 5 years after graduation
Example Evidence
– Narrow Scope
Have 100% of our 10th grade students
identified at least 3 occupations to
further explore that are based on their
interests, abilities, and knowledge of the
education and training requirements?
• Number of 11th and 12th grade students
participating in the career class that
demonstrated increased career maturity from a
pre- and post-test
Exercise 3 – Evidence
(p. 4 of Workbook)
• List evidence you need to have to
answer the question
Module 6 – Specifying
the Design
• Module 1 – Introduction
• Module 2 – Overview
• Module 3 – Defining the Purpose
• Module 4 – Specifying the Questions
• Module 5 – Identifying Evidence
Needed
• Module 6 – Specifying the Design
9-step Evaluation
Process
Step 3: Specify
Evaluation Design
Types of Designs
Relates to when data should be collected
• Status (here and now; snapshot)
• Comparison (group A vs. group B; program
A vs. program B)
• Change (what happened as a result of a
program; what differences are there between
time A and time B)
• Longitudinal (what happens over extended
time)
Exercise 4 –
Design (p. 5 of Workbook)
• What type of design fits each
evaluation question?
– Status
– Comparison
– Change
– Longitudinal
Module 7 – Data
Collection Plan
• Module 7 – Data Collection Plan
• Module 8 – How to Collect Data
• Module 9 – Using Commercial
Instruments
• Module 10 – Using Self-Constructed
Instruments
• Module 11 – Collecting Data
9-step Evaluation
Process
Step 4: Create a
Data Collection
Action Plan
Organize Your Evaluation
With a Data Collection
Action Plan
Evaluation
Question 1
From Whom/
Data Sources
What is
Collected
When
Collected
and By
Whom
How Collected/What
Technique
How Data are to be
Analyzed
Components of a Data
Collection Action Plan
• What Will be Collected?
–based on evidence required
• How Collected? Instrumentation
–surveys? published instrument? focus
group? observations?
Components of a Data
Collection Action Plan
• From Whom Collected?
–who or what provides evidence
• When Collected and by Whom?
–specific dates, times, persons
• How Data are to be Analyzed?
Data Sources —
Who and What
• Students
• Parents
• Teachers
• Counselors
• Employers
• Friends
• Documents and other records
Exercise 5 – Data
Sources (p. 6 of Workbook)
• Who/what are the data sources for
the following questions?
Module 8 – How to
Collect Data
• Module 7 – Data Collection Plan
• Module 8 – How to Collect Data
• Module 9 – Using Commercial
Instruments
• Module 10 – Using Self-Constructed
Instruments
• Module 11 – Collecting Data
Data Collection
Options
• Commercial instrument
• Survey/questionnaire
• Focus group/interviews
• Observations
• Archived information
Module 9 – Using
Commercial
Instruments
• Module 7 – Data Collection Plan
• Module 8 – How to Collect Data
• Module 9 – Using Commercial
Instruments
• Module 10 – Using Self-Constructed
Instruments
• Module 11 – Collecting Data
Commercial
Instruments
• Sometimes best to use published or
research instruments
– particularly for tough constructs
– since its not made specifically for you,
may not answer your question entirely
Sources of Information
on Instruments
• Counselors Guide to Career Assessment
Instruments
• Relevance, the Missing Link ─ A Guide for
Promoting Student Success Through Career
Development Education, Training, and
Counseling
• The Buros Institute
• ETS Test Collection
• The Association for Assessment in Counseling and
Education
Exercise 6: DecisionMaking Checklist (p. 7 of
Workbook)
• This checklist will help you conduct
a review of data collection
instruments that you are considering
using in your evaluation
Module 10 – Using SelfConstructed
Instruments
• Module 7 – Data Collection Plan
• Module 8 – How to Collect Data
• Module 9 – Using Commercial
Instruments
• Module 10 – Using Self-Constructed
Instruments
• Module 11 – Collecting Data
Self-Constructed
Instruments:
Questionnaires
• Focus on evidence you need
• Use simple language
• Ask only what you need; keep it short
• Don’t use jargon
• Each question should focus on one idea
• Make sure terms are clear
• Make it easy for person to answer the questions
(check rather than write, where possible)
• Use extended response when you want details
Types of Scales (1)
Specific (yes, no; number; gender)
Extended (1-3, 1-5, 1-7)
Types of Scales (2)
Anchored Scales
Scales for Younger
Students
Self-Constructed
Instruments: Focus
Groups/Interviews
• Good to use when you want extended and
detailed responses
• Craft an agenda and stick to it
• Keep groups small (6-10); time short (11.5 hours)
• Specify objectives of session
• Questions need to be clear; one question
at a time
• Encourage everyone to participate
• Use opportunity to probe deeper on a
topic
Observations and
Observational Checklist
• “You can observe a lot just by
watching.”
-- Yogi Berra
• Go to pages 8 and 9 of your
Workbook and review an example of
an observational checklist
Archives and
Documents
• Examine What’s Already Available
• Examples
– Attendance records
– Truancy reports
– Grades
– Bullying incidents
– Report cards
– Portfolios
– Discipline referrals
– Public service hours
– Police reports
Exercise 7 - Data
Collection Action Plan
• Review
examples of a completed
Data Collection Action Plan on
Pages 10-12 of the Workbook
Module 11 –
Collecting Data
• Module 7 – Data Collection Plan
• Module 8 – How to Collect Data
• Module 9 – Using Commercial
Instruments
• Module 10 – Using Self-Constructed
Instruments
• Module 11 – Collecting Data
9-step Evaluation
Process
Step 5: Collect
Data
How Much Data Should
You Collect?
• How much data do you need?
– 100% of target audience is ideal; may be too
expensive and time consuming
– If not 100%, sample is OK if group is
representative of group as a whole (population)
Types of Samples
Data Collection
Considerations
• When should you collect the information?
• Who should collect it?
Module 12 –
Analyzing Data
• Module 12 – Analyzing Data
• Module 13 – Drawing Conclusions and
Documenting Findings
• Module 14 – Disseminating Information
• Module 15 – Feedback for Program
Improvement
• Module 16 – Conclusion
9-step Evaluation
Process
Step 6: Analyze
Data
What is Data
Analysis?
• Data collected during program
evaluation are compiled and analyzed
(counting; number crunching)
• Inferences are drawn as to why some
results occurred and others did not
• Can be very complex depending on
your evaluation questions
• We will focus on simple things that can
be done without expert consultants
Types of Data Analysis
Simple Frequency Counts
Types of Data Analysis
Sort by Relevant
Categories
Types of Data Analysis
Calculate Percentages –
Exercise 8 (p.13 of Workbook)
Types of Data Analysis
Showing Change or
Differences
Types of Data Analysis
Reaching an Objective or
Goal
Types of Data Analysis
Observing Trends
Types of Data Analysis
Graph Results
Types of Data Analysis
Calculate Averages – Exercise 9
(p. 14 of Workbook)
Types of Data Analysis
Calculate Weighted
Averages
Types of Data Analysis
Calculate Weighted
Averages
Types of Data Analysis
Rank Order Weighted
Averages
Types of Data Analysis
Graph Weighted Averages
Using Focus
Group/Interview
Information
• Qualitative findings from focus
groups, extended response items,
etc., should be analyzed in a different
way
– Code words/frequency
– Identify themes
– Pull quotes
– Summarize and draw conclusions
Module 13 – Drawing
Conclusions and
Documenting Findings
• Module 12 – Analyzing Data
• Module 13 – Drawing Conclusions and
Documenting Findings
• Module 14 – Disseminating Information
• Module 15 – Feedback for Program
Improvement
• Module 16 – Conclusion
9-step Evaluation
Process
Step 7: Drawing
Conclusions and
Documenting
Findings
Drawing
Conclusions
• Examine results carefully and
objectively
• Draw conclusions based on your
data
• What do the results signify about
your program?
Exercise 10 – Interpreting
Results (p. 15-16 of
Workbook)
• Complete the Interpreting Results
Exercise on pages 15-16 of the Workbook.
Unintended
Consequences
• Watch for positive and negative
outcomes that you did not plan on
- For example, if your career development
program focuses on increasing students’
awareness of how to identify their interests and
skills, it may have the unintended consequence of
leaving little time for students to explore
occupations and jobs in their area.
- Or, if your program has overemphasized
the importance of getting a college education,
students may not be considering the positive
benefits of other kinds of postsecondary training.
What to Include in Your
Documentation
• Program description
• Evaluation questions
• Methodology (how and from whom and
when)
• Response rate
• Methods of analysis
• Conclusions listed by evaluation question
• General conclusions and findings
• Action items
• Recommendations for program
improvement and change
Document the
Successes and
Shortfalls
• Highlight and brag about positive
outcomes
• Document shortfalls
- Provides opportunities to
- improve program
- make recommendations to benefit the
program
Module 14 –
Disseminating
Information
• Module 12 – Analyzing Data
• Module 13 – Drawing Conclusions and
Documenting Findings
• Module 14 – Disseminating Information
• Module 15 – Feedback for Program
Improvement
• Module 16 – Conclusion
9-step Evaluation
Process
Step 8: Disseminate
Information
Determining
Dissemination Methods
• Inform all your relevant stakeholders
on results
• Dissemination methods should differ
by your target audience
Potential
Audiences
• Your program staff
• Businesses
Partners that work with your program
Employers
• School Level
School administrators
Counselors
Teachers
Students
Parents
Potential
Audiences
• Media
Local newspaper
TV station
Radio program
Community or school newsletter
• Education Researchers
• Members of Community or faith based organizations
Church members
Religious leaders
Rotary club
Boys or girls club
• Anyone who participated in your evaluation!
Dissemination
Techniques
• Reports
• Journal articles
• Conferences
• Career Newsletter/Tabloids
• Presentations
• Brochures
• TV and newspaper interviews
• Executive summary
• Posting on Web site
Exercise 11 – Disseminating
Information (p. 17 of
Workbook)
• Using the information provided in
exercise 6, describe how you would
disseminate the information to
– Program funders
– Parents
Module 15 – Feedback for
Program Improvement
• Module 12 – Analyzing Data
• Module 13 – Drawing Conclusions and
Documenting Findings
• Module 14 – Disseminating Information
• Module 15 – Feedback for Program
Improvement
• Module 16 – Conclusion
9-step Evaluation
Process
Step 9: Feedback
to Program
Improvement
Opportunities to
Fix Shortfalls
• Evaluation results may show areas where
improvement is necessary
- 25% of 11th graders are unable to
complete a skills based resume
- 85% of our students drop out of college in
the first year
- Most employers do not want your students
to serve an interns in their companies
Feedback to Program
Improvement
• You can use evaluation findings to
make program improvements
– Consider adjustments
– Re-examine/revise program strategies
– Change programs or methodologies
– Increase time with the program
• Use your results as a needs
assessment for future efforts
Module 16 - Conclusion
• Module 12 – Analyzing Data
• Module 13 – Drawing Conclusions and
Documenting Findings
• Module 14 – Disseminating Information
• Module 15 – Feedback for Program
Improvement
• Module 16 – Conclusion
Conclusion
Evaluation helps you:
• determine the effects of the program on
recipients
• know if you have reached your
objectives
• improve your program
Conclusion
• The 9-step process works
• A credible evaluation can be done with
careful planning and some basic math
skills
Exercise 12 – Developing a
Data Collection Action Plan
(page 18 of Workbook)
Using all the information you have gathered
from the workbook exercises and the
powerpoint slides, you can develop your own
Data Collection Action Plan on page 18 of your
Workbook