Chapter 6:
Program-Oriented
Approaches
Presentation by Jay Kerstetter, Amanda Brown,
Jody Yoos, & Jane Lightner
J.K.
Orienting Questions:
#1 What are the key concepts of the
objectives-oriented evaluation approach?
#2 How has this approach influenced
evaluation?
What is it?
 Objectives-oriented evaluation approach helps
determine whether some or all of the program
objectives are achieved and, if so, how well they
are achieved.
 Evaluators may work with stakeholders to
establish if program objectives are met.
 Information from this approach can assist with
deciding to maintain, terminate, or change
approaches within the program.
J.K.
1. Ralph Tyler is credited with initiating this approach in the 1930’s
2. Tyler began to formulate his views on education and evaluation.
3. His approach included the following steps:
 Establish goals or objectives
 Classify the goals or objectives
 Define objectives in behavioral terms
 Find situations in which achievement of objectives can be shown
 Develop or select measurement techniques
 Collect performance data
 Compare performance data with behaviorally stated objectives
4. This approach was readily adoptable by evaluators and had great
influence on evaluation theorists.
Tylerian Evaluation Approach
J.K.
Provus’s
Discrepancy Evaluation Model:
1.
2.
3.
4.
Developed by Malcolm Provus; viewed evaluation as a
continuous information management process.
Provus stemmed key characteristics of his proposal from Tyler.
Provus viewed evaluation as a process.
This process was called DEM; Discrepancy Evaluation Model,
which are broken into four developmental stages.
◦ Definition
◦ Installation
◦ Process
◦ Product
◦ Cost-benefit analysis (optional)
The DEM was designed to facilitate the development of programs
in large public school systems and later applied to state
evaluations by federal bureau.
 The DEM was one of the earliest approaches to evaluation and
elements of it can still be found today in many evaluation.

J.K.
Orienting Question #3:
How is the objective-oriented
evaluation approach used today?
Standards-based testing
Accountability in education
Performance monitoring systems
government programs
Many
1930’s
used in many
refinements to the system since the
J.Y.
Ralph W. Tyler:
Tylerian Evaluation Approach
Influenced the Elementary and Secondary
Education Act (ESEA) of 1965, first act to
require evaluation of educational
programs
 Started the National Assessment of
Educational Progress (NAEP), only way to
examine educational achievement of al 50
states.

J.Y.
Approach evaluated in Pittsburgh public
schools
 Viewed evaluation as a continuous
information- management process
designed to serve as “the watch-dog of
program-management” and the
“handmaiden of administration in the
management of program development
through sound decision making”

Malcolm Provus: Provus’s
Discrepancy Evaluation Model
J.Y.
Orienting Question #4a: How are
Logic Models used in evaluation?


Developed as
an extension of
objectivesoriented
evaluation
Designed to fill
in those steps
between the
program and its
objectives
Program planners/evaluator
Identify program
oInputs
oActivities
oOutputs (immediate
program impacts)
oOutcomes (long-term
objectives/goals)
J.Y.
Today Logic Models are used in program
planning/evaluation
 Help program staff articulate /discuss how
program might achieve goals
 What elements are important to evaluate at
any given time
 Build internal evaluation capacity or think in
an evaluative way
Example Companies:
 United way of America
 WK Kellogg Foundation
 Annie E Casey Foundation
J.Y.
Theory-Based evaluation
 Is used by evaluators to gain a better
understanding of the program
 Better define the evaluation questions the study
should address
 To aid their choices of what concepts to measure
and when to measure them
 To improve their interpretation of results and
their feedback to stakeholders to enhance use.
Orienting Question #4b:
How are Program Theories used in
evaluation?
J.Y.
Steps in Theory Based Evaluation
1) Engage Relevant stakeholder
2) Develop a first draft of program
theory
evaluator or evaluation team
3) Present draft to stakeholders for further
discussion, reaction, and input
4) Conduct a plausibility check
5) Communicate findings to key
stakeholders
6) Probe arrows for model specificity
7) Finalize program impact theory
J.Y.
Theory Driven Evaluation
Work with stakeholders to identify key questions to be answered
in the evaluation and the appropriate designs and methods for
answering those questions
Emphasis is on testing the program model
Provide guidance as to what to measure and when to measure it
The selection of the evaluation questions to be addressed depend
on the stage of the program and what stakeholders hope to learn
Provides evaluator with critical information that can be used
throughout the evaluation
J.Y.
Orienting Question #5: How do
theory-based evaluation and
objectives approaches differ?
Objectives Approach:
The objectives are
identified by group
looking for the
evaluation.
2.
Purpose of activity
specified and then those
purposes are being
acheived.
3.
Individual creditedRalph W. Tyler

1.
Theory-Based
The evaluator discusses
goals, basics, and
objectives of program
w/the stakeholders.
2.
The evaluator decides
how the program should
work and then sees if it
works that way.
3.
Huey Chen and Leonard
Bickman helped to
develop approaches to
theory based.
4.
Science based and
quantitative.

1.
A.B.
Question #6: What are the strengths and
limitations of program oriented
evaluation approaches?
Strengths:
 Objective oriented – the simplicity of the
concept and program makes it easy to
understand, follow and possibly
implement.
◦ Face validity – the evaluator is being held
accountable for what is being asked to be
evaluated. They want to see if what is currently
happening or working based on their own goals
and objectives for the program.
A.B.

Theory oriented – the chance for the
evaluator to partake in dialogue with the
stakeholders is a strength of the program
because it helps them to expand their
knowledge on the topic.
◦ This then leads to the evaluator to have a clear
understanding of the program so they know
how to properly evaluate the program.
Strengths (con’t)
A.B.
Weaknesses:

Objective oriented – the evaluator can have
a single minded focus on the objectives
which will cause them to overlook the
complications, elements and factors
contributing to that programs success or
failure.
a. This approach does not ask for the evaluator to
gain an understanding of the context in which
the program operates this could then be what is
affecting the program’s success or failure.
b. Evaluator could ignore the actual value of the
objective. Since the evaluator is told what the
objectives are they are not asked to evaluate
whether that objective even fits the program.
A.B.

Theory oriented – this like the objective
oriented can cause the evaluator to ignore
certain aspects of the program that are
important because the evaluator is so
focused on the theory of how it runs and not
how it is actually running.
a. Evaluators may ignore the needs or values of
stakeholders involved with the program.
b. This approach may also oversimplify the
complexity of the program making it feel easier
to evaluate than it really is because not all of the
surrounding factors are accounted for in the
process.
Weaknesses (con’t)
A.B.
Orienting Question # 7: What is
goal-free evaluation?



Rationale: “Goals should not be taken as
givens.”
Developed by Scriven (1972)---believes the
most important function of a goal-free
evaluation is to reduce the bias that occurs
from knowing program goals
Thus…increase objectivity in judging the
program as a whole.
J.L.
Orienting Question #8: What does
it teach us about conducting an
evaluation?


Goals can act as “blinders” causing us to
possibly miss the most important outcomes
not related to the goals.
Goal-free evaluation was proposed to
primarily indentify the unanticipated side
effects that an objectives-oriented
evaluation might miss
J.L.
Major Characteristics of a GoalFree Evaluation
1. Evaluator
purposely avoids
becoming aware of
program goals.
2. Predetermined
goals are not
permitted o narrow
the focus of the
evaluation study.
4. The goal-free
evaluator has
minimal contact with
the program
manager and staff.
3. Goal-free
evaluation focuses
on actual outcomes
rather than intended
program outcomes.
5. Goal-free
evaluation increases
the likelihood that
unanticipated side
effects will be noted.
J.L.
Internal Goal Evaluator vs.
Goal-directed
 How well is program
meeting goals
 Provide information
to administrator

External Goal-Free
Evaluator


What does the
program do?
Looking at ALL the
programs
outcomes,
intended or not
Goal Directed + Goal-Free=
can work together
J.L.
Information taken from
Program Evaluation:
Alternate Approaches and
Practical Guidelines
p. 153-171