EVAL 6000: Foundations
of Evaluation
Dr. Chris L. S. Coryn
Nick Saxton
Fall 2014
Agenda
• Quasi-evaluation studies
• Activity (if time allows)
Quasi-evaluation
studies
Quasi-evaluation studies
• Address specific questions (often
employing a wide range of methods)
• Advocate use a particular method
• Whether the questions or methods
are appropriate for assessing merit
and worth is a secondary
consideration
• Both are narrow in scope and often
deliver less than a full assessment of
merit and worth
Approach 7: Objectives-based
studies
• Advance organizers
– Statement of program objectives
• Purposes
– To determine to what extent a program
achieved objectives
• Sources of questions
– Objectives as defined by staff, funder, or
evaluator
• Questions
– To what extent were each of the stated
objectives met?
Objectives-based evaluation results
from a national research center
Objective
Relevance
60%
Need
67%
Quality of Design
62%
Quality of Measurement
50%
Quality of Analysis
75%
Impact on Policy
90%
Impact on Practice
70%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
• Methods
– Any relevant method for determining to
which operationally defined objectives
were met
• Pioneers
– Ralph Tyler
• Use considerations
– Must have clear, supportable objectives
• Strengths
– Ease of application
• Weaknesses
– Narrowness and inability to identify
positive and negative side effects
Approach 7: The success case
method
• Advance organizers
– Comparison between successful and
unsuccessful instances
• Purposes
– To determine how well and in what respects a
program is ‘working’
• Sources of questions
– Generally from program providers
• Questions
– What are the noteworthy successes?
– How were successes produced?
– What factors contributed to success/failure?
-4
-3
-2
-1
0
+1
+2
+3
+4
• Methods
1.
2.
3.
4.
Focus and plan the study
Create an impact model
Survey all participants
Interview a sample of success and nonsuccess
cases
5. Communicate findings, conclusions, and
recommendations
• Pioneers
– Robert Brinkerhoff
• Use considerations
– Intended to assist service providers in increasing
‘successes’ and decreasing ‘nonsuccesses’
• Strengths
– Ease of application
– Use for improvement
• Weaknesses
– Narrowness of scope
Approach 9: Outcome
evaluation as value-added
assessment
• Advance organizers
– System-wide measures of ‘growth’ or ‘gains’
• Purposes
– ‘Value added’ by a program and its constituent
parts
• Sources of questions
– Oversight bodies
• Questions
– What parts of a program contribute most to
‘growth’ or ‘gains’?
• Methods
– Gain score analysis, hierarchical linear
modeling, etc.
• Pioneers
– Raudenbush, Sanders, Horn, Timms, etc.
• Use considerations
– Can be used to make and/or support policy
decisions
• Strengths
– Longitudinal rather than cross-sectional
• Weaknesses
– Potential misuse by policy makers in
placing ‘blame’
Approach 10: Experimental
and quasi-experimental studies
• Advance organizers
– Cause-and-effect hypotheses, competing
treatments, etc.
• Purposes
– To determine causal relationships between
independent and dependent variables
• Sources of questions
– Researchers, developers, policy makers, etc.
• Questions
– To what extent is one treatment superior to
another?
Recruitment
Eligible Units
Excluded
Allocation
Randomization
Intervention
Control
Not Received
Follow-Up
Received
Completed
Not Received
Received
Discontinued
Discontinued
Lost to Follow-Up
Lost to Follow-Up
Completed
• Methods
– Random or other method of assignment to
conditions
• Pioneers
– Campbell, Cook, Shadish
• Use considerations
– Addresses only one particular type of
question
• Strengths
– Strong causal conclusions (if assumptions
are met)
• Weaknesses
– Requires substantial expertise, time,
money, etc.
Approach 11: Cost studies
• Advance organizers
– Costs associated with program inputs,
outputs, and outcomes
• Purposes
– The costs and outcomes of one more more
alternatives
• Sources of questions
– Policy makers, planners, taxpayers, etc.
• Questions
– What are the costs of obtaining desired
outcomes?
• Methods
– Analysis of monetary and nonmonetary
units
• Pioneers
– Levin, McEwin, Yates, etc.
• Use considerations
– Expertise required
• Strengths
– ‘Bottom line’ conclusions of interest to
most decision makers
• Weaknesses
– Often difficult to validly execute
Approach 12: Connoisseurship
and criticism
• Advance organizers
– Specialized expertise, sensitivities, tacit
knowledge, etc.
• Purposes
– To describe, appraise, and illuminate
• Sources of questions
– Determined by the ‘connoisseurs’ or ‘critics’
• Questions
– What are a program’s salient
characteristics, strengths, and
weaknesses?
• Methods
– Perceptual sensitivity, prior experience,
refined insights, etc.
• Pioneers
– Elliot Eisner
• Use considerations
– An audience willing to accept the
approach
• Strengths
– Exploitation of refined expertise
• Weaknesses
– Objectivity and reliability
Approach 13: Theory-based
evaluation
• Advance organizers
– A carefully specified ‘theory’ of how a
program is intended to operate
• Purposes
– To determine the extent to which a
program is ‘theoretically sound’
• Sources of questions
– Determined by the guiding program theory
• Questions
– To what extent does the program theory
‘work’ or not?
Program Process Theory
Inputs
Inputs
Activties
Activties
Outputs
Outputs
Program Impact Theory
Initial
Initial
Outcomes
Outcomes
Intermediate
Intermediate
Outcomes
Outcomes
Long-Term
Long-Term
Outcomes
Outcomes
Resources
Environment
Action Model
Implementing
Implementing
Organizations
Organizations
Associate
Associate
Organizations
Organizations and
and
Community
Community Partners
Partners
Intervention
Intervention and
and
Service
Service Delivery
Delivery
Protocols
Protocols
Ecological
Ecological
Context
Context
Implementors
Implementors
Target
Target Populations
Populations
Change Model
Program
Program
Implementation
Implementation
Intervention
Intervention
Determinants
Determinants
Outcomes
Outcomes
• Methods
– Any method appropriate for testing the
program theory
• Pioneers
– Chen, Donaldson, Weiss, Rogers, Rossi,
etc.
• Use considerations
– Difficulty in applying the approach
• Strengths
– Useful for determining potential
‘measurement’ variables
• Weaknesses
– Few programs are grounded by validated
‘theories’
Approach 14: Meta-analysis
• Advance organizers
– Sufficient studies of the same or similar
programs
• Purposes
– To assemble and (statistically) integrate
findings from multiple studies of the same
or similar programs
• Sources of questions
– Policy makers, ‘research repositories’, etc.
• Questions
– What is the average effect of a particular
type of program?
Study
Sample
Size
Odds Ratio and 95%
Confidence Interval
20% Equivalence
Range
0.80
Random-Effects
Weight
1.20
Fombonne et al. (2006)
180
0.02 (0.01, 0.04)
17.66%
Mrozek-Budzyn et al. (2010)
288
0.05 (0.01, 0.38)
11.82%
Takahashi et al. (2003)
63
0.18 (0.05, 0.62)
15.76%
Aldridge-Sumner et al. (2006)
114
0.51 (0.26, 1.02)
18.01%
Fombonne et al. (2001)
194
0.61 (0.36, 1.02)
18.50%
Uchiyama et al. (2007)
769
0.63 (0.34, 1.16)
18.24%
Random-Effects Model
1,608
0.20 (0.06, 0.63)
Fixed-Effect Model
1,608
0.31 (0.23, 0.43)
1.00
Does Not Favor MMR Vaccine
Favors MMR Vaccine
Odds Ratio and
95% Confidence Interval
• Methods
– Statistical methods for integrating study
results (varies widely)
• Pioneers
– Glass
• Use considerations
– Major source of contemporary ‘best
practices’ across a variety of domains
• Strengths
– Evidence of effectiveness over units,
treatments, observations, and settings
• Weaknesses
– Exclusive emphasis on program outcomes
Activity
Activity
• We will split the class into two sections (1
and 2)
– In each section, appoint one student to chair
your appointed group
– Each member of section 1 should select one of
the approaches discussed today and discuss
why it is useful
– Members of section 2 should listen and take
notes
– Members of section 2 should then outline
weaknesses of the selected approaches
– Finally, the chair of each group should discuss
the potential strengths, weaknesses, and
utility of the selected approaches
Encyclopedia Entries
•
•
•
•
•
•
•
•
•
•
•
Bias
Causation
Checklists
Chelimsky, Eleanor
Conflict of Interest
Countenance Model of
Evaluation
Critical Theory
Evaluation
Effectiveness
Efficiency
Empiricism
Independence
• Evaluability
Assessment
• Evaluation Use
• Fournier, Deborah
• Positivism
• Relativism
• Responsive evaluation
• Stake, Robert
• Thick Description
• Utilization of
Evaluation
• Weiss, Carol
• Wholey, Joseph