The Policy Process Model
Source: See Bretschneider
Module 5
Evaluation Approaches
Tip: Need a logic
model to evaluate well.
p. 147
Rossi, Lipsey,
and Freeman. (2004)/
Evaluation: A Systematic
Approach. Sage.
Adapted
from
United
Way
p. 95
Rossi, Lipsey,
and Freeman. (2004)/
Evaluation: A Systematic
Approach. Sage.
p. 143
Rossi, Lipsey,
and Freeman. (2004)/
Evaluation: A Systematic
Approach. Sage.
Program
Evaluation
Formative vs Summative
Formative Main purpose:
Evaluates how well pieces
of logic model are fitting
together (as you
implement
the program).
If
Summative Main
Purpose: Assessing
whether Short,
Intermediate, and LogTerm Outcomes are Met
a program fails, it could be because the theory is wrong (more
education does not reduce drug use) or because it was
implemented incorrectly (i.e. DARE cops made drugs sound cool)
Program
Evaluation
Formative vs Summative
A
If
B
a program fails, it could be because the theory is wrong (more
education does not reduce drug use) or because it was
implemented incorrectly (i.e. DARE cops made drugs sound cool)
Formative vs Summative
Formative Main purpose: Evaluates
how well pieces of logic model
are fitting together.
As program is
implemented
Often
strengthen or
improve the object
being evaluated
Video
Clip:
Office
Space
involves examining the delivery of the program or
technology, the quality of its implementation, and the assessment
of the organizational context, personnel, procedures, inputs, and
so on.
http://trochim.human.cornell.edu/kb/intreval.htm
short term
outcomes, those
changes or
benefits that are
most closely
associated with
or caused by the
programs outputs
intermediate
Long term outcomes or
outcomes, those program impacts, follow
changes that
from the benefits
result from an
accrued though the
application of intermediate outcomes.
the short term Tip: List out your outcomes
them put them in
outcomes.
chronological order.c
Summative Main
Purpose: Assessing
whether Short,
Intermediate, and LogTerm Outcomes are Met
http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V7V-3WR4K4M-8&_user=735929&_handle=B-WA-A-W-AV-MsSAYZW-UUW-AAUDUAADDV-AAUVCEWCDV-YBACCWCBC-AVU&_fmt=full&_coverDate=06%2F30%2F1999&_rdoc=8&_orig=browse&_srch=%23toc%235852%231999%23999779998%23102970!&_cdi=5852&view=c&_acct=C000040778&_version=1&_urlVersion=0&_userid=735929&md5=6e6b6f92758532f01a88230d7ec293
80
Plot of Pretest and Posttest Means for a Possible
Short, Median or Long-Term Outcome 1
step1
Step 2
A
Step 4
step3
step1
Step 2
B
Step 4
step3
Welch and Comer p. 9 Independent and Dependent Variables are implicit
in Logic Models
IV

DV
http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V7V-3WR4K4M-8&_user=735929&_handle=B-WA-A-W-AV-MsSAYZW-UUW-AAUDUAADDV-AAUVCEWCDV-YBACCWCBC-AVU&_fmt=full&_coverDate=06%2F30%2F1999&_rdoc=8&_orig=browse&_srch=%23toc%235852%231999%23999779998%23102970!&_cdi=5852&view=c&_acct=C000040778&_version=1&_urlVersion=0&_userid=73592
9&md5=6e6b6f92758532f01a88230d7ec29380
A Detailed Example of Design
Notation




Observations or measures (o)
Treatments (or programs) (x)
Groups (each is on a separate
line)
Assignment to a group




R = random assignment
N = nonequivalent groups
C = assignment by cutoff
Time (represented by moving
from left to right)
See Chapter 2
Welch and
Comer for
Research
Design and p36
for Pro/Con
Table
Yuen and
Terao 2003 :8
Applied Fact: The most widely used type of measurement
procedure in program evaluation is the written survey completed
by program participants.
Tip: once you know when you will
measure, decide what you will measure
Yuen & Terao2003
Choosing the Scale of
Measurement
VARIABLE
AGREE
NO OPINION
DISAGREE
Applied Fact: The most widely used type of measurement
procedure in program evaluation is the written survey completed
by program participants.
http://www.lafollette.wisc.edu/Courses/PA819/Categorical%20Data%20Analysis_Logistic%2
0Regression.ppt
Yuen & Terao2003
Starting next class, We will watch two
program overviews, for each we will:
Watch video
Practice Writing out notation
Discuss issues of validity
Practice the relevant stats approaches in
lab form
1.
2.
3.
4.

2 stats approaches:



Continuous Variables
Categorical Variables
Continuous
Categorical
You will use one or the other for each
outcome (short, intermediate, long-term)
Overview: To be prepared for next class you first
need to know what types of outcome variables
you have (just one each for short, intermediate,
and long-term i.e. 3 total)
Continuous
Data
Continuous
Analysis
Outcome
Response
Categorical
Categorical versus Continuous Data
Analysis
http://www.lafollette.wisc.edu/Courses/PA819/Categorical%20Data%20Analysis_Logistic%2
0Regression.ppt
Categorical
Nominal Variables
Variable: Kind of Beverage
OR
1
2
3
1
2
3
Order anyway you please!
http://www.lafollette.wisc.edu/Courses/PA819/Categorical%20Data%20Analysis_Logistic%2
0Regression.ppt
Categorical
Ordinal Variables
Variable: Size of Beverage
Small
Medium
Large
http://www.lafollette.wisc.edu/Courses/PA819/Categorical%20Data%20Analysis_Logistic%2
0Regression.ppt
Continuouspredict
GPA
Continuous Variable
s
?
associated
with
in.
lb.
Income
Is
weak
0
Weight
Purchase
associated
with
STRONG
Height
Income
?
1 Purchase
http://www.lafollette.wisc.edu/Courses/PA819/Categorical%20Data%20Analysis_Logistic%2
0Regression.ppt
?
Starting next class…
For your final paper: Evaluation sections
need to address 4 issues
(and include design notation)
Will you compare (measurement of objectives) by:
I.
Time?
Pre-test, post-test
Group?
II.
Comparison/control vs.
treatment/intervention/experimental
III.
How have you considered internal validity?
alternative explanations addressed
IV.
What statistical analysis will you use?
Labs 5a and 5b
Will watch two program
overviews, for each will:
1.
2.
3.
4.
Watch video
Practice Writing out notation
Discuss issues of validity
Practice the relevant stats approaches in
lab form
Validity
Does the instrument measure
what it is meant to measure.
External
A reliable
instrument
yields the
same
results on
repeated
measures.
Validity: the degree
to which the conclusions in
your study would hold for
other persons in other places
and at other times. (i.e. is it
reasonable to believe that the
program would work the
same way in other
communities?)
Conclusion
validity: the degree to which
conclusions we reach about relationships in
our data are reasonable, credible or believable.
(i.e. don’t claim more than your data shows – if
you note an increase in reading level, don’t
assume this means an increase in grades or
income)
Trochim Text
The Idea of Construct Validity
“Does
your program/ treatment accurately reflect/measure
what you intended?”
(i.e.
you say
more thorough
baggage
screening
reduces
terrorism, but
are you actually
measuring
changes in
screening and
terrorism?
Trochim Text
Construct Validity
When you measure what
you term "self esteem" is
that what you were really
measuring?
Content
Validity:
Whether a measuring
device covers the full
range of meanings or
forms that would be
included in a variable that
is being measured.
http://www.socialresearchmethods.net/kb/yinyangmap.htm
For instance, self esteem can relate to many realms of people’s
lives: work, school, family, etc.
A content valid would include indicators (i.e. questions) that
have to do with each of these settings.
Source: Sullivan 2001 Methods of
Social Research p.131-132
Internal Validity
Internal
Validity:
observed changes can
be attributed to your
program or intervention
(i.e., the cause) and not
to other possible causes
(sometimes described as
"alternative explanations"
for the outcome).
(i.e.)
there may be lots of
reasons, other than your program,
why test scores may improve or
symptoms may reduce, What are
they and how can you control for
them?
Baltimore
Watch video
Practice Writing out notation
Discuss issues of validity
Practice the relevant stats approaches in
lab form
1.
2.
3.
4.


Choose stats from chart
Lab 5a compare means