-II-
.;--I-I-..:II
--I711.I....
..,:,....I;p-.:,-II-...
Z.I-.1.,.-... I--:-,-;--..1,-.i1."-.-'-..-I.I;-i..711.,i..
., .1I..
-.
I -.I-I.II....--.-..-i,-I--I-I..-I1.--1,,....
I...-,I-..-I:-o-I.....-,-I-o.-.--,,,--...-1.-I.
-- ,,,I
-I
-.I,
:.,.I..--.,-,--.-I.--.
:,.--1:I--_:,.-:".--,.,..I-iI:I
,I-...
,I
I,.
- o, f,-I,
mIII-1--:,
,,:Z.I.1,I.-,.
I,--.
..
I.-.--..,
-I.--11.-.-,I,.-,,-,-.I:I ,III:-.,, ..II-I.-.,-I
,; "., -I-1,-----.,o..I..,
.--...,--,-,. II---.,.,
-,,:.:
-,Z,.
,,-i,.,
.- ...::,,i,,I.-..-.--.-.-I.,-.
,--.i"-.:..,I:.I
-,.,II,.I
,-,-- :,.:,-.I:--,.-,-..--,-.-.-,,,-:,-,,2,
I,.-.-..--,-,,-- :11.II.-.:
I,
::I,...-.I.
."-.
.,.--I,...,j,-...-::-.--,---.,--,,,
---I,.,----II---1..
---..
"'.
:1
.",
,
,
---:
-,,,
,--,.,,,,,,--Z.-.,,,-..-,-,:.;;,
;,.%.,,
..
,-.,,
-.
.-.
,1,
,.1-,; ,-.- i -,
,-I
..,--,,--.
,!.---I.1,.-.:,.,..--.I;-,,
,,,."-,-;.1 -,-,..-:,
.- --, ,,-,. I-I- . .,- .-,-I..1 ..-.
,!-;
--.
. ,.,--.-I,.-,
II------..rl.:,--,,
-,:
F,
a", .,.--1
""';, "
:--- , ,7-,-I
II.--.- -1
,,I .
:-'.----,
"
_,t--.
-,
,---,
,-,-I.%
,,--,..,.,.:-I-,. ;,,
IIt,1.I.
i,
I:, .
..-I. .----4,.-1,,-I.,,:,:-..,
-.
II--.--.,,,,..--i-:.e.-.,,-. ,.,,I.-.
:..I -.I--j.-,.-.,-,-,,,
,2,.-,
,, ,-', .- -,- - I-: ,
-1- -:,
,I. -. ,.,-,.I,,,,I: ,,. ,m , '.
-%
, . . -.
: I
:
-.,
;,.
I
I---- -I -,
I
, ".---.,:-,-,...,..I..-- II
.,..I. .
- --.
,.-,I,-, 4.-II.-.- .:--..- i
I
,,,-:z;,.,
,,,,.
-1
,,,,,: ,-,. ...I,-,.
....
-.
1-L,
-,.-. ",.lI,, -- --.---.
,--.-- ..
,
I....-III
-.4 :z-:,,
-,
, ,:.-,,.,,,.-,,..-,I--.-,.
-.w:...1,-,:,:.-.II,,-,.i---I- ..,,,II ,,-.,
.I---::.-.,.,,:.:;,--"-,,-,..,,
,,.I....-'ill
..
:.,,..
""
.,.I...
,,:-Z,:,I1,I.I-..-..: --........-,-,--,,,.
II- -t,7.
,z,
;. : --.
.
,;III..-.. .,I
I-.-.-', ,li -II,,-.
I,--,.,-,--,
----.
-,n-,
-:...11
-I
4.-,I:,.,
,-..I-I,.,I-I,-1,
,---,I;..q.
,...-..I
I.I,.,,.,..
.r ,-,-,
-..- .-"--,,..4,-.:-:-I--:--. -I.-.--II
..
,-,,.,.I..,.
: .: - -,--":1
,.--;
,.,,-- . ;---,,--, , ..,.-,.;:,,:
.,I,
,; ,,,,.i.,..-,,--,,"
,--,r-:.:
.- -7
,I1:
-,:`:,..
.,. .,,----,.-;,.,
,--"
..
:.:-.-,!-I,.
:---,
,:,
.,
,,1
,-,;-.,
,I-.
.;:.
-,-i"1:--:
.,
,., -.,,
.:,I.1:
,--,-,.,,--.
_-,-C-_-.
.1I.
!,
57
,-..-'
-i ,:;,,.-,,.-, -:i,`.---.--.1.,
;.
,--.-.-..,:,Z-;,
------`
-,
-,
-,
,-, :.;,-i--,,,..
-,
,:,:
--.-,,--,I.
...
::,,.,.,,,--,,I:.....,.--I,:....
, ----.
--.
7- -,,m-..-,I-.,,.
Ir..,--1
,,,-:,
--,I,-.-.-,.-,,II,--.I
I,--..II11
1,,.
,.::,----.,.I,
f,,I..,"-:-1 ----.-,,
--.. ----11,---4.,i,--7
,,,..--,
-,-.
I---,..,.,-,-,,-,,,-,
.",-I;-::,
,,-,!-.". ',,-,--II-,I
-I-..1..,--,
., ,:I.
....
:.m,-.-I,-q.IZ--.
I- .
,-`I.:-I--..
,-,
-;I.I..III1..--.
--.
.11
.-,!..--..;.--:.
,-.:-,-r:: ..:
.%.-.I-,,,,,--I,I%.I.-..,-;.I..-,..I
-',.'
::,...,-I, ,-II.--.
-.,
..: -..I,.,
: --, ,.
I . . ..
,,-..-,
I .:7
-.
I..!:I.--,I,I-II
,-I..,.--I-4.I---: .I-I.,- -- :.:.;I
,,-,,-,
1- :: ,-..-.-,,I-!.-.
----.--.IJ,-,- -- 1:
-I,.
II----,
,,.,-I:,---,.
II-. ..--- I
-.- '--- I:.:I..-I.II!,
-;-. .---,--.
,-,.II:,,,
III
,..,-q--,,,I-I.-.
,".--,.
...-. ---.--. .----.
--,
-- ,,.-,,
-:- !,
.
.";:
-, ,,, -,----'i,,
I. -,II:.
-..-.:,I1:,,--,-:i:,,,.-,I.--,:
-,.,m.:
,.,,-,:,.: ,-,:1
-`,
:.- ,
,
:
..
,----,.--I
.------,"'
.
:.
:--.:::.:-,.
,,o,--.II--,
-I-.
I.---,I--.-I,..
:I.
;
--:,-:IIP-.,-Il-:,
-'.,--I,:..I1.-I-,.-.II--i,
-1
I
.
J-.,1,I
,7 -1---.
. -,.I
I-..---.
I,
:
----,.
,
..
.-i ..I :.
.,o..-;,---"-,
1. -,.
,,
4.,I:: -`,--,-'.-,-,.
,--I
,-:t
,--,.,
,
,'.1:I,'.-,
-I-II
;
,I--,II--I.II
.
I-.:-,,
-.:-I
,,.-:,::Z
,-,I
-i
-,
.--:,-L,-:-,..
.,
,: -;
I:I..I
-I i.,.I,
,,
-.
, 7 , .. ,
.-- I,---!,,, ,-,I.:.:.-:
.- -.
.,.i
. I.I-----:,
:-.1;--.--.Z:
II-----,
IiII- .:cIII-,i,-,--I.I:"
:,:.----.
:- J,.:. -, A,,
P,--.-.,-,-,
-I-...I,
.,.,,
,
.-,
.
;I--.:I,II.I.-..:I-...1I-.
,-;1;-I.-.,,
I-.II
.,,I-;
1.
... ...
,
-I--,--rp-.II..--'-I-.,:--,-,,I.',
,-,I--,
: .:---,-.,:,% :-..
,7,:-,..,:I,.
. I:,I,,,-,dI,,
,.'", -,...--,.
-:1II---ZI..
,..-,
:..,
..
-,
-,,
,
.,
:
.,-:I.Z-II..-,--.--,.
.%
-.
:,
- -,.
-:. Z
-, .-.-i11,
-:,
.,,-,.-I-; .-7
-::-,
-I!-II
,I:,.',
I-.:..:;,
.,,j
.,::-,
I
.11
:1.;p.-,.
,.--:.
,--.-.i
.-..,
-,
,.
-;I
,:,-,
-,,.f ,-1%,
:..---,-,.,:--.
.
.,, ,.;,-..,.-I
,--.,-.V-.....I-,..
,,Z
,,-:.I
-,-..,I
I-I-,I,,,,III
.,
,--.
..,,,.,..11--.-I.
.,.II. ... --,--,-,
----,
-...
,,
-::-I.,,I-.,--,.-,
..
. .Z.
,, q,.,,.,.,
.,I,,
:. -:1,,7:..,.-.:,,,"
,,"..-,
I-..I.
-,i,:. :,.-i.-,,-.,,--".":-,,
:
,-:.,,-I,
,.v.
",
,I-"..-,:,
,,,,
.,- -:--, ,, : ,",
..,,-,,.
,-,-_:,,,.. 2,,,,,,..,-:,
,,--., ,
.-III!
.i-,-,,.;
._r,
I.,.
III :;,.-:..
,I:....-,-I
,;.--.-.,,-, ,.I -. ,.-;-,4-.,--.-.:
.1.-I-,
I.I ,
,,
,,,,I .: .,
-,,-- ,:."-.- ,. I
-. .---_-,
-,-,.-;,-.-.
--,-.I ,.-., ,..,,.,.,,Z.
, ",.",,,'.,.-,
,-J , ,,.-.,
:-"..,,.I,,--,-.
,,.II
:..--,
-,--1--.------,.---,
, ,- Z
F,-,--.- .t.-;',,--.-.,.,,.
---.:..-,,.,.-.. ,,:Z-I
":--..
I,-...: I .,,--I.II
I : I-.--I
1.-:
,,:,,IV--,.:
- ,.
,:-,;I-----I
I:..:,...
I,-.-,-....m:,.,.,-.., -,.-I"-.-I .- .,:,1.,--,.,I-I.....,
,.-- ..,`: :',::, :-I
,. .,,.
,1,- .,.,-.,.;,. -I--:
,.. .. , .7
-1. -;
-I..
: '.,:, 1,:,4, , I-..I..I,-..
a,
,-, :-..
-,
.II
I
,
;
11,II--I.,
.I-I
I-.-I",--;-.
,IT,",
,-.I-..1
-;'-;7-,.
I,.1 ..: -.-.
,..
II.
..11 I11I,,
----17:',
, smalk,--,: -,,-- l.
.i,..
,,t-..
...
,,,5i.-. ,
..-..-.:ID
-,,,-,
--1
: ..--..1I..r-W.
Is,:,,,. .
. -,-,:.I.-,, :%..w'r
,.I.
".
,---,
.---,---.,..,.:
:
.,,:
,I.,
-.
r%:. ,
1: -; : iI.-7,,-,..
".:,.:.,--.:-I:,.
: ..
r,,.I.r...
,r.,---:,-.--,.t
--.... :- L7--.
.,.--,
-.L
-_
.I.,- :-.--.
i,..,Ir,,,,. -,,-,
,.I.-.O
,.,` " -. - ,,- - ,-,-,,.--.,--,I.I
-- -. I..-,
1:.,.
",,,.,
--.,--!I
,,-:
r-, -,,-,-.
11..--,.11r,,I-.j,.--,,,:I,
-,:
,,",.I:.---,
,--,,,.-,-,.-.
: , ,:, ,,.I-,..,.-,-,--I-..
-.I-1.-..
II ,.;4;I
.,,"-11
.4--....,4Ii
1: ..
' ,.1.LII.,-I.I,;
,-I1ri
,,.
..
,,-v
;1,:::
,,-,
r : .__,:,I
.,,
,1,-,-I,.
--I -, ,.
.I._-..-,
, ". ,, i.
.1
,.I.IS.
: I,o.pI 4,--.i:I:Ib,.
,,-.. ,-: ......
4-,,7-..
,..
,.
.,-,
-.I:;,-,,
,--I , 1.,.-I,
..I---.I,
;--"
,.,-:::i
.,
:
,,,
,
:.1
.,.,..Ii:--,,-...
:.,
-I:14
.,."....:.-.
.,-'L-,
III--..,
,,,,--m11. ;:--I,1,.-Ir.I-.-.,, ,: ,,II
II.,I. I
I.,,-.-,,,..,
,,--,----, -,,.-,,:.,I.-,.:..
,-:-,.:,I...vII.-,.,
.--:i.--1
,,-1,I....IIr,,,,.I,: .I.:: ` - ,,:
:.,-.:
----I. ..
1.,--,..
,-,,-,--r:.--.I,7,.::,.---1.
.-...,:-.-.,:,
-..L...,
,.
::,,-III
,...1.I-,.--,,,
-,
I---I.I,-I-7.i-I.-..
,,,-I.
-,..
:--.-:. :",
,,,--I,.:-:,-:.
,
-...-.-,.:.
-,--::---,.-I-rL.
.--I:2.-I
I..--.-,..
.. ,-I , - I-----,,.-r..II...-.:- ..
,,.
--.
:-.
I--.
,:-.I-I-----,--:"
:. .,I-i...
,I,-I,,I..i;-:1.
-_---:r
.II;,I.
,,,.II,.,.I:--..,
Z
I
:1, - ':r,-'.'".I.-,.;,--.I.-.--I:..
m.,
: .."
II-..
:.,1.I.
-.
-1-..
,,,-11::
.,,.
.I.-I1.L.I
,,-,-.
,,,---,.--.--,I-._,_.L.I.%.--,
I.---,-411-''--,..
--.
,I-i--I-,
:,..",.1----:-,,I
q----I,I.,..I--:-.
...I,-I,.-..
:4----I
....
I%
--Z,-,,,..,,,.
:Lj,:I-L-.,II..I
.-.---,',.-7-,_-..-I-.I1
.I-P
,-:-.1
:-.,-4:'
1,
.- ;,-I..I.I_L
--L :1..I-I.,-.I..--I-.:-II-,--II,..Z
.-..
--.
L-I-.
. II:.'I--I- -L ,,--. ,..`I I-.
.,,..
.. Ir.I.,.:
.I. ,-1---._,
- .,-,_I
j-,-,IL:t-I--:-I1,,I-,--,-,-,.-L."L-.,.I
,,.I-II..-.--I'
,,'_I-.-,.-,,,%
'-. -,., : '
' :
: - ,.. -' . ,,
,I..-, . I , -.
-I
-,
,I
-,-,.'
,:
.
L
_
L.
L
,I--.,--,1-7
.
11-,
.L--.-,,.L
.
.I .
IL , .
- . ,-: -.
: ',-', - I I
-,-.- -,,
I
: r,: - . .--,
,r.,.:-,,-..Z:--.--,_
II.1.
-,..,,,-, :::-.-. ,,.,.,
:,
I-.,- -:,,
--1I.II---:-_.:-:L
,--_LI.
:.:: ,: -1 I-.
mII:I
i :,,--Z
:II.I-.74.,--..:i.
.-7.I1.-_,,.-.:,
L.,.----I-,___,.
.L.L-7.--7
:,I-,,.---I,-,I-,_..-,
.L.I--L,,-,_.,;:,
,I._,.
-.
._;I-.,,-,I.
,,.,
Ip-,
t;I,_,.,-:-L,..I,,r..-,-.
I,;-1..%-LI
,I;..I.---.
:1,.,I I .1-: .,. :I.I..:-;:`I--..LII,;I
LI--.%
-'-II11.-,.,.:
):...I-T.
,.-r,-.-,",_I-----,
:, --,,,'LL-,1.,-.
-I Ii
-.1L,.-L,..II
,: I.I-:.,Z..-,.
.! ..I:I-I--:-.I-i...-I-.I:,,L-I!---,-I"I
,: ,.
,L
,..::-.
..I- . .Ll
. .LL]::
:.-.,.-rZ.,
-,.:,..
,.-.I
-,
r,II,_.",,..II.1-':----:,
-:-,-..---:'...
L,-.-_..,-'l
L.-:-.,I.,:,.kl
-,-.
:,---.1.:
, ,
:,;.,,L','
-.
,:: ,-,..I---,:,
,-, ., -I I-: -..-,,, ;:;.--Ii
,I,.:II
:L...
:.iL1....--.I..-,,,.
7
,X.-'L,
:___,-I.
-1
i,
.I-.-..LII
-i -I.iL--L;,
.;.:I;
.-,
.: ,::,,Z,
'-I'-.,,I.,..-I-...L-,';,L .,_,
.,:--!"'-',.
.L.1..r.-L1
.-.
L,-,_.
_,,I
I._....
-: .
I-I,.'-.-.-,.
,.,-,:,I-L-.. -I.
l I.II-I,IL_.-j,:,.;L.-.,
I.I:.'.IwI..-_-1
II,-.-.
r.
Ii.-...
-i,:%
II:',
: ,
.:.-'...
:
II.
I..,,..L.-L.
-, _ .. ..
---,I
_-_, :'`,:I..
;,i: ,..,,--:-.I-:
,i,,. ..L,-c-:!:
I ,-L..,
.,.,I
II..IL.
I . :__ :1
:-,,-,I.,I-I.:--I-..-,Ii
-,I
,I; -'L'-:.--,,,,, IL--7.N-1
i";!.,I,-,-r..-...,I..;
I..-,I.-.,,,:.I
-I.
..
IITI Li,--,--I
,.,,
-I;I..,11--_--.,,,I.-_,
- ! I--I-.L:.I.,,.i.,;,-,.I-I-...I-7.L,:
I
ZI-"-:...I.1,,LI...,._
IrI
-..,,.I,,.,:,,p.-.-I
11,I.
I, -,.:`
I-.I-I:z:,
-,I,-?
,L,-----_.L_I.
:, ,:--.,-,--I,-,:,
.------: I,-,,,.-,:.
,:
.I---.
..I..
'-.IL.--,,,,,I.I:,:-..
;,--,II.: ,-,---LL I.-I.
-::-:
:--,,,-.,,,-,:..---1,:!..,.I-:'
,
-:
- ,-,----, -,-,
,..
,7.,---,,
-t
,Z.
.o
,
,i
I
.
I,I
-.
.',-----.
.:
,-'--Ir..I-,,-:
:
.II.L. L.-I.-III.I.--.-ILI
I-1.-L,,,,,-.
., .--.
I
..
I .,4I..
:,-:.,-...I,-.,.1
.
-:,I-.,.,
,-I.rIr.I
-,.L:;%_
-- ,:'-,I.--...II--I
I4
i,:
I II--:
;:
I..I.,
-I,
:, Li
, :,-,.,-kL :,--.
II..,
-.
L-L-.I-,Ir-,::L,
.. 'II,-i,I....I...I,__.
-.,:-Z.Ir..
6II_,,
,,-.-"11
i.,,
..I-711..1%
- ,-,,'_-:
, ,,,'x
...--,._.II-I-._..__:..
-_:,. ,:,-.I,
f,-II.I-..-,-.---1._,,;
L,.-.,
" ,__M" L,_'-.
.I ,-.,
. .I...I-,IL..
,.1,_-I-.
I.-I.-L
II
...-,
,,.
,_LL.-I--I.-.
..L.;, ._I,,,:, _L
-,-,.q
.
.
1,
--.
,
,
-,
II
,!,oI-...
-1
4:,,
-,:.,:'
,,r,
:,
-,, :,Z,;-,,I-.,
L..,,,
;-I-I.I.
.I'--iiI
.II.-I.L-.
..r.,-I.11.,.-.:
,---,-
,----,I-.I-ff.,--,-.1
-.--0 -,-Ipm---Z-,
,,---- ,iL^...
ll"I..
, Am,
"
."
-,,-`r7-."-.7
,-:,.
'',-I..
,.
-:.--. _,, I.,,...--- -- -.1-.,-...
-,.
---,.,,,;-,"I-rl:
"-:,.-.-
I.r%
I,-,- ,.- --- .-.I1:
:-96-P.-,
I--.-Ki-.
:.
-1I
--- ,-, 1,I, ,-11I-- ,---,
,-.,
,:-,,
I-_,
I ar,''M : , I.
. 1.-I11.
I- il.-, ,,--.1.-:_,,,,,..-,
1,_::
,--.
,...-:-.
-'I,-
, %O-3,,%:-.
INJ
- I Ft.-.---I.
,-., -.'.---
7-.I.-kL
&-;:,.,'.I-.-1:,::--
.- ,
-'
-.-:jt-I.!IL-.
.-,.II.-I.I-I.
-.
--,:--
-,-,,..I.,_--'---,':
. 1.II.
,"..,.
... ,.;
OFTECHOLOG
,,;.........,...-
.............
-
X
'0'y0Et~fV0JEV' H
R)
-00
........
ON EVALUATING EVALUATIONS
by
Richard C. Larson
Leni S. Berliner
OR 096-79
December 1979
Prepared under Grant Number 78NI-AX-0007 from the National Institute
of Law Enforcement and Criminal Justice, Law Enforcement Assistance
Administration, U.S. Department of Justice. Points of view or opinions
stated in this document are those of the author and do not necessarily
represent the official position or policies of the U.S. Department of
Justice.
Table of Contents
Page
Abstract
i
Introduction
1
Evaluation Inputs, Process, and Outcomes
3
Evaluation Inputs
5
Evaluation Process
7
Evaluation Outcomes
12
Why Evaluate Evaluations?
16
Evaluation Documentation Requirements
19
Summary and Suggested Research
21
Notes
23
References
24
Figure 1.
Evaluation Viewed as a System
4
Table 1.
Inputs to an Evaluation
6
Table 2.
Process Components of an Evaluation
8
Table 3
Evaluation Outcomes
15
i
ABSTRACT
Researchers in the field of public program evaluation have recently
been arguing for comprehensive evaluation of programs, evaluations that
examine program inputs, processes, and outcomes.
In this paper we propose
that the same approach is useful for evaluating evaluations themselves.
We identify tentative sets of evaluation inputs, processes, and outcomes
and describe their utility in evaluating evaluations.
Since evaluations
are commissioned to produce information useful to decision makers, it is
suggested that the only fundamental outcomes of an evaluation are
decisions influences by the evaluation.
This fact precludes evaluating
an evaluation solely from the final report of the evaluation, which is
usually written before the relevant decisions are made.
However,
evaluators can and should document useful information about evaluation
inputs and processes, and recommendations along these lines are included
in this paper.
INTRODUCTION
Increased attention has been focused in recent years on the merit
of various public programs.
Legislators, citizens' groups, and others
have been concerned with the extent to which programs meet their
objectives.
programs.
Program personnel want to know how to improve on-going
Researchers as well as legislators have been interested in
programs as social experiments, in attempts to validate or refute various
intervention theories which may later lead to new standardized (nonexperimental) programs.
All of this concern has pushed the burgeoning
field of public program evaluation into the limelight.
many promises:
Evaluation offers
accountability for public expenditures; improvement for
on-going programs; identification of promising new theories leading to
new programs; even generation of new research knowledge.
Yet evaluation to date has seemed to flounder.
evaluation work has been spotty.
ingly small impact.
The quality of
Evaluation findings have had a disappoint-
While considerable attention has been given to the
evaluation of programs, it is somewhat ironic that relatively little
attention has been devoted to evaluating the evaluations (and the
evaluators).
(Except Bernstein and Freeman, 1975; Cook and Gruder, 1978;
GAO, 1978a and b; MSR, 1973; Cook, 1978; Stufflebeam, 1974).
can an evaluation be said to be successful?
On what basis
Attempts to answer this
question have often focused on one sub-element of the evaluation, namely
the technical aspects of evaluation process--as reflected, say, by
-2-
the statistical methodology employed or the survey research methods
used.
This limited focus creates problems analogous to those which
would occur if one conducted an evaluation of a program, exclusively
examining some part of program process, while ignoring program inputs,
outcomes and other aspects of process.
Thus, in the same sense that
evaluation of programs requires a comprehensive analysis of program
inputs, process and outcome, so too evaluation of evaluations requires
analysis of evaluation inputs, process, and outcome.
One of the goals
of this paper is to lay out the elements of such an approach.
In identifying the various components of an evaluation, particularly
outcomes, we are guided by the fact that evaluations are commissioned by
an individual or group to produce information which hopefully has some
decision-consequential impact.
More succinctly,
1.
Evaluation is a process that produces information;
2.
Information is useful only to the extent that it informs
decisions;
3.
A decision is an irrevocable allocation of resources;l
4.
Thus, evaluation is a process that produces information
to assist in the allocation of resources.
The allocation of resources referred to here must be considered broadly.
It can range from obvious programmatic changes (e.g. shifting staffing
patterns or operating procedures), to a change in program budget level
as determined by the funding agency, to a congressional authorization to
spend more (or less) money on similar programs nationally, to a change
in research efforts by one or more researchers who may want to devote
more (or less) of their resources to that programmatic area.
It can
-3-
even refer to decisions by memebers of a program's client group to increase
(or decrease) participation (e.g. allocation of their time and energies)
in the program.
Much confusion, we believe, in the evaluation literature
can be traced to a lack of attention to the identification of decision
makers and decisions to be made (resources to be allocated) as a
consequence of an evaluation.
Our discussion of evaluation outcomes
will be motivated by this decision-oriented point of view.
EVALUATION INPUTS, PROCESS, AND OUTCOMES
Any evaluation is a process, having inputs and yielding outcomes
as illustrated in Figure 1. It is our contention that any comprehensive
evaluation of an evaluation should examine all three of these evaluation
components.
An examination of process alone may, for instance, verify
exemplary technique, but reveals nothing about evaluation impact.
A
review of evaluation outcomes alone is not sufficient for explaining
the causal mechanisms linking evaluation input through process to those
outcomes; indeed, one who analyzes only outcomes is often hard-pressed
to attribute outcomes to the evaluation.
An examination of inputs alone
reveals little more than the collection of resources mustered to conduct
the evaluation.
In this section we present an initial listing of detailed
elements comprising evaluation input, process, and outcome.
-4-
Evaluation Inputs
EVALUATION
Evaluation Outcomes
.
PROCESS
Figure 1:
Evaluation Viewed as a System
r1
-5-
Evaluation Inputs
The inputs to an evaluation may be considered to be an inventory
of resources and methodologies brought to bear on the evaluation, and
a description of the evaluation setting.
is summarized in Table 1.
Our proposed set of inputs
Obvious inputs include evaluation budget
(both in absolute terms and as a percentage of program budget), duration
of the evaluation, timing of the evaluation with respect to the program
being evaluated, and skills (and other attributes) of the evaluation
personnel.
Despite the indisputable importance of these items and
despite the urgings of evaluators to consider program inputs during
evaluations, few evaluators themselves document these rudimentary
evaluation inputs.
Any comprehensive evaluation of an evaluation is
thwarted at an early stage if these inputs are not known;
Other necessary inputs also listed in Table 1 include attributes
of the program being evaluated and its personnel (e.g. training and
experience, determining their attitudes toward the evaluation),
evaluation methodology and design, and audience or client group for the
evaluation, and the purpose of the evaluation.
Here, purpose of the
evaluation refers to the resources which one or more client groups
(decision makers) are likely to (re)allocate as a consequence of
information provided by the evaluation.
A description of evaluation
methodology and design should include not only the statistical procedures
to be used to analyze data, but also the entire plan for considering
program inputs, process and outcomes.
Moreover, it should indicate
milestones along the way, points at which information may be fed back
-6-
Table 1
Inputs to an Evaluation
Evaluation inputs; An inventory of resources and methodologies brought
to bear on the evaluation and a description of the evaluation setting.
(i) Budget of the evaluation (and other material resources
available to the evaluators).
(ii) Duration of the evaluation.
(iii) Timing of the evaluation with respect to the program being
evaluated.
(iv) Attributes of evaluation personnel (e.g., training, experience,
"world view").
(v) Attributes of program personnel (e.g., experience, commitment,
education).
(vi) Attributes of the program being evaluated (e.g., goals, substantive
area of concern, clieht group).
(vii) Evaluation methodology and design.
(viii) Audience, or "client group", or decision makers for the evaluation
and purposes of the evaluation.
-7-
to program staff for possible program modification, and other points at
which information obtained so far may be used adaptively to change
evaluation and/or program procedures; rules for such adaptive change,
to the extent possible, should be stated explicitly a priori (i.e.,
such rules for adaptiveness are themselves evaluation inputs).
Evaluation Process
Evaluation inputs are utilized to develop information which leads
to evaluation outcomes (i.e., decisions to be made).
of inputs is evaluation process.
This utilization
Paralleling the idea of program process,
evaluation process is the actual conduct of the evaluation as compared
with that planned in the evaluation design.
evaluation process are given in Table 2.
Our proposed components of
As the discussion will indicate,
this list is meant to be illustrative, but by no means exhaustive.
The
first three elements of evaluation process relate to the interaction
between the evaluators and the program being evaluated.
Obviously
important are (i) the types, intensity, and frequency of interactions
between evaluators and program staff members.
This, in turn may lead
to (ii) (an inadvertent) response of program staff and client groups to
the presence of the evaluators (e.g., "Hawthorne effect").
While this
response may be difficult to detect and measure, it constitutes an important
element of evaluation process, and attempts should be made to estimate and
report it.
The final element of evaluator/program interaction is (iii)
the extent to which information acquired during the evaluation is fed
back to program staff, perhaps modifying program procedures.
This reflects
deliberate transfer of information that may result in changes in program
-8-
Table 2
Process Components of an Evaluation
Evaluation process: Actual conduct of the evaluation compared with
that planned in the evaluation design.
(i)
(ii)
(iii)
(iv)
(v)
(vi)
(vii)
(viii)
Types, intensity and frequency of interactions between
evaluators and program staff members.
Response of program staff and client groups to the presence
of evaluators.
Extent to which information acquired during the evaluation
is fed back to program staff, perhaps modifying program
procedures.
Extent to which information acquired during the evaluation
is used to modify the allocation of evaluation resources.
Adaptivness of evaluation design (i.e., capacity to respond
to changes in the program) and history of adaptations.
Turnovers in personnel (e.g., evaluators, program staff,
client groups of both program and evaluation).
Testing of hypotheses regarding the program.
Documentation of findings.
-9-
operation, in contrast to inadvertent interactions identified in (ii)
which may also affect program operation.
Item (iii)
relates to the
controversy in evaluation about whether or not the evaluators should
"board the train" of the program or ride along in a different train on
parallel tracks (Rossi and Wright, 1977).
On the same train, they can
provide information thought to be useful during the course of the
evaluation, usually yielding changes in the program during the evaluation.
The two-train model precludes such useful exchange of timely information.
In our view, a reasonable way to tackle this question is to view the
choice as being directly dependent on the intended audience or client
of the evaluation.
If, for example, the evaluation is being done
primarily for the purposes of the program clients or program staff, then
it appears that the evaluator(s) should "board the train."
When, however,
the evaluation is being done as part of an independent assessment for a
funding agency, then to avoid possible conflicts of interest the "two
train" approach seems more appropriate.
The next two items relate to the adaptability of the evaluation.
First, one is interested in (iv) the extent to which information acquired
during the evaluation is used to modify the allocation of evaluation
resources.
Is the evaluation in a "straight jacket" design, or do the
evaluators modify the design in response to information obtained so far
in the evaluation?
Adaptability may be reflected in elements of process
evaluation such as the allocation of participant observers and/or
interviewers to various parts of the program.
Or it could relate to the
sequential adaptive generation and testing of alternative hypotheses
-10-
regarding program operation.
It is our conjecture that many evaluations
in practice are adaptive, but, lacking rules and encouragement for
adaptability, the evaluators in their reports are likely not to describe
this element of their evaluation.
A related issue is (v) the adaptive-
ness of the evaluation design in response to changes in the program being
evaluated.
For instance, during the operation of the program, an
employee strike could occur, a new relevant law could be enacted, or a
citizens' group could protest against some particular aspect of the
program.
To what extent is the evaluation jeopardized by such program
changes and interruptions, and to what extent can it adapt to them?
This concern stands in contrast to (iv), in which the program was
assumed fixed, but evaluative information about components of the program
changed over time, perhaps resulting in changes in evaluation tactics.
No evaluation design can stand impervious to all conceivable unforseen
changes in the program and its operating environment, but some are more
robust than others.
A chronological history of adaptions of the evaluation
to changes in the program would seem to be an important part of evaluating
evaluation process.
The next item is (i)
turnover in personnel (e.g., evaluators,
program staff, client groups of both the program and the evaluation).
This is one of perhaps several internal unplanned changes in program or
evaluation process.
However, it appears to be a critical one, in that a
change in one or more key persons in the evaluation or in the program
can markedly affect the outcome of both.
We are aware, for instance,
of at least one evaluation that had three different directors during
the course of its operation; one can speculate as to the quality and
-11-
impact of the final evaluation.
A change in the client group of either
the program or the evaluation is also important.
For instance, a
significant fraction of evaluations that have had little or no
eventual
decision impact appear to fall victim to the "vanishing advocate"
syndrome, in which the person who originally commissioned the evaluation
has moved to another professional position, only to be replaced by someone
unsympathetic to the original purposes of the evaluation (Chaiken et al.,
1975).
The seventh entry in Table 2, (vii) testing hypotheses regarding
the program, is one of perhaps many elements of analysis of evaluation
findings.
We believe that all such analyses, both statistically and
subjectively based, constitute a part of evaluation process, not outcome.
It appears to be this element of evaluation process--evaluation technique
and methodology as applied in practice--that has received most scrutiny
by evaluators of evaluations.
Perhaps this is because manipulation of
numbers is one of the few elements of evaluation process that can be
replicated and scrutinized by others after termination of the evaluation.
And statistical procedure is one of the few components of evaluation
process in which apparently universal "scientific" measures of accountability can be applied.
But one should not fall prey to the trap of
misplaced emphasis on statistical method.
A statistically elegant
evaluation may be seriously flawed in other respects; and statistical
correctness by no means guarantees decision impact.
On the other hand,
a statistically flawed evaluation may indeed present imperfect information to decision makers.
The imperfections may lead to decisions that
would have been improved if more accurate information had been available.
-12-
the "costs" of such imperfect information can be considerable.
Yet, when
balanced with other components of evaluation process, it is quite possible
that a statistically flawed evaluation can still present useful information to decision makers--where usefulness implies decisions being made
that are in some sense "better" than those that would have been made in
the total absence of the evaluation.
In order to place in perspective
the importance of statistical procedure and to estimate the cost of
statistical error, we would argue strongly for a comprehensive evaluation
of evaluation process, as reflected by the other elements in Table 2.
The eighth and final entry in the Table is (viii) documentation of
findings, i.e., creation of the "final report."
While the final report
may represent a key product of the evaluation, we believe that it is
related to evaluation process; it is not a fundamental evaluation
outcome.
Like several other entries in Table 2, this one also could be
broadened, perhaps to "communication of final evaluation findings."
Certainly the final report--its structure, content, level and style of
presentation--is an important part of the communication process.
But
also important are oral presentations, use of teaching aids to convey
the essential results, and other activities and devices for communication
and dissemination of results.
For instance, a methodologically flawless
evaluation whose findings are unintelligible to decision makers will have
at best marginal impact.
Evaluation Outcomes
For many public programs, the boundary separating process and
outcome and their corresponding measures is indeed fuzzy.
As one
-13-
example, "citizen and client satisfaction" with a particular program has
been considered by evaluators both as a process measure and an outcome
measure.
However, improved citizen satisfaction as an outcome measure is
problematic; its basis may be illusory (e.g., citizens may feel safer
when true crime rates increase; students who "like" their teachers may
in fact be learning less, etc.).
Extending this example to our concern
here, program staff satisfaction with an evaluation may be well founded
if the evaluators--funded by the program in order to help improve the
program--helped the staff people to discover ways of making their
program more effective.
On the other hand, such positive feelings
toward the evaluators may be little more than self-serving if the
evaluators-commissioned by the program's funding agency--find nothing
to recommend discontinuance of the program.
Again, a dilemma in
evaluation is resolved in part by examining the primary client group
of the evaluation.
Any attempt to demarcate the boundary between evaluation process
and outcome and measures of their effects is done in the presence of
ambiguity and controversy.
Still, the inherent difficulties should not
act to preclude discussion on this vital matter.
In an attempt to provide
one input to the debate, we take a relatively firm stand on evaluation
outcome, motivated by our decision orientation:
ultimate outcomes of an
evaluation are the decisions (resource allocations) influenced by the
evaluation.
All other outcomes are process outcomes.
Most evaluators
discover the decision consequences of their evaluation only long after
submission of the final evaluation report, if at all; because of this it
-14-
is inappropriate for those who evaluate evaluations to judge their impact
only from reading the final report (Larson et al., 1979).
The time period
of the evaluation of an evaluation must extend beyond that of the
original evaluation in order to assess its ultimate decision consequences.
As a first cut, we have identified the different types of decisions
that may be affected by an evaluation as follows:
(i)
"the funding agency's decision": decision by funding agency
to fund, refund, modify or cancel the program;
(ii)
"the program staff's operational decision":
decision by
program staff to modify any of the program procedures;
(iii)
"the program client group's decision":
decision by members
of the program client group to increase, decrease or
otherwise alter participation in the program (assuming
they are informed of the evaluation to begin with);
(iv)
"the research community's decision":
decision by one or
more members of the research community to study (or not to
study) further the questions raised in the evaluation;
(v)
"the decision of those involved in related programs":
decision by one or more other funders and/or program personnel
in other jurisdictions to initiate, modify or terminate
similar programs.
Decision makers under (iv) and (v) above may be said to be "second-order
decision makers," since they are not directly involved with the program
being evaluated or the evaluation.
The evaluation's ultimate impact on
these decision makers will in many cases be very difficult to assess.
-15-
TABLE 3
Evaluation Outcomes
Evaluation Outcomes:
A listing of decisions influenced by the evaluation.
(i) Decision-by funding agency to fund, refund, modify
or cancel program.
(ii) Decision by program staff to modify any of the program
procedures.
(iii)
Decision by members of the client group to alter
participation patterns in the program.
(iv) Decision by one or more members of the research community
to study further the questions/issues raised in the evaluation.
(v) Decision by one or more other funders and/or program
personnel (in other jurisdictions) to initiate, modify,
or terminate similar programs.
-16-
Recalling that retention of the status quo is also a decision (although
occasionally it is mislabeled as "no decision"), then it is useful to
note that each of the decisions cited above will be made whether or not
the evaluation is conducted.
(For convenience, the decisions above are
summarized in Table 3.)
WHY EVALUATE EVALUATIONS?
Evaluations are performed to provide information about a program to
decision makers.
Thus, evaluations of evaluations are performed to provide
information about an evaluation to a possibly different set of decision
makers.
For instance, an evaluation of an evaluation can provide an
independent assessment for decision makers of the quality of the information
presented in the evaluation.
This would enhance the extent of "informedness"
of the resulting decisions, but at a cost--the cost of the evaluation of
the evaluation.
Clearly if this cost exceeds some threshold, its marginal
information value may not be adequate to justify its cost.
In this context,
the decision to evaluate an evaluation is also an allocation of resources
which may or may not be justified at a particular point in time, given
one's knowledge about the original evaluation, the program being evaluated,
and the marginal cost and expected marginal information content of the
evaluation of the evaluation.
In another application, a collection of evaluations of evaluations
(and evaluators) could be used in the selection of an evaluator of a
-17-
program.
The selection criterion would probably include a trade-off
between evaluation quality and cost.
Clearly other reasons for evaluating evaluations abound.
One,
for instance, is to pool together in a "meta evaluation" framework
the results of many different evaluations of similar programs to obtain
synthesized research results.
This and other examples of evaluating
evaluations are discussed by Cook and Gruder (1978).
One perhaps not so apparent reason for evaluating evaluations relates
to the evaluation enterprise itself.
Suppose one or more sets of decision
makers are concerned with examining evaluation per se, and are not
concerned directly with the program being evaluated.
For instance,
a potential funding agency may be interested in evaluating a particular
evaluator, or a researcher may be interested in evaluating a new
experimental design procedure.
Then, in what we have discussed so far,
"program" becomes "evaluation," i.e., that set of activities having
programmatic goals and objectives.
to evaluating a program.
Evaluating the evaluation is analogous
Correspondently, there is a set of evaluation
inputs, processes, and outcomes directly analogous to those listed in
Tables 1, 2 and 3 for the evaluation of a program.
As an example, it is
instructive to consider the possible decisions that could be affected by
information produced by the evaluation of an evaluation.
The "funding
agency's decision" could relate to supporting additional research related
to the specific methodology employed in the (original) evaluation. 3
The
"program staff's operational decision" could refer to potential modifications of evaluation conduct by the (original) evaluators.
The "program
client group's decision" could relate to decisions by those awaiting
-18-
information from the (original) evaluation to utilize the information
in different ways.
The "research community's decision" would most likely
be a decision by one or more members of the research community to study
(or not to study) further issues related to that type of evaluation.
The "decision of those involved in related programs" could be the
decision by one or more evaluation funders and/or evaluation personnel
in other areas to initiate, modify, or terminate similar types of
evaluations.
Thus, the comprehensive evaluation of evaluations has the
potential to provide the full spectrum of information and decision
consequences for evaluations that is expected of evaluations of programs
for programs.
One final note of the utility of evaluating evaluations:
any
evaluator of evaluations--like any other evaluator--is imperfect, requiring
then an evaluator to evaluate evaluators of evaluations.
But this new
third level evaluator is also imperfect, requiring yet another evaluator.
This potentially infinite morass can be avoided by considering the
expected marginal costs and informational utility of the concept of the
expected value of perfect information; this is the expected improvement
in the decision maker's outcome measure that can be attributed to obtaining
perfect information (i.e., eliminating uncertainties) at the decision stage.
No decision maker should support a hierarchical evaluation superstructure
having an expected cost greater than the expected value of perfect
information (Thompson, 1975).
-19-
EVALUATION DOCUMENTATION REQUIREMENTS
The framework described in the previous sections has implications
for documentation of program evaluations.
Given our definition of
comprehensive evaluation and our listing of inputs, processes and
outcomes, the information required to do a comprehensive evaluation of
evaluations can become considerable.
Whether or not an independent
evaluation of an evaluation is to be performed, documentation in the
final evaluation report of inputs, processes and outcomes (as far as
possible) of the evaluation would seem to be essential.
As we discovered in an empirical study of criminal justice program
evaluations (Larson et al., 1979), current evaluation documentation
practice is uneven and sorely lacking.
Of the roughly 200 studies
in the sample, only 4 percent indicated the percentage of the program
budget allocated for the evaluation, and only 2 percent indicated total
evaluation budget.
Thirty-one percent of the reports in the sample
indicated the total duration of the evaluation, while 8 percent at least
indicated (though not aways expZlicitZy) the timing of the evaluation
with respect to the program being evaluated.
None of the reports
described the professional or other attributes of either program staff
or evaluation personnel.
Finally, while 90 percent of the reports made
at least some reference to the context or purpose of the effort, actual
potential users of the evaluations were rarely identified explicitly.
Only 58 percent of the reports contained an analysis of program goals,
and only 47 percent discussed the program's client group in any way.
-20-
Evaluation process components fared much worse than evaluation
inputs in the sample of final reports.
The constituent parts of
evaluation process listed in Table 2 were rarely if ever included in
the final reports.
In our overall review of the evaluation research
literature, we have found that elements of evaluation process do appear
in the growing number of anecdotal reports on non-utilization (Weiss,
1977).
But there appears to be little tradition of evaluators routinely
reporting on their own evaluation process.
Such lack of self-reporting
reduces the ability of decision makers to assess the quality of information
produced by the evaluation.
And, information on evaluation process could
only help to enhance evaluators' and program managers' awareness of
evaluation limitations and pitfalls, thus leading to improvement of
evaluation practice.
Evaluation outcomes in terms of decisions influenced by the evaluation
are rarely documented in the final report, due in part to the timing of
the final report with respect to decisions yet to be made.
Even after
decisions are made, it is often exceedingly difficult to estimate what
influence (if any) the evaluation had on the decisions.
Here, it seems,
we need new methods for follow-up attribution and documentation.
Thus, at the current time, we can only make a plea for more complete
self-reporting of evaluation inputs and process.
Self-reporting is open
to criticism on grounds of objectivity, particularly in the area of
evaluation process.
Yet even imperfect information in this area would
be more valuable than the present state of nearly no information.
Particularly for those second-order decision makers not directly
-21-
affiliated with the program being evaluated, it seems that at least
rudimentary knowledge of evaluation inputs and process would be
necessary to assess the possible relevance of the findings to them.
SUMMARY AND SUGGESTED RESEARCH
We have proposed that evaluation is a process producing information
that can be evaluated on the basis of its relevance to decision makers.
Paralleling program operation, an evaluation too can be characterized by
inputs, process, and outcomes.
We have provided a suggested list of
each, arguing that the only ultimate outcomes of an evaluation are
decisions influenced by the evaluation.
Thus, however difficult to measure,
the impact of an evaluation must be judged on the basis of resources
(re)allocated as a consequence of evaluation information provided.
of evaluations per se have several potential purposes:
Evaluations
to provide an
independent assessment to decision makers of the quality of information
contained in an evaluation; to provide guidance in selecting an evaluator;
to assimilate "research knowledge" from a number of separate but similar
programs; to provide a vehicle for examining the evaluation enterprise
itself.
Our concern for evaluation inputs, processes and outcomes
extends to recommendations for improved evaluation documentation in these
areas.
Evaluating an evaluation utilizing only inputs, or only process,
-22-
or only outcomes suffers drawbacks identical to similarly limited
evaluations of programs.
The need for comprehensiveness is apparent,
but the means for bringing it about in practice is not.
Thus, further work is needed in attempting to devise methods for
carrying out comprehensive evaluations of evaluations, within time and
budget constraints that are acceptable to potential decision makers.
Numerous important questions abound:
evaluations?
Who should conduct evaluations of
When is self-reporting of evaluation inputs and process
adequate?
How do we measure the effect of evaluation information on a
decision?
How do we historically recreate a decision maker's state of
(imperfect) knowledge at the time of-decision?
If a follow-on program
is instituted to measure an evaluation's ultimate impact, who should
fund it and who should do it?
Each of these questions, and more, should
provide fruitful areas for future research.
-23-
NOTES
1.
This definition of decision is taken from Howard (1966).
"'Irrevocable' does not imply 'for all time', but at least for the next
short time interval that an allocation of at least one resource has
been made.
That is, a decision is not a 'decision to.make a decision',
but rather the concrete action implied by the decision.
After any time
interval, a decision may be replaced by another decision, perhaps based
on updated information."
2.
The small Roman numeral in parentheses identifies the particular
point in Table 2.
3.
We use the term "(original) evaluation" to indicate the evaluation
being evaluated.
-24-
REFERENCES
BERNSTEIN, I.N. and H.E. FREEMAN (1975) Academic and Entrepreneurial
Research:
Srudies.
The consequences of Diversity in Federal Evaluation
New York:
Russell Sage Foundation.
CHAIKEN, J., L. CRANBILL, L. HOLLIDAY, D. JAQUETTE, M. LAWLESS, and
E. QUADE (1975) Criminal Justice Models:
Monica, CA:
An Overview.
Santa
Rand Corporation Technical Report R-1859-DOJ.
COOK, T.D. (1978) "Utilization, Knowledge-Building, and Institutionalization:
Three Criteria by Which Evaluation Research Can be Evaluated,"
(Introduction) in T.D. Cook (ed.) Evaluation Studies Review Annual,
Vol. 3.
Beverly Hills:
Sage.
COOK, T.D. and C.L. GRUDER (1978) "Metaevaluation Research." Evaluation
Quarterly 2(February):
5-51.
HOWARD, R.A. (1966) "Decision Analysis:
Applied Decision Theory," in
D.B. Hertz and J. Melese (eds.) Proceedings of the Fourth International
Conference on Operations Research. New York:
Wiley.
LARSON, R.C., V.M. BIER, E.H. KAPLAN, C. MATTINGLY, T.J. ECKELS,
N. REICHMAN, and L.S. BERLINER (1979) Interim Analysis of 200
Evaluations on Criminal Justice. (LEAA Grant 78NI-AX-0007).
Operations Research Center.
Institute of Technology.
Cambridge, MA:
Massachusetts
-25-
MINNESOTA SYSTEMS RESEARCH, INC. (1973) Proposed Review Systems for
the Evaluation of Technical Quality & R & D Projects in the
Department of Health Education and Welfare.
ROSSI, P.H. and S.R. WRIGHT (1977) "An Assessment-of Theory, Practice,
and Politics."
Evaluation Research l(February):
STUFFLEBEAM, D.L. (1974) Meta-Evaluation.
5-52.
Michigan University Evaluation
Center, Occasional Paper #3.
THOMPSON, M. (1975) Evaluation for Decisions in Social Programmes.
England:
Westmead,
Saxon House, D.C. Heath Ltd.
UNITED STATES GENERAL ACCOUNTING OFFICE (1978a) Status and Issues:
Program Evaluation. (PAD-78-83).
Washington, D.C.
(1978b) Assessing Social Program Impact Evaluations:
A Checklist
Approach (Exposure Draft) (PAD-79-2), Washington, D.C.
WEISS, C.H. (1977) Uising Social Research in Public Policy Making.
Lexington, MA:
Lexington Books.
Federal