EVAL 6000: Foundations
of Evaluation
Dr. Chris L. S. Coryn
Nick Saxton
Fall 2014
Agenda
• Evaluation theory
• Standards for program evaluations
• Background for assessing evaluation
approaches
• Psuedoevaluations
• Brief activity
Evaluation theory
General features of evaluation
theory
• Unlike social science theories, which
are
– A set of interrelated constructs,
definitions, and propositions that
present a systematic view of
phenomena by specifying relations
among variables, with the purpose of
explaining and predicting phenomena
• Evaluation theories
– Describe and prescribe what evaluators do
or should do when conducting evaluations
– They specify such things as evaluation
purposes, users, and uses, who
participates in the evaluation process and
to what extent, general activities or
strategies, method choices, and roles and
responsibilities of the evaluator, among
others
– Largely, such theories are normative in
origin and have been derived from practice
rather than theories that are put into
practice
Theory’s role
• Has contributed to continuous
development of evaluation
– Many positive developments
– Many negative developments
• Allows for a large variety of diverse,
often conflicting ideologies
Research on evaluation
• Any purposeful, systematic, empirical
inquiry intended to test existing
knowledge, contribute to existing
knowledge, or generate new
knowledge related to some aspect of
evaluation processes or products, or
evaluation theories, methods, or
practices
Shadish, Cook, and Leviton’s
criteria for theories of program
evaluation
•
•
•
•
•
Social programming
Knowledge construction
Valuing
Use
Practice
Miller’s standards for empirical
examinations of evaluation
theories
•
•
•
•
•
Operational specificity
Range of application
Feasibility in practice
Discernable impact
Reproducibility
Standards for program
evaluations
The need for standards
• So that members of a profession
provide competent, ethical, and safe
services
• To ensure high-quality services and
protect the interests of the public
• So that evaluators deliver sound,
useful evaluation services
Existing standards
• Joint Committee’s The Program
Evaluation Standards
• The American Evaluation (AEA)
Association’s Guiding Principles for
Evaluators
• U.S. Government Accountability
Office’s (GAO) Government Auditing
Standards
Joint Committee standards
•
•
•
•
•
Utility
Feasibility
Propriety
Accuracy
Evaluation accountability
AEA guiding principles
•
•
•
•
•
Systematic inquiry
Competence
Integrity/honesty
Respect for people
Responsibilities for general and
public welfare
GAO (general) auditing
standards
•
•
•
•
Independence
Professional judgment
Competence
Quality control and assurance
Background for
assessing evaluation
approaches
Why study evaluation
approaches?
• To help evaluators and clients
identify, avoid, or expose misleading
or corrupt studies
• There is no one ‘right way’ of
conducting evaluation
– To understand the strengths and
weaknesses of evaluation approaches
and the circumstances under which each
is appropriately applied
Pseudoevaluations
Background
• Pseudoevaluations sometimes falsely
characterize constructive efforts—such
as providing evaluation training or
capacity building—as sound evaluation
• Some are conducted for corrupt, hidden
purposes
• Others are motivated by political or
profit motives
• In general, these approaches threaten
the credibility and integrity of
evaluation
Approach 1: Public relations
studies
• Use data to convince constituents
that a program is sound and effective
• Avoid gathering or releasing negative
findings
• Typically use surveys from biased
samples
• Only positive findings are
disseminated
Approach 2: Politically
controlled studies
• Conducted with the intent to
– Withhold the full set of findings from
right-to-know audiences
– Violate agreements to fully disclose
findings
– Selectively release findings
• Often, decisions are ‘predetermined’
and the evaluation is used as
justification
Approach 3: Pandering
evaluations
• Caters to predetermined evaluation
conclusions
• Places evaluator in a favored position
to conduct future evaluations
• Ultimately, both the client and
evaluator act together in producing
‘favorable’ findings
Approach 4: Evaluation by
pretext
• Client intentionally deceives evaluator
as to true intent of the evaluation
– Note: It is the evaluator’s responsibility for
investigating the ‘true’ intentions of the
client
• Often emphasizes negative aspects of a
program (e.g., so that information
could be used to ‘improve’ a program)
to support a client’s predetermined
decisions
Approach 5: Empowerment
under the guise of evaluation
• Cedes authority of evaluation to
groups likely not competent to
conduct sound evaluation
• Gives power and authority to such
groups to prepare evaluation reports
while claiming that an independent
evaluator prepared them
• Essentially, the approach is directed
toward helping groups gain ‘power’
Approach 6: Customer
feedback evaluation
• Supposedly independent ratings and
reviews of products and services
provided by consumers
• Whether products and services were
actually purchased by those
providing ratings and reviews is often
unknown
• Often the ‘vendor’ manufactures
ratings and reviews to increase sales
Activity
Activity
• Discuss the rationale that supports
the following statement: “Evaluators
should not lend their name and
endorsement to evaluations
presented by their clients that
misrepresent the full set of relevant
findings”
Encyclopedia Entries
•
•
•
•
•
•
•
•
•
•
•
Bias
Causation
Checklists
Chelimsky, Eleanor
Conflict of Interest
Countenance Model of
Evaluation
Critical Theory
Evaluation
Effectiveness
Efficiency
Empiricism
Independence
• Evaluability
Assessment
• Evaluation Use
• Fournier, Deborah
• Positivism
• Relativism
• Responsive evaluation
• Stake, Robert
• Thick Description
• Utilization of
Evaluation
• Weiss, Carol
• Wholey, Joseph