EVAL 6000: Foundations
of Evaluation
Dr. Chris L. S. Coryn
Nick Saxton
Fall 2014
Agenda
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Introduction to evaluation
What are appropriate objects of evaluations and related subdisciplines of
evaluation?
Are evaluations enough to control quality, guide improvement, and protect
consumers?
Evaluation as a profession and its relationship to other professions
What is evaluation?
How good is good enough? How bad is intolerable? How are these questions
addressed?
What are performance standards? How should they be applied?
Why is it appropriate to consider multiple values?
Should evaluations be comparative, noncomparative, or both?
How should evaluations be used?
Why is it important to distinguish between informal evaluation and formal
evaluation?
How do service organizations meet requirements for public accountability?
What are the methods of formal evaluation?
What is the evaluation profession, and how strong is it?
What are the main historical milestones in the evaluation field’s development?
Today’s learning objectives
• The distinction between formal and informal evaluation
• The potential contributions and limitations of formal
evaluation
• Evaluation as a profession and its relationship to other
professions
• Conceptual and operational definitions of evaluation
• Key criteria for evaluating programs, including merit and
worth
• The roles of values clarification and setting standards in
reaching evaluative conclusions
• Four main uses of evaluation
• Distinctions between formative evaluation and summative
evaluation
• Distinctions between research and evaluation
• Historical milestones in the development of professional
evaluation
Introduction to
evaluation
• Evaluation is the single most
important and sophisticated cognitive
process in human reasoning and
logic
• Permeates all aspects of human
activity and has important
implications for maintaining and
improving services in all areas of
society
• Ultimately, a means for making
informed, credible, defensible, nonarbitrary decisions
What are appropriate
objects of evaluations
and related
subdisciplines of
evaluation?
• Objects of evaluation are referred to as
‘evaluands’
• In personnel evaluation the term
‘evaluee’ is used
• Subfields include, but are not limited to
(Scriven refers to these as the seven
‘P’s):
–
–
–
–
–
–
–
Product evaluation
Personnel evaluation
Policy evaluation
Performance evaluation
Proposal evaluation
Program evaluation
Portfolio evaluation
• Program evaluation typically receives
the greatest amount of attention
• Scope of application is nearly
infinite:
– Educational evaluation
– Evaluation of development and aid
– Consumer product evaluation
– Hospital evaluation
– Drug testing
– Human and social services evaluation
– Science policy evaluation
– Etc.
Are evaluations enough
to control quality, guide
improvement, and
protect consumers?
• For evaluations to make a positive
difference, policymakers, regulatory
bodies, service providers, and others
must obtain and act responsibly on
evaluation findings
• The production and appropriate use
of sound evaluation constitutes one
of the most vital contributors to
strong services and societal progress
Evaluation as a
profession and its
relationship to other
professions
• As a distinct profession, evaluation is
supportive of all other professions and
in turn is supported by many of them
– No profession could exist without
evaluation
• It now also has the ‘trappings’ of a
unique, specialized (trans)discipline
–
–
–
–
–
–
Graduate education programs
Scholarly journals
National associations
Professional conferences
Accredited standards
Guiding principles
What is evaluation?
e·val·u·a·tion
noun
“…the act or process of determining
the merit, worth, or significance of
something or the product of that
process”
— Scriven
e·val·u·a·tion
noun
“…evaluation is the systematic
assessment of the worth or merit of an
object”
— Joint Committee on Standards for
Educational Evaluation
• Evaluation’s root term, value,
denotes that evaluations essentially
involve making value judgments
• Accordingly, evaluations are not
‘value-free’
• They should reference pertinent
‘values’ (i.e., qualities or
characteristics considered important
in a given context)
A note on merit and worth
• Merit (synonymous with ‘quality)
concerns the intrinsic value of an
object, independent of context and
costs
• Worth (synonymous with ‘value’)
concerns the extrinsic value of an
object, considering both context and
costs
Needs and needs assessment
• Needs (something without which
satisfactory functioning cannot
occur) and needs assessments are
one source of ‘values’ used for
evaluation and determining the merit
and worth of evaluands
– Can be categorized as instrumental
needs and treatment needs
e·val·u·a·tion
noun
An extended, ‘values-oriented’
definition…
“…the systematic assessment of an
object’s merit, worth, probity,
feasibility, safety, significance, and/or
equity”
— Stufflebeam & Coryn
Evaluation should not be
confused with…
•
•
•
•
•
•
•
Assessment
Review
Analysis
Applied research
Audit
Diagnosis
And, many other terms that are
sometimes used synonymously
How good is good
enough? How bad is
intolerable? How are
these questions
addressed?
• This process requires clearly defined
‘decision rules’ (e.g., standard
setting) and using all relevant
‘criteria’ (i.e., the characteristics
defining a good or valuable object of
a certain type in a particular context)
• There are no ‘universal,’
‘standardized’ methods for
constructing evaluative conclusions
• ‘Beyond a reasonable doubt’ (as
used in jurisprudence) has often
been used as a ‘standard’ for warrant
supporting claims of merit or worth
What are performance
standards? How should
they be applied?
• Evaluation is sometimes
characterized as comparing a
performance to a standard
• This process involves setting one or
more cut scores against which the
performance of something is judged
– These methods are typically either
‘norm referenced’ or ‘criterion
referenced’
Why is it appropriate to
consider multiple
values?
• Very often different ‘stakeholder
groups’ have competing and
conflicting values
• One of the primary tasks of an
evaluator is to determine how to
adjudicate these conflicts, with
professional integrity and careful
attention to ethical and moral
implications
• Many, many methods have been put
forth for addressing this problem
Should evaluations be
comparative,
noncomparative, or
both
• In general, the question of whether
an evaluation should be comparative
or noncomparative depends on how
the evaluation will be used
– Comparative evaluations are required
when decisions between competing
alternatives are required
– Noncomparative evaluations are
required when a particular course of
action has already been determined
How should evaluations
be used?
• Formative evaluations for
improvement
• Summative evaluations for
accountability
• Evaluations to assist dissemination
efforts
• Evaluations to foster enlightenment
• Other types of uses include, but are
not limited to:
– Instrumental use
– Process use
Why is it important to
distinguish between
informal and formal
evaluation?
• Informal evaluation is what could be
considered ‘everyday evaluation’
– Often not systematic and biased in
known and unknown ways
• Formal evaluation should be
systematic and meet professional
standards
– Conducted by those having appropriate
competency
How do service
organizations meet
requirements for public
accountability?
• High quality efforts to improve
services and other types of efforts
can be accomplished through:
– Accreditation
– Internal evaluation
– External, independent evaluation
What is the evaluation
profession, and how
strong is it?
• Many technical books, specialized Web
sites, discussion groups, blogs, and
listserves, master’s and doctoral programs,
institutes and workshops on specialized
evaluation topics (for example, the
Evaluators’ Institute, client organizations
that fund a wide range of evaluations;
evaluation companies, guiding principles for
evaluators, and standards for program,
personnel, and student evaluations)
• These are substantial gains considering the
field’s status in 1964, when it had none of
these elements
• The evaluation field is still immature,
however, when compared with established
professions, such as medicine
What are the main
historical milestones in
the evaluation field’s
development?
• The pre-Tylerian period: Developments before 1930
– Initial use of standardized tests as measures of
educational quality
• The Tylerian period: 1930-1945
– Development of objectives-based methods for evaluating
educational programs
• The age of innocence: 1946-1957
– Very little evaluation requirements; most were internal,
self-evaluations
• The age of realism: 1958-1972
– Elementary and Secondary Education Act (ESEA) issued,
requiring evaluation of all federally-funded programs and
policies, Title I, etc.
• The age of professionalism: 1973-2004
– ‘Identity crisis’ among evaluators and there roles
• The age of global and multidisciplinary expansion:
2005-present
– Professionalization of the field
Encyclopedia Entries
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Bias
Causation
Checklists
Conceptual Use
Consumer
Effectiveness
Efficiency
Epistemology
Evaluation Use
Experimental Design
Experimental Society
Impartiality
Independence
Instrumental Use
Intended Uses
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Judgment
Merit
Modus Operandi
Ontology
Outcomes
Paradigm
Positivism
Postpositivism
Process use
Quantitative Weight and
Sum
Recommendations
Synthesis
Transdiscipline
Treatments
Worth