Review: Alternative Assessments I

Describe the two epistemologies in ch. 3 (o/s)

Compare the two principles for assigning value
(util/int-pl)

Identify pros/cons of the two evaluation
approaches we discussed last week
Alternative Approaches to
Evaluation II
Dr. Suzan Ayers
Western Michigan University
(courtesy of Dr. Mary Schutten)
Consumer-Oriented Approach

Typically a summative evaluation approach

This approach advocates consumer education
and independent reviews of products

Scriven’s contributions based on groundswell of
federally funded educational programs in 1960s

Differentiation between formative/summative eval.
Consumer-Oriented Checklist
(Scriven, 1974, p. 102)



Need
Market
Performance









True field trials [tests in a “real” setting]
True consumer tests [tests with real users]
Critical comparisons [comparative data]
Long term [effects over the long term]
Side effects [unintended outcomes]
Process [product use fits its descriptions]
Causation [experimental study]
Statistical significance [supports product effectiveness]
Educational significance
Cost effectiveness
 Extended support [in service training]
Producer’s efforts to meet these standards
improve product effectiveness
 Key Evaluation Checklist developed to evaluate
program evaluations
 Educational Products Information Exchange
(EPIE): Independent product-reviewer service
 Curriculum Materials Analysis System (CMAS)
checklist: Describe product, analyze rationale,
consider: antecedent conditions, content,
instructional theory & teaching strategies, form
overall judgments

Uses of Consumer-Oriented
Evaluation Approach

Typically used by gov’t. agencies and consumer
advocates (i.e., EPIE)

What does one need to know about a product
before deciding whether to adopt or install it?
Process information
 Content information
 Transportability information
 Effectiveness information

Consumer-Oriented Pros/Cons

Strengths: valuable info given to those who
don’t have time to study, advance consumers’
knowledge of appropriate criteria for selection
of programs/products

Weaknesses: can increase product cost,
stringent testing may “crimp” creativity, local
initiative lessened b/c of dependency on outside
consumer services
Consumer-Oriented Qs

What educational products do you use?

How are purchasing decisions made?

What criteria seem to most important in the
selection process?

What other criteria for selection does this
approach suggest to you?
Expertise-Oriented Approach

Depends primarily upon professional expertise
to judge an institution, program, product, or
activity

This is the first view that relies heavily on
subjective expertise as the key evaluation tool

Examples: doctoral exams, board reviews,
accreditation, reappointment/tenure reviews
etc…
Expertise-Oriented Types

Formal Review Systems (accreditation)


Informal Review systems (grad S committee)


Existing structure, no standards, infrequent
schedule, experts, status usually affected
Ad hoc panel review (journal reviews)


Existing structure, standards exist, set review
schedule, experts, status usually affected by results
Multiple opinions, status sometimes affected
Ad hoc individual review (consultant)

Status sometimes affected
Expertise-Oriented Pros/Cons

Strengths: those well-versed make decisions,
standards are set, encourage improvement
through self-study

Weaknesses: whose standards? (personal
bias), expertise credentials, can this approach
be used with issues of classroom life, texts, and
other evaluation objects or only with the bigger
institutional questions?
Expertise-Oriented Qs

What outsiders review your program or
organization?

How expert are they in your program’s context,
process, and outcomes?

What are characteristics of the most/least
helpful reviewers? (list brainstorms on board)
Participant-Oriented Approach

Heretofore, the human element was missing
from program evaluation

This approach involves all relevant interests in
the evaluation

This approach encourages support for
representation of marginalized, oppressed
and/or powerless parties
Participant-Oriented Characteristics

Depend in inductive reasoning [observe, discover,
understand]

Use multiple data sources [subjective, objective,
quant, qual]

Do not follow a standard plan [process evolves as
participants gain experience in the activity]

Record multiple rather than single realities [e.g.,
focus groups]
Participant-Oriented Examples

Stake’s Countenance Framework


Description and judgment
Responsive Evaluation
Addressing stakeholders’ concerns/issues
 Case studies describe participants’ behaviors


Naturalistic Evaluation
Extensive observations, interviews, documents and
unobtrusive measures serve as both data and
reporting techniques
 Credibility vs. internal validity (x-checking, triangulation)
 Applicability vs. external validity (thick descriptions)
 Auditability vs. reliability (consistency of results)
 Confirmability vs. objectivity (neutrality of evaluation)


Participatory Evaluation


Utilization-Focused Evaluation


Collaboration between evaluators & key organizational personnel for practical problem solving
Base all decisions on how everything will affect use
Empowerment Evaluation
Advocates for societies’ disenfranchised, voiceless
minorities
 Advantages: training, facilitation, advocacy,
illumination, liberation
 Unclear how this approach is a unique participantoriented approach
 Argued in evaluation that it is not even ‘evaluation’

Participant-Oriented Pros/Cons

Strengths: emphasizes human element, gain
new insights and theories, flexibility, attention to
contextual variables, encourages multiple data
collection methods, provides rich, persuasive
information, establishes dialogue with and
empowers quiet, powerless stakeholders

Weaknesses: too complex for practitioners
(more for theorists), political element,
subjective, “loose” evaluations, labor intensive
which limits number of cases studied, cost,
potential for evaluators to lose objectivity
Participant-Oriented Qs

What current program are you involved in that
could benefit from this type of evaluation?

Who are the stakeholders?
Alternative Approaches Summary
Five cautions about collective evaluation
conceptions presented so far
1) Writings in evaluation are not models/theories

Evaluation is a transdiscipline (not yet a distinct discipline)

“Theoretical” underpinnings in evaluation lack
important characteristics of most theories

Information shared is: sets of categories, lists of
things to think about, descriptions, etc.
2) “Discipleship” to a single ‘model’ is dangerous

Use of different approaches as heuristic tools, each
appropriate for the situation, recommended
3) Calls to consolidate evaluation approaches into
a single model are unwise
These efforts based in attempts to simplify
evaluation
 Approaches are based on widely divergent
philosophical assumptions
 Development of a single omnibus model would
prematurely close a divergent phase in the field
 Just because we can does not mean we should;
would evaluation be enriched by synthesizing the
multitude of approaches into a few guidelines?

4) The choice of an evaluation approach is not
empirically based

Single most important impediment to development
of more adequate theory and models in evaluation
5) Negative metaphors underlying some
approaches can cause negative side effects

Metaphors shared in ch. 3 are predicated on
negative assumptions in two categories:
 Tacitly assume something is wrong in system
being evaluated (short-sighted indictment)
 Based on assumptions that people will lie, evade
Qs or withhold information as a matter of course
Alternative Approaches’ Contributions
Approaches shared in ch. 4-8 influence evaluation
practices in important ways
 Help evaluators think diversely
 Present & provoke new ideas/techniques
 Serve as mental checklists of things to consider,
remember, or worry about
 Alternative approaches’ heuristic value is very high,
but their prescriptive value is less so
 Avoid mixing evaluation’s philosophically incompatible
‘oil/water’ approaches; eclectic use of alternative
approaches can be advantageous to high-quality
evaluation practices
Table 9.1
Exercise

Clearly identify your evaluand
Is it a program, policy, product, service, other?
 Who does it (or should it) serve?
 Who is in charge of it?


Find a partner and explain what you have written
Does it make sense?
 Does it match what you wrote?
 Does it avoid specifying criteria?
 Is it simple enough?
 Did you avoid commenting on the merits of the
evaluand?
