Understanding Evaluation
Definition of Evaluation
Systematic determination of the quality or value
of something (Scriven, 1991)
 What can we evaluate?







Projects, programs, or organizations
Personnel or performance
Policies or strategies
Products or services
Proposals, contract bids, or job application
Almost everything in our daily life because before you
make decision, you do the evaluation first.
Comparison is a kind of evaluation.
Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage
Terminology (Davison, Glossary)

Evaluand
That which is being evaluated (e.g. Program, policy,
project, product, service, organization, almost
everything)
 In personnel evaluation the term is evaluee

S519
Issues of Evaluation

Evaluation is for



Find areas for improvement
Generate an assessment of overall quality
Answer the question of „Merit“ or „Worth“ (Scriven,
1991)


Merit is the „intrinsic“ value of something = „quality“
Worth is the value of something to an individual, an
organization, an institution – contextualized merit -- = „value“
S519
Choosing the right group

Accountability evaluation

It is important to conduct an independent
evaluation

i.e. Nobody on the evaluation team should have a
significant vested interest in whether the results are
good or bad
S519
The steps involved (D-p4)
Step1: understanding the basic about
evaluation (ch1)
 Step2: defining the main purposes of the
evaluation and the „big picture“ questions that
need answers (ch2)
 Step3: Identifying the evaluative criteria (ch3)
 Step4: Organizing the list of criteria and
choosing sources of evidence (collecting data)
(ch4)

S519
The steps involved (D-p4)

Step5: analyzing data
dealing with the causation/correlation issue (which
cause what, why), to avoid „subjectivity“ (ch5+6)
 importance weighting: weight the results (ch7)




Meric determination: how well your evaluand has done on
the criteria (good? Unacceptable?) (ch8)
Synthesis methodology: systematic methods for
condensing evaluation findings (ch9)
Staticistical analysis: Salkind (2007)
S519
The steps involved (D-p4)

Step6: result


Putting it all together: fitting the pieces into the
KEC framework (ch10)
Step7: feedback

Meta-evaluation: how to figure out whether your
evlauation is any good (ch11)
S519
The Key Evaluation Checklist
(Davidson, 2005, p. 6-7)
I. Executive
Summary
II. Preface
1. Background 2. Descriptions
& Context
& Definitions
6. Process
Evaluation
7. Outcome
Evaluation
III. Methodology
3.
Consumers
4.
Resources
8 & 9. Comparative
Cost-Effectiveness
5. Values
10.
Exportability
11. Overall Significance
12.
13. Responsibilities
Recommendations
& Explanations
S519
14.
Reporting
&
Follow-up
15. Meta-evaluation
Step 1: Understand the basic of
evaluation


Identify the evaluand
Background and context of evaluand


Descriptions and definitions


Why did this program or product come into existence in the first
place?
Describe the evaluand in enough detail so that virtually anyone can
understand what it is and what it does
How: collect background information, pay a firsthand visit or
literature review
S519
Are you ready for your first evaluation
project?

Some tips before you start
Make sure that your evaluand is not difficult to
access (geolocation, inanimate objects)
 Make your evaluand a clearly defined group (avoid
abstract and complex system)
 Avoid political ramification (assess your boss pet
project, university administration)
 To avoid your invovlement in the evaluand (to
assess a class which you are teaching, etc.)

S519
Previous Projects


Metadata discussion group, Brown Bag discussion group,
Twitter, SLIS website, Information Visulization Lab website,
Media and Reserve Services in Wells Library, IU CAT, SoE
website, Chemistry library website
How about something else related to real-world problems



For social good (http://dssg.io/projects/)
Industry and Government talks at KDD214
(http://www.kdd.org/kdd2014/)
Maybe this will inspire you with a great project to work on
Standards for Evaluation (Patton, 1997, p. 17)

Utility


Feasibility


To ensure that an evaluation will be realistic, prudent, diplomatic,
and frugal
Propriety


To ensure that an evaluation will serve the practical information
needs of intended users
To ensure that an evaluation will be conducted legally, ethically,
and with due regard for the welfare of those involved in the
evaluation, as well as those affected by its results (IRB:
http://research.iu.edu/rschcomp/revlocation.html)
Accuracy

To ensure that an evaluation will reveal and convey technically
adequate information about the features
S519
http://www.wmich.edu/evalctr/jc/
Guiding Principles for Evaluators (American
Evaluation Association, 1995)
Systematic inquiry
 Competence
 Integrity/honesty
 Respect for people
 Responsibilities for general and public welfare

S519
Step1: Output report

Output: one or two page overview of the
evaluand and findings
What is your evaluand
 Background and context of your evaluand
 Description of your evaluand


Try to be as detail as possible
S519
Step2: Defining the Purpose of the
Evaluation (D-Ch2)
Who asked for this evaluation and why?
 What are the main evaluation questions?
 Who are the main audience?

S519
Evaluation purposes

A. what is (are) the main purpose(s) of the
evaluation?

To determine the overall quality or value of something
(summative evaluation, absolute merit)


i.e. Decision making, funding allocation decision, benchmarking
products, using a tool, etc.
To find areas for improvement (formative evaluation, relative
merit)


To help a new „thing“ to start
To improve the existing „thing“
S519
Big picture questions

Big picture questions:

B. What is (are) the big picture question(s) for
which we need answers?

Absolute merit


Do we want to invest this project?
relative merit


How does this project compare with the other options?
Ranking
S519
Step2: Output report

Your step2 output report should answer the following
questions:

Define the evaluation purpose




Do you need to demonstrate to someone (yourself) the overall quality of
something?
Or Do you need to find a file for improvement?
Or do you do both?
Once you answer above questions, figure out what are your big picture
questions:


Is your evaluation related to the absolute merit of your evaluand?
Or the relative merit of your evaluand
S519