S519: Evaluation of
Information Systems
Analyzing data:
Merit
Ch8
Merit determination

Keep in mind of what the evaluation is.


Evaluation is the systematic determination of the
quality or value of something (Scriven, 1991).
Today we will discuss how to determine the
importance of evaluand components or
dimensions
Merit determination

It is the process of setting „standards“
(definitions of what performance should
constitute „satisfactory“, „good“, etc.) and
applying those standards to descriptive data
to draw explicitly evaluative conclusions
about performance on a particular dimension
or component.
Two steps


Step 1: defining what constitutes poor,
adequate, good, very good and excellent
performances on a particular dimension (or
component)
Step 2. using this definition to convert
empirical evidence into evaluative
conclusions (e.g. something explicit about
quality or value)
Determing merit
Decscriptive
facts about
performance
Quality or
value
determination
guide
Evaluative
conclusions
Using single quantitaive
measure

In a simple case, performance is measured
on a single quantitative dimension

The quality or value determination guide is just a
set of cutoffs



E.g., >90%=A/excellent, 80%-89%=B/good, 70%79%=C/adequate
E.g., satisfactory/unsatisfactory
Difficulty: where to put the cutoff score and how to
compare with different systems?
Exercise

School grading system




USA: A(>90%), B(80-90%), C(70-79%), D(60-69%),
F(<60%)
New Zealand: A(>80%), B(65-79%), C(50-64%),
D(35-49%), F(<35%).
Does that mean in New Zealand, it is easier to get A?
Can this way of grading ensure objectivity and
consistency of grading across courses? If yes, why? If
no, why not?
Using qualitative or multiple
measures



Using a single measure is not generally good
practice.
When using multiple measures, it is tricky on
how to merge them together to come out with
conclusion.
See table 8.1
Experience




Do not try to go for high precision
It is perfectly appropriate to give an answer that
still has a certain amount of fuzziness or
uncertainty associated with it.
Please do not oversell the precision of your work
Providing a well-supported broad-brush answer
to an important question is not a bad idea.
Rubric


Rubric is a tool that provides an evaluative
description of what performance or quality
„looks like“.
It has two levels:


Grading rubric is used to determine absolute
quality or value (e.g., Table8.2)
Ranking rubric is used to determine relative
quality or value
Rubric for absolute value

Rubric for “grading” is based on:





Discussion with domain expert
Discussion with upstream stakeholders
Existing rules (scope of duties) or literatures
Evaluand expectations (needs assessment)
Evaluation context (job market, current situation)
Sample grading rubric 1



Table 8.3
Using Table 8.3 to grade Table 8.1 – what is
grade for this master program? Why?
Form a pair and discuss your point
Sample grading rubric 2

Table 8.4 provides you a better grading rubric



Scope
Performance indicators
Scores (1-5)
Exercise

Develop a grading rubric for your evaluation
project

Take Table 8.4 as example

Please include scope, performance indicators and
ranking description
Using rubric for determining
„relative“ merit

Relative metric is important for experiment
that uses a control or comparison group.


Student scores are interpreted by comparison
with other similar schools
It simply tells us how the person or program
did relative to peers or competitors.
Using rubric for determining
„relevative“ merit
Score falls in
Grade assigned
Top 10%
A
Next 20%
B
Next 50%
C
Next 15%
D
Next 5%
F
“Grading on the curve”: instructors rank
students into different percentage
E.g.: GRE, SAT, GMAT, IQ
Significance

Statistical significance:


Any observed difference (or statistical
relationship) is unlikely to be due to chance
Practical significance:

Real impact on people‘s life


E.g., the difference has a noticeable and nontrivial
effect on functioning or performance
When determining the merit of a particular
outcome, we should take both significance
into consideration
Relative merit

Using comparison to determine relative merit

Benchmark process, outcome, and cost criteria
against what has been achieved elsewhere (e.g.
by other evaluands of a similar setting).
Benchmarking


It is a systematic study of one or more other
organizations‘ systems, processes, and outcomes to
identify ideas for improving organizational effectiveness.
It refers to a process of gathering comparison data about
what organizations in similar or related industries are
achieving (e.g. About process, outcomes, and costs).


Quantiative data
Qualitative data (observation of processes)
Some hints




Keep an open mind
Any help that can help you to make good
sense of the data
Balance of time and effort
Balance of time and level of details
Exercise

Grading Table 8.8 according to the rubric in
Table 8.7




What is your grading
Why is it?
How can you improve table 8.7?
Form a group to discuss
Exercise

Take the grantsmanworkshop, draw the
absolute and relative rubrics to grade this
training program


Absolute rubric (see Table 8.4)
Relative rubric (see Table 8.7)