Topic 7 : Part I
Measurement and Evaluation
第七讲： 第一部份
量度及評估
Paul Gandel
Syracuse University
Singapore Management University
Methods to Measure and
Display Outcomes
量度方法及展示结果
2
Basics of Measurement and Evaluation
量度及评估的基本因素
 Defining measurement
and evaluation
 为量度及评估下定义
 Frameworks for
measurement
 量度架构
 Outcomes-based
evaluation
 基于结果的评估
 Balance scorecard
approach
 平衡计分卡法
3
Why Measure?
为何要量度？
 Reference point
 参考点
 Where service was in the past
 过往的服务
 Current situation
 目前的景况
 Future trends
 未来的趋势
 Informs
 报告
 Decision makers
 决策者
 Managers
 经理
 Staff
 员工
 Sets Priorities
 设定优次序
4
Problem 问题
 Not too little data
 数据量不宜太少
 NOT THE RIGHT
DATA!
 不是正确的数据！
5
Measurement vs. Evaluation
量度與评估
 Measurement is collecting
the raw numbers
 量度是收集原始数目
 Evaluation is the
determination of the value
of the service
 评估是决定服务的价
值
 Evaluation requires a
perspective
 评估需要一套正确的
方法
 Creating indicators allows
for incorporating some
context
 设定指标时容许加入
某些内容
6
Measurement Framework
量度架构
From Joseph Mathews Measuring for Results
7
Measurement Framework量度架构
F
e
e
d
b
a
c
k
Resources
资源
Capability
能力
Input
Measures
投入量度
Process
Measures
过程量度
Quality
质量
Demand
Utilization
效用
Impact &
Effect
影响及效果
Output
Measures
產出量度
Outcomes
Value
结果
价值
8
Input Measures
投入量度
 Measure the
capacity/potential
 Easy to collect and report
 Examples:
 量度能力或潜力
 容易收集及报告
 范例:
 Income/Expenditures
 Staffing
 Collection (total, growth,
 收入/ 支出
recency)
 Infrastructure (cost, speed,
space, number of
workstations, etc.)
 User base statistics
近购买的)
 基础设施(成本，速度，
空间，工作间数目等)
 基于用户的统计
 人力资源
 馆藏 (总数，增长，最
9
Process/Efficiency Measures
过程 / 效率量度
 Are we doing things correctly?
 我们是否做得正确?
 Efficiency—how economical is
an activity? Cost per transaction
for reference service, document
delivery, etc.
 Productivity—focus on time to
complete task or activity. Time
to catalog an item, time to
process a transaction, etc.
 System Reliability—focus on
percent of uptime or availability
of the system
 效率 ─ 活动是否够经济？
每项咨询参考及文献送
递等服务的成本
 生产力 ─ 着眼于完成任
务或活动的时间。将一
项馆藏分类的时间，处
理一项服务的时间。
 系统可靠性 ─ 着眼于正
常运作时间的百分比或
系统可用性
10
Output Measures
产出量度
 Looks at the use of the
Library or a particular service
 考虑图书馆或某项专门
服务的用量
 Services. Counts and use
per capita statistics. Annual
circulation, attendance, etc
 服务。以人均统计方法
计算。书籍流通量，出
席率等。
 Quality—what was provided
and how was it provided.
Lead to measures of
satisfaction and value.
 质量 ─ 有何服务及如何
提供該服务。带出满意
度及价值的量度。
11
Satisfaction Measures
满意度量度
 Blanket question = useless
information
 一般的问题= 没用的信息
 Ask about specific components
of the experience
 询问某经验的特别部分
 Ask about how the user is doing
not just how the service is doing
 询问用户的感受如何，而
不单只是询问服务如何
 Ask about expectations
(LIBQUAL)
 了解期望(LIBQUAL)
 Ask about importance—was this
service important to the user?
 询问重要性 ─ 该项服务
对用户是否重要?
12
Priority and Performance
优先次序及表現
 What priority should the
library give to the following?
How important?
 图书馆如何为以下项目定
优先次序？ 有多重要？
 How well does the library
perform the following
services or functions?
 图书馆在以下服务或功能
表现如何？
 Usually use a 7 point Likert
scale
 通常用 Likert 的 7 级分表
13
Success Metrics: Purpose
成功规范：目的
• To establish clear
accountability and clarity of
purpose
– Both measured group and
the group for whom
measures are collected are
clearly stated
• To create focus and
alignment
• 设立清晰的责任和明确的
目的
– 明确定义被量度的小组及征
用资料的小组
• 重点设定及调整
– The measures collected are
in line with the mission, goals,
vision, and values of the
library
– 收集的数据应与图书馆的使
命、目标、愿景及价值配合
14
Partially from Matthews, J. (2004) Measuring for Results: The Dimensions of Public Library Effectiveness
Success Metrics: Purpose
成功规范：目的
• 容许持续改善
• To enable continuous
refinement
– Measures are reviewed and
changed to keep up with
changes in environment
• To provide rigorous
indicators
– 检讨及改变数据以配合环
境转变
• 提供严密的指标
– Measures are well-defined
using unambiguous language
and with as little bias as
possible
– 量度指标应用不含糊的语
言作严格的定义以尽量减
少偏见
15
Partially from Matthews, J. (2004) Measuring for Results: The Dimensions of Public Library Effectiveness
Balancing Success Metrics
平衡成功规范
• Absolute versus relative
measures
– Comparisons add context
(but are not always
appropriate)
• Objective versus subjective
– How much does background
of evaluator affect the
interpretation of the results?
• Outcomes over process
– Focus on outcomes, not on
the process
• Performance versus diagnostic
– Measure achievements and
discover problematic areas
• Peer institution benchmarks
versus best practice
• 絶对指标与相对指标
– Comparisons add context
(但并不总是适合)
• 客观与主观
– 评估人的背景对结果的演绎影
响有多深
• 结果与过程
– 侧重于结果而非过程
Partially from Matthews, J. (2004) Measuring for Results
• 表现及诊断法
– 量度成就及发现有问题的地方
• 同水平学院的标准与优良惯例
16
Types of Performance Indicators
表现指标
• Direct versus indirect indicators
– Measuring a specific activity vs.
measuring a surrogate for an
activity (looking at citations as
surrogate for measuring value of
materials)
• Lagging versus leading indicators
– Lagging = actual performance,
while leading = prediction of
something else
• Social versus economic
indicators
– Measures indicative of the impact
of the library on a community as
compared to the economic
importance (and cost) of the
library
• 直接与间接指标
– 量度一项特定的活动与量度一
项相关的活动 (考虑以引用次数
作为量度实际价值的相关指标)
• 落后的与领先的指标
– 落后 =实际表现,而领先 =其他
预测
• 社会指标与经济指标
– 此量度方法表示出与图书馆的
经济重要性(及成本)相比图书馆
对社会的影响
17
From Matthews, J. (2004) Measuring for Results: The Dimensions of Public Library Effectiveness
Balanced Scorecard Approach
平衡計分卡
•
•
•
•
•
User perspective
– How do users perceive the
library? Is that perspective
changing?
Internal perspective
– In which services should the
library aim to excel?
Information resources perspective
– What should be in the library
collections?
Innovation perspective
– How can the library continue to
develop new services?
Financial perspective
– How can the library
demonstrate the best value for
the money invested by the
institution in the library?
• 用户观感
– 用户对图书馆有何观感? 他们
的观感是否在转变?
• 内部认知
– 图书馆应着意突出哪几项服务？
• 资源概念
– 图书馆藏应包含甚么？
• 创新观念
– 图书馆如何继续新服务？
• 财务视角
– 图书馆如何展示学院对图书馆
的投资物有所值？
18
Based upon: Matthews, J. (2004) Measuring for Results: The Dimensions of Public Library Effectiveness
Balanced Scorecard 平衡计分卡
Library Perspective 以图书馆为例
Mission 使命
Financial Perspective
财务构面
Information
Resources
Perspective
资源构面
Customer Perspective
顾客构面
Process Perspective
流程构面
19
Organizational
Dashboards
组织仪表盘
 容易使用的表现展示工具
 User-friendly tool to display
performance
 有效及有意义的进度追踪
 Effective and meaningful way to
方法
track progress
 View results: efficiency,
 看得见的结果：效率、效
effectiveness, and impact
力及影响
 Should integrate into the
 将图书馆整体表现的某些
strategic plan a limited number of
关键指标纳入策略规划
critical indicators of overall library
 各部门可有自己的仪表
performance
盘
 Departments may have their
own dashboards Source: Butler, L. The Nonprofit Dashboard: A Tool for Tracking Progress (BoardSource, 2007)
20
Organizational Dashboards / 组织仪表盘
 What to measure?
 Outcomes
 Mission as spine
 Strategic initiatives
 Drivers of success
 Risk factors
 Service/resource
matrix
量度甚么？
结果
使命是支持因素
策略上的主动权
成功动力
冒险因素
服务/资源矩阵
 How to track and
display?
 Combine statistics
with visual
indicators
 Graphs and
如何追踪及展示
将统计数字与看
得见的指标结合
图形及图表
Source: Butler, L. The Nonprofit Dashboard: A Tool for Tracking Progress (BoardSource, 2007)
21
Relating the Mission to the Dashboard
将使命与仪表盘联系起来
Mission Statement /使命宣言
Purpose
目的
The library provides
information …
图书馆提供信息…
Audience
读者
for faculty and
students of the
university …
为大学学系及学生
Methods
方式
through electronic and
print resources …
透过电子及印刷资源
Outcomes
结果
to enable our clients to
meet their research
and learning needs
以令顾客满足教研需要
Performance Indicators / 表现指标
Number of searches
performed using eresources
电子资源的搜寻次数
Use of facilities for
Involvement of faculty
research, group and and students
individual study,
presenting their
determined through research results based
random counts
upon library resources
以随机数确定为研究， 学系及学生参与介绍
小组与个人学习而建 以图书馆资源为基础的
的设备的运用
研究成果
Adapted from: Butler, L. The Nonprofit Dashboard: A Tool for Tracking Progress (BoardSource, 2007)
Client responses re:
lessons learned and
quality of research or
learning enhanced by
library
顾客就习得经验及图书
馆对教研质量提升的回
馈反应
22
Final Step: The Reality Check!
最後的步驟：實際檢驗
23
Developing measures — Class Exercise
设定量度方法 ─ 小组研习
 Start with a topic area of the
library service
 以图书馆服务为一专题
开始
 Think about different parts of
that library services and
operations
 考虑图书馆服务的不同
部分及运作
 Objectives, Inputs, Output,
Efficiency, Quality, Immediate
Outcomes, Long-Term Outcomes
 Consider what could be
measured about each part of
the process, considering the
balances just discussed
 目标，投入，产出，效
率，质量，实时结果，
长远结果
 考虑过程的每一部分需
量度甚么，考虑之前讨
论过的平衡
From Matthews, J. (2004) Measuring for Results:
The Dimensions of Public Library Effectiveness
24