Peer Review Can Make Metrics Accountable
同行评议可以赋予指标以责任
Kelli Barr
凯利·巴尔
Department of Philosophy and Religion Studies
Center for the Study of Interdisciplinarity
University of North Texas
哲学与宗教学系
跨学科研究中心
北德克萨斯大学
Translation by ZHANG Wei
The Question at Hand
目前的问题
Are metrics better tools for evaluating research than peer review?
在评估研究方面，指标是比同行评议更好的工具吗？
Increasingly, academics and administrators are answering with an unqualified “yes
专家学者和管理者越来越多地倾向于给出肯定的回答，但这是有问题的。
– H-index, g-index, webometrics
H指数、g指数、网络计量学
– US News and World Report, Shanghai Rankings
美国新闻与世界报道，上海（交通大学）大学排名
– National research evaluations
国家研究评估
– Hiring, promotion, and tenure decisions
人才招聘、晋升和终身教职决定
‘Peer Review Can Make Metrics Accountable’
International Workshop on Peer Review
Dalian University of Technology
Monday, May 21 – Wednesday, May 23, 2012
Peer Review
同行评议
Subjective
主观性
Contingent
差异性
Non-transparent
不透明性
Unaccountable
不可解释性
Costly
高成本性
‘Peer Review Can Make Metrics Accountable’
International Workshop on Peer Review
Dalian University of Technology
Monday, May 21 – Wednesday, May 23, 2012
Bibliometrics
文献计量学
Objective
客观性
Replicable
可重复性
Transparent
透明性
Accountable
可解释性
Low(er) Cost
低成本性
Defining the dichotomy using these terms creates a distinction without
much of a difference
用这些术语定义的二分法给出了一个其实并没有太多差别的区分
– Bibliometrics are ultimately based on peer review processes
文献计量方法最终要建立在同行评议的程序之上
All metrics instantiate specific values that affect the outcome of the
measurements
所有的指标都是某种具体价值标准的体现，都会对评估结果产生
影响
‘Peer Review Can Make Metrics Accountable’
International Workshop on Peer Review
Dalian University of Technology
Monday, May 21 – Wednesday, May 23, 2012
Forbes – concerned
with economic,
financial, commercial,
and business
outcomes
福布斯—
Academic
ranking of
涉及经济、金融、商
业、贸易的结果
world
universities –
concerned with
bibliometric
(research) outcomes
世界大学的学术排
名—
涉及文献计量学的
（研究）结果
http://chronicle.com/article/30-Ways-to-Rate-a-College/124160/
Journal impact factor, calculated using Web of Knowledge
期刊影响因子——基于Web of Knowledge数据库
Sciences
自然科学
Social Sciences
社会科学
Philosophy
哲学
Nature
《自然》
Cell
《细胞》
Science
《科学》
J. of Economic Lit.
《经济文献杂志》
Political Analysis
《政治分析》
Ann. Rev. of Sociology
《社会学年评》
Am. J. of Bioethics
《美国生物伦理学杂志》
Phil. and Public Affairs
《哲学与公共事务》
Environmental Values
《环境价值》
‘Peer Review Can Make Metrics Accountable’
International Workshop on Peer Review
Dalian University of Technology
Monday, May 21 – Wednesday, May 23, 2012
34.48
31.152
29.747
6.919
3. 756
3.702
4.000
1.957
1.250
Do social science and
philosophy journals
really have that much
less “impact”?
社会科学和哲学期刊
真的是只有这么点儿
影响力吗？
Bibliometrics assume that all research conforms to the model of scientific research
文献计量学假定所有的研究符合自然科学的研究模式。
– Cumulative
累积的
– Disciplinary
分科的
Bibliometrics highlight certain kinds of research:
文献计量学突出强调某些种类的研究：
– Journal publications
期刊发表
– Quantitative
定量的
– Methodologically focused
关注方法论的
– Review papers
评议论文
‘Peer Review Can Make Metrics Accountable’
International Workshop on Peer Review
Dalian University of Technology
Monday, May 21 – Wednesday, May 23, 2012
Implications 启示
Bias toward disciplinary, status quo research
对学科的偏见，现状分析
Reinforcing the status quo is not morally neutral
对现状的加强在道德上不是中立的
Appeals to the supposed objectivity, transparency, etc. of metrics are used as evidence
of their normative superiority to peer review
对指标的所谓的客观性、透明性等属性的诉求被用来作为它比同行评议具有规范的优
越性的证据。
‘Peer Review Can Make Metrics Accountable’
International Workshop on Peer Review
Dalian University of Technology
Monday, May 21 – Wednesday, May 23, 2012
Reflexive evaluation of activity
toward goals
对面向目标的行动的自反性评价
Phronesis
实践智慧
Episteme
知识
‘Peer Review Can Make Metrics Accountable’
International Workshop on Peer Review
Dalian University of Technology
Monday, May 21 – Wednesday, May 23, 2012
Techne
技艺
Rational
calculation
of goals
对目标的
理性计算
Accountability is about rational justification
责任性涉及合理的辩护
• Definitions vary widely with context:
定义会因为语境的不同而产生很大的变化：
– Legal liability
• Common
denominator: evaluating whether or not current tactics meet
overall
strategic goals
法律责任
– Return on investment
共同点：评估当前的策略是否能够达到总体的战略目标
对投资的回报
• Accountability
is more than just rule following
– Academic rigor
责任性不仅仅是遵守规则
学术的严格性
‘Peer Review Can Make Metrics Accountable’
International Workshop on Peer Review
Dalian University of Technology
Monday, May 21 – Wednesday, May 23, 2012
Conclusions:
结论
Neither metrics nor peer review are an unqualified good
指标和同行评议都不是最好的方法
The question of the meaning of accountability is also the question of the meaning of
responsibility
责任性的意义问题同时也是一个义务的意义问题
If science funding is to be a goal-directed activity, then evaluations of the science
funded cannot normatively privilege values such as objectivity for their own sake.
如果科学资助是一个以目标为导向的活动，那么对受资助的科学研究的评估就不
能给予诸如为其自身着想的客观性这样的价值以特权。
‘Peer Review Can Make Metrics Accountable’
International Workshop on Peer Review
Dalian University of Technology
Monday, May 21 – Wednesday, May 23, 2012