CITATION INDICES
Philip Purnell
March 2010
PART I
RESEARCH EVALUATION
HOW DO WE EVALUATE RESEARCH?
• Research grants
– Number and value
• Prestigious awards
– Nobel Prizes
• Patents
– Demonstrating innovative research
• Faculty
– Number of post-graduate researchers
• Citation analysis
– Publication and citation counts
– Normalised by benchmarks
• Peer Evaluation
– Expensive, time consuming and subjective
DO WE NEED MORE METRICS?
A SIMPLE COUNT IS GOOD ENOUGH (1962)
CHAMPIONS LEAGUE 2009
METRICS ARE HERE TO STAY !
Barcelona
Team Statistics
Manchester United
2
1
7
4
1
4
7
1
1
0
87
18
72.2
52.7
Goals
1st Half Goals
Shots on Target
Shots off Target
Blocked Shots
Corners
Fouls
Offsides
Yellow Cards
Red Cards
Passing Success
Tackles
Tackles Success
Possession
0
0
2
7
2
7
10
5
3
0
82.9
21
85.7
47.3
55.3
Territorial Advantage
44.7
5
A BRIEF HISTORY OF THE CITATION INDEX
• Concept first developed by Dr Eugene Garfield
– Science, 1955
• The Science Citation Index (1963)
–
–
–
–
SCI print (1960’s)
On-line with SciSearch in the 1970’s
CD-ROM in the 1980’s
Web interface (1997) Web of Science
• Content enhanced:
– Social Sciences Citation Index (SSCI)
– Arts & Humanities Citation Index (AHCI)
• The Citation Index
– Primarily developed for purposes of information retrieval
– Development of electronic media and powerful searching tools have
increased its use and popularity for purposes of Research Evaluation
WEB OF SCIENCE
JOURNAL SELECTION POLICY
• Why do we select journals?
THOMSON REUTERS
JOURNAL CITATION REPORTS
120
40% of the journals:
% of database
100
80
• 80% of the publications
60
• 92% of cited papers
40
4% of the journals:
20
• 30% of the publications
0
0
1000
2000
3000
4000
# of journals
Articles
Citations
5000
6000
• 51% of cited papers
COMPARE LIKE WITH LIKE
WHAT IS THE VALUE OF A CITATION?
• Why do people cite?
–
Pay homage / give credit to pioneer
–
Identifying a methodology
–
Provide background reading
–
Quotations
–
Authenticating data, reproducing work etc
–
Corrections
–
Criticizing/Disclaiming someone's work/opinions
• Citations are an indicator of an article’s impact and usefulness to the
research community; they are the mode by which peers
acknowledge each other’s research.
• The value of a citation is only as important as its source.
–
Clearly a citation from a prestigious peer review journal has more value than a
citation from non-scholarly material.
–
How can you be sure that the citing source is reputable?
“When to Cite”, E. Garfield, Library Quarterly, v66, p449-458, 1996
WEB OF SCIENCE
JOURNAL SELECTION POLICY
• Approx. 2000 journals evaluated annually
– 10-12% accepted
• Thomson Reuters editors
– Information professionals
– Librarians
– Experts in the literature of their subject area
Web of Science
Journals under evaluation
Journal ‘quality’
THOMSON REUTERS
JOURNAL SELECTION POLICY
• Publishing Standards
– Peer review, Editorial conventions
• Editorial content
– Addition to knowledge in specific subject field
• Diversity
– International, regional influence of authors, editors, advisors
• Citation analysis
– Editors and authors’ prior work
Regional Journal Content Expansion 2007-2009
Contribution of Each Region to Subject Areas
Region
AP
EU
LA
MA
NA
Total
A&H AB&ES
13
55
150
88
29
55
13
23
8
1
213
222
Regions:
AP : Asia Pacific
EU : European Union
LA : Latin America
MA : Middle East/Africa
NA : North America
CM
EC&T
105
45
161
81
33
13
32
10
13
2
344
151
LS PC&ES S&BS
47
41
89
50
116
200
14
15
42
15
7
31
1
0
3
127
179
365
Total
395
846
201
131
28
1601
Subject Areas:
A&H :
Arts &Humanities
AB&ES : AgBio & Environmental Sci
CM :
Clinical Medicine
EC&T : Engineering Computing & Technology
LS :
Life Science
PC&ES : PhysChem & Earth Science
S&BS : Social & Behavioral Science
12
GROWTH IN COVERAGE OF
REGIONAL AND CZECH JOURNALS
Web of
Science
2005
2010
Growth
Variation
Total
Journals
Indexed
8,834
11,423
2,589
+ 29%
Regional
Journals
Indexed
1,716
3,605
1,889
+ 110%
Czech
Journals
Indexed
30
56
26
+ 87%
13
GROWTH IN COVERAGE OF CZECH
JOURNALS IN WEB OF SCIENCE
Web of
Science
2005
2010
Growth
Variation
Science
Citation
Index –
Expanded
23
36
13
+ 57%
Social
Sciences
Citation
Index
4
11*
7
+ 175%
Arts &
Humanities
Citation
Index
3
12**
9
+ 300%
*1 title is covered in both SCIE and SSCI
**2 titles are covered in both SSCI and A&HCI
14
GLOBAL RESEARCH REPRESENTATION
WEB OF SCIENCE COVERAGE
Region
# Journals from Region in Web of Science
Europe 5,573
49%
North America 4,251
38%
Asia-Pacific 965
9%
Latin America 272
2%
Middle East/Africa 200
1%
Language
# Journals in Web of Science
English
9114
81%
Other
2147
19%
GOVERNMENTS AND INSTITUTIONS
USING TR DATA FOR EVALUATION (INCL.)
• Czech Republic: Czech Academy of Sciences; Government
• France: Min. de la Recherche, OST - Paris, CNRS
• Germany: Max Planck Society, several gov’t labs, DKFZ, MDCUS:
National Institutes of Health
• United Kingdom: King’s College London; HEFCE
• European Union: EC’s DGXII(Research Directorate)
• US: NSF: biennial Science & Engineering Indicators report (since 1974)
• Canada: NSERC, FRSQ (Quebec), Alberta Research Council
• Australian Academy of Science, gov’t lab CSIRO
• Japan: Ministry of Education, Ministry of Economy, Trade & Industry
• People’s Republic of China: Chinese Academy of Science
16
EVALUATING COUNTRIES
CHANGING NATURE OF CZECH
RESEARCH
Czech researchers
averaged just over
4,000 papers a year in
the 70s – 90s.
Now over 10,000
papers per year
18
CHANGING NATURE OF CZECH RESEARCH
INTERNATIONAL COLLABORATION
Between 1970 and 1999, 14%
of Czech papers had at least
one international co-author
19
CHANGING NATURE OF CZECH RESEARCH
INTERNATIONAL COLLABORATION
In the last decade 2000 – 2010,
this figure has risen to 63%
20
SCIENTIFIC RESEARCH PRODUCTIVITY
IN CENTRAL EUROPE
Thomson Reuters InCites
21
SCIENTIFIC RESEARCH IMPACT
IN CENTRAL EUROPE
Thomson Reuters InCites
22
CZECH RESEARCH
BY OUTPUT AND IMPACT
23
Ranking by percentage share of Thomson Reuters (ISI) indexed papers 2004 - 2008
EVALUATING INSTITUTIONS (1)
EXTERNAL COMPARISONS
EVALUATING INSTITUTIONS
Number of citations
to North American
scientific papers
Source: Thomson Reuters
North America University Science Indicators
PUBLICATION AND CITATIONS
TOP CZECH ORGANISATIONS
26
COMPARE YOUR INSTITUTION’S
PERFORMANCE AGAINST GLOBAL PEERS
Thomson Reuters InCites
27
INSTITUTIONAL COMPARISONS
COMPARING IMPACT IN BIOLOGY?
Cambridge
Oxford
Stirling
Thomson Reuters InCites
29
INSTITUTIONAL COMPARISONS
IN SPECIFIC FIELDS
Thomson Reuters InCites
30
INSTITUTIONAL COMPARISONS
BIOLOGY PERFORMANCE
Thomson Reuters InCites
31
EVALUATING INSTITUTIONS (2)
INTERNAL ANALYSIS
WHAT IS OUR RESEARCH OUTPUT?
33
BENCHMARKING YOUR PAPERS
AGAINST GLOBAL AVERAGES
Articles published
in ‘Blood’ from
2004 have been
cited 34.30 times
Hematology articles from
this year have been cited
18.83 times
This article is in the 12.92nd
percentile in its field by
citations
34
WHAT IS OUR RESEARCH OUTPUT?
35
WHICH ARE OUR
CENTERS OF EXCELLENCE?
Physical
Chemistry
41% below
average
Computer Science
220%
above
average
Thomson Reuters InCites
36
WITH WHOM DOES OUR FACULTY
COLLABORATE?
37
WHICH OF THOSE COLLABORATIONS ARE
THE MOST VALUABLE?
38
INCITES
CITING ARTICLES LISTING
Thomson Reuters InCites
CITING PAPERS
IN THE FIELD OF PHYSICS
Thomson Reuters InCites
40
EVALUATING INDIVIDUALS
FRANTISEK VYSKOCIL
PROF. AND D.Sc CHARLES UNIV, PRAGUE
• Described: Potassium movement in the brain during migraine
• Member: Physiological Society of Cambridge and London
• Founding Member: Learned Society of the Czech Republic
• Awards:
– Prize of the Czech Academy of Sciences
– Lifelong Contribution Award of the Czech Academy of Sciences
– Purkynje Medal of the Czech Academy of Sciences
• First violinist: Quartet of the Herold Chamber Music Club
• Author: >60 popular science articles in "Vesmír“
42
FRANTISEK VYSKOCIL
WEB OF SCIENCE PUBLICATIONS
247 Publications
43
FRANTISEK VYSKOCIL
GLOBAL INFLUENCE
Citations by country
Citations: 2.375
Cites per paper: 9,73
H-Index = 30
Citations by journal
WHO ARE OUR
MOST PRODUCTIVE AUTHORS?
Thomson Reuters InCites
45
WHO ARE OUR
MOST INFLUENTIAL RESEARCHERS?
Thomson Reuters InCites
46
WHICH AUTHORS HAVE
MOST IMPACT?
Thomson Reuters InCites
47
WHICH AUTHORS’ PAPERS HAVE
PERFORMED BEST IN THEIR FIELD?
Thomson Reuters InCites
48
HOW CAN WE COMPARE RESEARCHERS?
Author A: 60 papers
Author B: 117 papers
Thomson Reuters InCites
49
EVALUATING JOURNALS
EFFICIENCY
JOURNAL IMPACT FACTOR
All
Previous
Years
2005
2006
2007
2008
2009
2008 Impact
Factor
Citations
Source paper – published in 2008
Cited reference – published in 2006 or 2007
CALCULATING 2008 IMPACT FACTOR
COLLECTION OF CZECHOSLOVAK CHEMICAL COMMUNICATIONS
Citations in 2008
To items published in 2007 =
78
To items published in 2006 = 100
Sum
= 178
178
= 0,784
Number of items
Published in 2007 = 118
Published in 2006 = 109
Sum
=
227
227
JOURNAL IMPACT FACTOR
CZECH JOURNALS
53
CITATION BEHAVIOUR VARIES
BETWEEN SUBJECT CATEGORIES
JOURNAL 5-YEAR IMPACT FACTOR
CZECH JOURNALS
55
HOW DO CZECH INSTITUTIONS
EVALUATE JOURNALS FOR LIBRARIES?
• Faculty head request
• Publisher packages
• Budget constraints
• Library recommendation
56
IN WHICH JOURNALS DO OUR
BIOLOGISTS PUBLISH?
Thomson Reuters InCites
57
AND THE GLOBAL COMMUNITY
INFLUENCED BY YOUR RESEARCH?
Thomson Reuters InCites
58
SUMMARY:
THE DATA SOURCE
• Thomson Reuters solutions utilize the data from the Web of
Science to provide reliable and consistent evaluation tools
• It is essential to have a consistent, authoritative and clearly
defined body of data to create meaningful and dependable
statistics. This is why we use Web of Science:
– The gold standard citation resource, used by over 4,000 institutions in
more than 90 countries
– A comprehensive resource covering over 11,000 journals and
thousands of conference proceedings in the fields of Science, Social
Science, Arts and Humanities
– Unmatched retrospective depth of citation data (up to 110 years!)
– Selected content using unbiased, time tested and stringent journal
selection process
– Web of Science is also the source of the Impact Factor the most
widely accepted indicator of journal performance
PART II
THE DATA
THE DATA
• Web of Science
– Conference Proceedings Citation Index
– Century of Social Sciences
• ISI Web of Knowledge - The Citation Universe
– Web of Knowledge 5 upgrade
– Biosis Citation Index
– Chinese Science Citation Database
• Research Analytics
61
THE DATA
• Web of Science
– Conference Proceedings Citation Index
– Century of Social Sciences
• ISI Web of Knowledge - The Citation Universe
– Web of Knowledge 5 upgrade
– Biosis Citation Index
– Chinese Science Citation Database
• Research Analytics
62
CPCI – ON WEB OF SCIENCE
Search by
- Conference title
- Location
- Date
- Sponsor
63
WEB OF SCIENCE
CONFERENCE PROCEEDINGS CITATION INDEX
• Two Editions:
- Science & Technology
- Social Sciences & Humanities
• Approx 5.5 million records
• Updated Weekly
• Over 385,000 records per year
• More than 120,000 conferences covered in total
• Approximately 12,000 conferences covered annually
• Coverage from 1990, cited references from 1999
• Comprehensive, multidisciplinary and international coverage from
conferences proceedings published in books, journals, reports,
series, and preprints
• Details of each conference are indexed and searchable, data
includes: conference title, sponsor, location, description and date
A combined set of results
from both Conference
Proceedings and Journal
literature.
Refine options are a useful tool to
narrow a search. For example narrow
by subject classifications or document
type to focus on the most relevant
materials
This Citation Report is based on the
entire publication portfolio of 2008
Nobel laureate Roger Y. Tsien.
Many records are from Conference
Proceedings, without this content the
portfolio is incomplete.
Full Web of Science functionality
with Conference Proceedings
content
Details of both
conference and
source
publication
Classified by
Document Type
Source item is assigned to
standard subject
classifications
THE DATA
• Web of Science
– Conference Proceedings Citation Index
– Century of Social Sciences
• ISI Web of Knowledge - The Citation Universe
– Web of Knowledge 5 upgrade
– Biosis Citation Index
– Chinese Science Citation Database
• Research Analytics
69
WEB OF SCIENCE
DEPTH OF CITATION INDICES
1900
Century of Science
1900
Century of Social Sciences
1945
Science Citation Index
SSCI
1956
1975
56 years of added coverage
in the Social Sciences
A&HCI
1990
CPCI
>300 journals added that
focus on the first half of
last century
CENTURY OF SOCIAL SCIENCES
BREAKDOWN BY FIELD
CoSS distribution by field
Title Count
ANTHROPOLOGY
24
COMMUNICATION
18
ECONOMICS & MANAGEMENT
47
EDUCATION
31
GEOGRAPHY
12
HISTORY & PHILOSOPHY OF THE SOCIAL SCIENCES
1
LAW
30
POLITICAL SCIENCE
10
PSYCHIATRY
21
PSYCHOLOGY
65
PUBLIC HEALTH
20
SOCIAL ISSUES & SOCIOLOGY
29
Total
308
COVERAGE IN THE FIELD OF GEOGRAPHY
• There are more than 40,000 records in the field of Geography
between the years of 1900 and 1955
• The key journals from this period have been evaluated, sourced and
indexed to provide a unique resource:
–
–
–
–
–
Geographical Journal
Scottish Geographical Magazine
Geography
Geographical Review
Annales de Geographie
–
–
–
Petermanns Geographische Mitteilungen
Economic Geography
Annals of the Association of
American Geographers
• The works of many ground breaking researchers from around the
world have been included, such as:
–
C. Warren Thornthwaite
–
Harlan H. Barrows
–
John R. Borchert
CASE STUDY:
Charles Warren Thornthwaite
• C. Warren Thornthwaite was a Geographer
and Climatologist best known for devising
a Climate Classification System proposed
in his landmark paper in 1948 which also discusses the
importance of evapotranspiration on climate.
• Thornthwaite published 28 papers between 1931 and 1965,
mostly in the fields of Geography and Geochemistry &
Geophysics.
• His works have gone on to be highly influential to researchers
in many fields and in recent years have been particularly
utilized by researchers involved in climate change and water
management
The works of C. Warren Thornthwaite
Thornthwaite
produced 28 highly
influential works in the
Social Sciences which
have been
consistently cited
throughout their
history
THE DATA
• Web of Science
– Conference Proceedings Citation Index
– Century of Social Sciences
• ISI Web of Knowledge - The Citation Universe
– Web of Knowledge 5 upgrade
– Biosis Citation Index
– Chinese Science Citation Database
• Research Analytics
76
ISI WEB OF KNOWLEDGE 5
webofknowledge.com
77
ISI WEB OF KNOWLEDGE 5
CITATION UNIVERSE
78
ELIMINATION OF STOPWORDS
79
LEMMATIZATION
run, runs, ran and
running are forms of
the same lexeme, with
run as the lemma.
80
FULL RESULT COUNTS
NO MAXIMUM ON ‘ANALYZE RESULTS’
81
LEFT-HAND TRUNCATION
All users are
accustomed to using
right-hand truncation
in their searches
82
LEFT-HAND TRUNCATION
Now with left-hand truncation,
searches can be even more
accurately defined
83
ABSTRACT PREVIEW
84
ABSTRACT PREVIEW
85
CITATION COUNTS USING
WEB OF KNOWLEDGE VALUES
86
WEB OF KNOWLEDGE
CITATION BREAKDOWN TABLE
87
MARKED LIST ENHANCEMENTS
- ANALYZE TOOL & CITATION REPORTS
Analyze tool and citation
report avaiable for specific
products. Future release will
include these features for the
All Databases results
88
CITED REFERENCE SEARCHING
NOW FOR ALL CITATION INDICES
New cited reference search
parameters facilitate a more
accurate search
89
CITATION COUNTS IN FULL
90
THE DATA
• Web of Science
– Conference Proceedings Citation Index
– Century of Social Sciences
• ISI Web of Knowledge - The Citation Universe
– Web of Knowledge 5 upgrade
– Biosis Citation Index
– Chinese Science Citation Database
• Research Analytics
91
BIOSIS CITATION INDEX
BIOSIS CONTENT WITH CITATION DATA
• Biosis Previews
– Widest collection of life sciences fields in the world
– agriculture, biodiversity, biotechnology, botany, drug
discovery, gene therapy, marine biology, wildlife
conservation, zoology
• Biosis Citation Index
– Provides cited references for BIOSIS Previews
• Cited references for all Previews content (beginning 2006)
• Cited references for 60% of high impact Previews journals
(1926 - 2005)
• Exclusive to ISI Web of Knowledge platform
BIOSIS CITATION INDEX
ON ISI WEB OF KNOWLEDGE 5
93
BIOSIS CITATION INDEX
SEARCHING
Set your search
parameter defaults
Select years, use
lemmatization and sort
by publication date
94
BIOSIS CITATION INDEX
REFINE RESULTS
Identify the most abundant
sources of relevant results
or concentrate on harderto-find document types
Refine results or exclude
results from your dataset
via a series of parameters
95
BIOSIS CITATION INDEX
SORTING AND EXPORTING RESULTS
Select the records of your
choice and print, email or
export them via your favourite
bibliographic management
system
Sort your results to find
breaking new research papers
or to identify the most
influential or relevant papers
in your field
96
BIOSIS CITATION INDEX
MARKED LIST
97
BIOSIS CITATION INDEX
CITATION BREAKDOWN TABLE
98
BIOSIS CITATION INDEX
CITATION REPORT
99
BIOSIS CITATION INDEX
ANALYZE RESULTS
Analyze by Super Taxa
100
BIOSIS CITATION INDEX
CITATION MAP
Citation maps can find
relevant papers not returned
in your search results
2nd generation
citations reveal
citation pathways
101
BIOSIS CITATION INDEX
CITED REFERENCE SEARCHING
Cited reference
searching helps users
to find ‘missing
citations’
102
BIOSIS CITATION INDEX
SPECIALIZED INDEXING
Searching Biosis Citation Index for
‘Fossil* and Africa and humans’
returns this paper
The same search in ‘All
Databases’ reveals that this record
is also present in Web of Science
But the same search in Web of
Science does not find the article
It is the Biosis specialized indexing
fields that has found these terms
that are not indexed in Web of
Science
103
BIOSIS CITATION INDEX
SPECIALIZED INDEXING
• All BIOSIS records are indexed by biology specialists using
the following unique fields
Major Concepts
Concept Code(s)
Taxonomic Data
Disease Data
Chemical Data
Gene Name Data
Sequence Data
Geographic Data
Geologic Time Data
Methods and Equipment Data
Parts & Structures Data
Miscellaneous Descriptors
104
BIOSIS CITATION INDEX
SPECIALIZED INDEXING
All the specialized indexing
fields are searchable
The concept code 00506
codes for Explorations and
Expeditions
This paper is returned
because the concept code is
found on the BIOSIS record
105
BIOSIS CITATION INDEX
SUMMARY
• Combines
– World-leading Biosis life sciences content
– with valuable cited references back to 1926
• Enhances
– Web of Science searches via the All Database Search
using common indexing backbone
• Employs
– ISI Web of Knowledge cited reference searching
• Exclusive
– ISI Web of Knowledge
106
THE DATA
• Web of Science
– Conference Proceedings Citation Index
– Century of Social Sciences
• ISI Web of Knowledge - The Citation Universe
– Web of Knowledge 5 upgrade
– Biosis Citation Index
– Chinese Science Citation Database
• Research Analytics
107
EMERGENCE OF CHINESE RESEARCH
108
CHINESE SCIENCE CITATION DATABASE
AVAILABLE NOW!
• Over 2 million records
• Over 13 million cited references
• Subject Areas
– Mathematics, Physics, Chemistry, Geosciences, Biology,
Agricultural Sciences, Engineering (Computer Science,
Mechanics, etc.), Medicine, and Management
• Dual Language
– Since 2002, 80% of records have full English bibliographic
information and most have English abstracts
• Content is searchable in both English and Chinese
110
CSCD
CITATION BREAKDOWN TABLE
Bibliographic
information in
English and
Chinese
111
THE DATA
• Web of Science
– Conference Proceedings Citation Index
– Century of Social Sciences
• ISI Web of Knowledge - The Citation Universe
– Web of Knowledge 5 upgrade
– Biosis Citation Index
– Chinese Science Citation Database
• Research Analytics
112
AUTHOR NAMES
113
AUTHOR NAMES
114
RESEARCHERID
SCHOLARLY RESEARCH COMMUNITY
• Accurate Identification
• Organize and Manage
• Increase Visibility and Recognition
• Measure Performance
• Collaboration
• Security
Science, March 2009
INDIVIDUAL LEVEL
RESEARCH EVALUATION
See personalized
metrics using Web
of Science citation
data
View accurate publication
list due to unique author
identification
RESEARCHER ID
ANALYZE COLLABORATION NETWORK
Seek global collaboration
opportunities by author,
field, institution or country
RESEARCHER ID
VISUALIZE CITING ARTICLES NETWORK
RESEARCHER ID
ELECTRONIC CV RESOURCE
ResearcherID Profile:
http://www.researcherid.com/rid/A9180-2008
RESEARCHER ID
GLOBAL PARTICIPATION
Top Ten Institutions
TopCollege
Five Institutions
University
London
The University of Queensland
Monash University
Harvard University
University of Michigan
University of Pennsylvania
ETH Zurich
University of Cambridge
Stanford University
McGill University
Top Five Countries
United States
United Kingdom
China
Australia
Spain
France
Germany
Brazil
Canada
Italy
RESEARCHERID
UPLOAD SERVICE
• Institutions can upload content on behalf of their
researchers
– Upload researcher names  Obtain a ResearcherID
account
– Upload individual publication portfolios  Articles are
matched to Web of Science records and ResearcherID
portfolios are generated
RESEARCHERID
DOWNLOAD SERVICE
• Download data about the individuals at your
institution
– Names and name variants, current and past affiliations
• Download ResearcherID publication portfolios
– Bibliographic details of each item in the portfolio
– For articles that were successfully matched to Web of
Science records there will be a Times Cited count and UT
tag.
• Can be used by the customers for their own
internal systems