Academic Consortium 21 International Forum
October 18-21, 2010.
Understanding Research
Performance
The Evolution of UWA’s Research
Management Tool, Socrates
N.G. Mast, D.G. Glance & R.A. Owens
University of western Australia
Presented By Natalie Mast (natalie.mast@uwa.edu.au)
What is Socrates?
• In late 2006 The University of Western
Australia (UWA) launched Socrates, an
online application designed to draw data
from key research information systems, in
order for the University to prepare
portfolios for the Government’s Research
Quality Framework (RQF).
What does
Socrates do?
Publications
Database
•The system draws data from a
number of core UWA databases,
as well as two ISI databases and
Scopus.
Grants
Database
Student
Database
(HDR)
Web of
Science
Scopus
Journal
Citation
Report
•Socrates is then able to collate
data for individual staff members
to create a detailed research
profile, showing an individual’s
publications, citations, grant
income and HDR load.
How does Socrates measure
research Performance?
• Having established a profile for individual
researchers, it was then necessary to provide a
measure, whereby the research performance of
staff could be judged.
• A Socratic Index was introduced to help the
University gauge the level of research
productivity for its staff.
The Socratic Index
•
The Socratic Index is calculated using data from a six year period
(2004-2009).
•
Previously, there were 3 Socratic indexes in use within Socrates.
•
All three indexes relied on two forms of data, Research Outputs
(ROPs), namely publications, and Research Inputs (RIPs), that is,
competitive grant income (categories 1, 2, and 4 of the HERDC grants
data collection).
•
The difference in the three measures rests in the manner in which
publications are weighted.
The Original Socratic
Indexes
Government SI reflects the way in which income is gained via the Federal
Government’s block grant schemes; It includes only HERDC publications and
uses the same weightings for publications as the HERDC.
UWA SI reflects the way in which internal funding flows according to research
publication activity; a much wider range of publications are included, for
example, scholarly reviews, musical compositions, and book reviews. There is a
significant increase in the weighting of books, while the points attributed to
conference papers are halved when compared to Government points.
TISI SI relates only to performance in publications that are indexed through
the Thomson Reuters ISI Web of Science.
Socratic Index II
•
In January 2009, UWA launched Socrates II, with a new, single Socratic Index
(SI II) that focuses upon a six year research period, currently 2004-2010. SI II
was designed to reflect the research aims of the University and measure the
performance of our researchers accordingly. Overall, a more holistic measure
of research performance was desired, so in addition to measuring research
income and publications, higher degree by research (HDR) figures have been
included.
•
To determine the SI II, Research Output Points (P), Research Input Points (I),
and HDR completion points (S) for the five year are summed, and can be
represented thus:
•
SI II= P1+I1+S1
SI II Indicators
Publication Type
Points
Nature & Science papers
5
Books
15-20 As per UWA publication schedule
Non Indexed Journal Papers
1
Journal Papers indexed in Social Sciences Citation Index
4
Journal Papers indexed in Social Sciences and Science 3
Citation Indexes
Journal Papers index in Science Citation Index
2
Refereed Conference paper
0.5
All other publication types
As per UWA publication schedule (Nothing > 5)
HDR Completion Type
Points
Value of the Grant
Points
> $500,000
3 points
PhD completion
2 points
> $50,00 and < $500,000
2 points
Masters by research completion
1 point
> $5000 and < $50,000
1 point
Tabs for individuals
within Socrates
Analysis Tab
Within the Analysis Tab,
in addition to providing an
individual score for each
researcher, Socrates is also able
to produce averages for:
•The University
•Academic Levels
•Faculties
•Schools
History Tab
Protecting Staff
Privacy
• Socrates has been designed with a number of differential
access levels.
• Individual researchers can see their own detailed record
as well as all average figures.
• Heads of School can see all the detailed results of their
school members, as well as all averages.
• The same privileges exist for Deans at a faculty level.
• Members of the University Executive, the University’s
Tenure and Promotions Committee and a small number
of central administrative staff have access to all records.
Proving Assumptions about
Research Performance
• The University was for the first
time able to conclusively verify
a number of assumptions.
Socrates shows that, at the
University
level,
the
productivity of staff members
increases with their academic
role level. The same holds true
at the faculty level.
• Socrates allowed us to chart
the differences in performance
by academic role.
* LVLA = Associate Lecturer, LVLB = Lecturer, LVLC = Senior Lecturer, LVLD = Associate Professor, LVLE = Professor.
Using Socrates to chart
the impact of R&D
Policies
 Feng et al. (2004, p. 182) claim that ‘the key factor which will bring
out an increase in strength (in R&D activities) is the efficient
management of R&D activities’.
 Socrates plays a significant role in managing research performance
activities, in part due to its adaptability.
 When the University implements policies aimed at improving its
research performance, the Socratic Index formulas can be
recalculated to take into account the aims of the University.
 The University Executive can then gain an understanding of its
strengths and weaknesses in regards to its new aims, and staff can
quickly begin to change their behaviour accordingly.
Change in the SI II from
2009 to 2010
2009*
2010
Change
Total Academic Staff Numbers: 1249
1203
-46
Mean Research Input Points:
4.92
5.07
0.15
Mean Research Output Points:
19.3
21
1.7
Mean HDR Completion Points:
2.02
2.17
0.15
Mean Research Points (SI II):
26.25
28.26
2.01
*Scores for 2009 are based upon data from the period 2003-08, while scores for 2010 are generated using data from 2004-09
UWA Indexed Articles in the
Web of Science* 2004-2009
 The introduction of the Socratic Index,
has lead to a change in publication
behaviour.
 Staff members are now more aware of
the influence of citation data, and
understand that, as the University strives
to improve its position in the Shanghai
Jiao Tong Ranking, a greater importance
is now placed on journal articles indexed
by ISI.
*SCI Expanded; SSCI and HCI only
Year
Indexed
Articles
Percentage
Increase
All
Indexed
Items
Percentage
Increase
2009
1,867
107.0%
2580
107.2%
2008
1,745
109.9%
2,406
108.0%
2007
1,588
109.7%
2,228
107.2%
2006
1,447
109.2%
2,079
105.3%
2005
1,325
102.5%
1,975
104.8%
2004
1,293
1,884
Patterns of Publication
Benefits for Research
Management
• The ability to compare the research productivity
of staff is highly valuable.
• The University has been able to properly gauge
areas of strength and weakness in its research
performance.
• We can identify research groups working well
together.
• This is particularly useful when nominating
groups for schemes such as the ARC Centres of
Excellence Programme.
Benefits to the
University
• Numerous scholars have described the difficulty
in mapping the disciplinary fields of research
groups and then measuring a group’s performance
(see Bourke, & Butler, 1998).
• By introducing a measurement, based on each
individual’s performance, Socrates has managed
to bypass many of the difficulties involved in
measuring Research performance.
Additional Benefits
 The application is used by the Promotions and Tenure Committee to
chart the productivity levels of staff members and to determine their
patterns of publication and grant application behaviour.
 Anecdotal evidence suggests that individuals have begun to hold off
on promotion until they exceed the average for their current level, and
begin to perform at a level more consistent with the next step of the
academic ladder.
 Younger staff members have expressed satisfaction at having a gauge
by which they can measure themselves, and many consider the
average SI score for their level as the minimum score they should be
aiming for.
 Fundamentally, it is felt that Socrates has played an important role in
clarifying for staff members the requirements of the University in
terms of their research activity.
Benchmarking
Opportunities
•
Socrates has provided UWA with the opportunity to benchmark our
research performance against other Universities.
•
UWA is currently involved in a benchmarking exercise with one of our
Group of Eight partners, the University of Queensland (UQ).
•
Once UWA and UQ have agreed which elements of their research
performance data they wish to compare, a benchmarking index (BI)
based on, for example, the level of research inputs, outputs and research
training can be established, for comparison at the University, Faculty,
research discipline (FoR code) and staff classification level.
What else is
Socrates used for?
• The ERA has been designed to inform the
Government of the quality of the research being
undertaken in Australian Universities.
• The results of the ERA will significantly affect
reputation and drive government funding to
universities from 2011 onwards.
What does the ERA
Measure
•
Publications from 2003-2008 for eligible staff present on the census date of 31
March 2009, including honorary and adjunct staff who use a university’s byline, were required for the submission.
•
Research Income for the same period, but limited to income earned at the
Institution.
•
The ERA required Research Outputs and Income to be divided across the
eight clusters of disciplines using the 2008 Australian and New Zealand
Standard Research Classification (ANZSRC) hierarchical structure.
•
For the ERA only the two digit Divisions, of which there are 22, and the four
digit Groups, of which there are 157, ANZSRC Field of Research (FoR) codes
were used. Outputs could be assigned up to three FoR codes, and Inputs could
be apportioned to as many FoR codes as necessary.
Socrates' role in UWA’s
submission to the ERA
•
Using the FoR codes, Socrates was able to draw together the some of the necessary data at the four and
two digit level for the University’s submission to the ERA. However, a significant proportion of the FoRs
were manually added or updated in Socrates.
•
As part of the ERA, the ARC also established a ranked list of 20,712 journals and a ranked list of 1950
conferences. Socrates was used to match journal articles and conference papers to the four point scale
for journals and the three point scale for conference papers. Socrates was also used to match
publications to their citation count in Scopus (the RQF had planned to use Thompson Reuter’s Web of
Science but the ARC moved over to Scopus).
•
Having drawn data from the various databases, Socrates was also used as a collection tool, allowing
researchers to input their indicators of recognition, and the appointed “stewards” from research areas
were able to view and edit these submissions in Socrates. Background statements on each of the two
digit FoR codes were also entered into Socrates, and the application was used to edit those statements.
•
The University’s XML submission was generated in Socrates and then uploaded to the ARC’s’ System to
Evaluate the Excellence of Research’ (SEER) platform, which supports the submission, evaluation and
administration workflows of the ERA initiative.
ERA summary data
Future uses of Socrates
 Socrates should be able to provide teaching details for staff members, such as :
• # units where they are course controller;
• # units they teach per semester;
• # lectures given each semester;
• # tutorials given each semester;
• # student satisfaction survey results
 Such measures will also be useful in charting the University’s progression in
University rankings that take factors beyond research performance into
consideration.
 Additionally, university committee service could also be recognised by Socrates,
thereby allowing the Promotions and Tenure Committee with a more complete
view of a staff member’s teaching, research and community service work.
 We also hope to add additional data from the new product Incites, which will
provide comparative citation data for both articles published in a particular
journal and in specific fields.
Conclusion
 Socrates has lead to a significant change in the way
individuals view their research performance.
 It has changed the way the University views research
productivity at UWA.
 Socrates has proven to be an effective tool for the
measurement of research performance at the individual
and group level.
 It will also prove particularly useful in the long term as
an application by which to chart the impact of research
policy decisions upon the behaviour of researchers.