Excellence in Research for
Australia
Dr Marcus Nicol
ARMS 2014
18 September 2014
1
What is ERA?
• ERA evaluates the quality of the research undertaken in Australian
universities against national and international benchmarks.
• The outcomes (ratings) are determined and moderated by committees of
distinguished researchers, drawn from Australia and overseas.
• The unit of evaluation is broadly defined as the Field of Research (FoR)
within an institution based on the Australia and New Zealand Standard
Classification (ANZSRC).
• The indicators used in ERA include a range of metrics such as citation
profiles which are common to disciplines in the natural sciences, and peer
review of a sample of research outputs which is more broadly common in
the humanities and social sciences.
• ERA is a comprehensive collection. The data submitted by universities
covers all eligible researchers and their research outputs.
ERA Process Overview
Citation Analysis
and/or Peer Review
Journal Quality
Volume and Activity
Research Income
Applied Measures
Esteem
Note - There are no weightings
Evaluation Stages
The ERA 2012 National Report presents data submitted as part of a comprehensive
assessment by discipline of the research quality and research activity within Australia’s
higher education institutions.
ERA 2015 changes
• Streamlined data collection –
• book publisher and conference series lists
• digital storage of peer review outputs
• New data collections for reporting and analysis
(not evaluation):
• Gender
• Open access
• New category of research reports for external
bodies
The ERA Data Collection
Data provided by Universities & Citation
Data Supplier
Coded by 4-digit Field of Research
codes
Presented by indicator category
ERA Indicator Categories
Research quality
Research volume and activity
Research application
Research recognition
ERA Unit of Evaluation – the FoRs
2-digit
4-digits
6-digits
19 Studies in Creative Arts and Writing
1901
1902
1903
1904
Art Theory and Criticism
Film, Television and Digital Media
Journalism and Professional Writing
Performing Arts and Creative Writing
>
190401
Aboriginal and Torres Strait Islander Performing Arts
>
190402
Creative Writing (incl. Playwriting)
>
190403
Dance
>
190404
Drama, Theatre and Performance Studies
>
190405
Māori Performing Arts
>
190406
Music Composition
>
190407
Music Performance
>
190408
Music Therapy
>
190409
Musicology and Ethnomusicology
>
190410
Pacific Peoples Performing Arts
>
190499
Performing Arts and Creative Writing not elsewhere classified
1905
1999
Visual Arts and Crafts
Other Studies in Creative Arts and Writing
•
The ERA Unit is not the department nor the individual researcher
•
Various 6-digit codes will sit within a 4-digit UoE
Coding Outputs
Assigning outputs to FoR codes
•
•
•
Submitted outputs may be assigned to up to three FoR codes relevant to
the output.
For journals only, an article must be assigned to any of the FoR codes listed
for the journal.
Reassignment exception - institutions may assign journal articles
containing significant discipline content (≥66%), but published in nondiscipline journal, to the relevant discipline code.
Apportionment of outputs
•
•
•
Where more than one FoR code is attributed to a single output,
institutions must apportion across FoR codes.
This prevents outputs being counted multiple times in different FoRs
within the same institution.
The minimum is 20% per FoR code. The total of apportionment for each
output equals 100% within an institution.
ERA 2015 Reference Periods
Data type
Reference period
Years
Research outputs
1 Jan 2008-31 Dec 2013
6
Research income
1 Jan 2011-31 Dec 2013
3
Applied measures
1 Jan 2011-31 Dec 2013
3
Esteem measures
1 Jan 2011-31 Dec 2013
3
Staff census date: 31 March 2014
We: arc.gov.au
Email: era@arc.gov.au
9
ERA 2015
Explanatory Statement
Volume
and
Activity
Publishing
Profile
Peer
Review
Citation
Analysis
Esteem
Measures
Research
Income
Applied
Measures
Indicators of Research Quality
Publishing behaviour (Universities & Citation Provider)
Output metadata
Citation analysis (Citation Provider)
Average RCI against world benchmark &
Australian HEP benchmark by UoE
% of papers indexed by Citation
Provider
% of contribution
to the Australian HEP FoR total
for papers & for citations
ERA peer review (Peer Reviewers, REC members & Universities)
ERA Reviewer reports
Research statements
for original creative works
Public policy documents/reports
Peer-reviewed Australian and International income (Universities)
Apportioned
By year
Indicators of Research Volume & Activity
Research outputs (Universities)
Output metadata
(including type of output and quantity)
Research statements
for original creative works
Research staff (Universities)
FTE & headcount
Status
Discipline base
Other research items within the context of eligible researchers (Universities)
Research income
Applied measures
Esteem measures
Research Application
Applied Measures (Universities)
Plant Breeder’s
Rights
Patents
Commercialisation
income
NHMRC endorsed
guidelines
Registered designs
Some categories of research income (Universities)
CRC income (Cat. 4)
Industry research income (Cat. 3)
Some categories of Esteem (Universities)
Count of Membership of Statutory Committee
Research Recognition
Measures of Esteem (Universities)
•
•
•
•
Editor of a prestigious work of reference
Fellowship of a learned academy and membership of AIATSIS
Recipient of a nationally competitive research fellowship
Recipient of an Australia Council grant or fellowship
Example - Applied/Impact Data
Explanatory
Statement
• Included for each assessable 2-digit
unit of evaluation
Publishing profile
• Research statements for creative works
• Public policy reports
Applied Measures
• Patents
• Plant Breeder’s Rights
• Registered Designs
Income
• Industry (Australian and International A & B)
• CRC research income
• Commercialisation income
Example - matrix of indicators
ENGINEERING & ENVIRONMENTAL SCIENCES
Electrical and Electronic Engineering (0906)
 Citation Analysis
 Patents and Research Commercialisation Income
BIOLOGICAL SCIENCES & BIOTECHNOLOGY
Crop and Pasture Production (0703)
 Citation Analysis
 Patents and Research Commercialisation Income
 Plant Breeders Rights
PUBLIC & ALLIED HEALTH SCIENCES
Nursing (1110)
 Citation Analysis
 Patents
 Membership of a Statutory Committee
 NHMRC-Endorsed Guidelines
Interrogating the ERA Data Set
Topic: Maximise social and
economic participation in society
Data is shown at the four digit Field
of Research (FoR) code level:
•
Relevance - The colour of the
circle represents the relevance of
that FoR to the keyword
•
Level of activity - The size of the
circle indicates the level of activity
in that FoR
ERA data: informing research
policy
‘Renewable Energy’ research not
immediately apparent at the fourdigit Field of Research (FoR) code
level.
Text mining techniques applied to the
ERA data set identifies the range
and focus of related activity within
the higher education sector.
ARENA report is a collaboration with
CSIRO.
http://arena.gov.au/files/2014/05/AR
C-study.pdf.pdf
Thank you
20