Quality Assurance in Higher Education
and Vocational Education and Training
WS 8/Panel 1: Reflection on the demands on quality in HE
Gudrun Biffl/AUT/WIFO
Globalisation and Creation of a
Common HE-Area by 2010 in
Europe (Bologna Process)
• increased the demand from students, employers, and
academics for indicators of the international academic
standing of universities. Global companies recruit
globally; students intending to study abroad choose from
an array of universities; international agencies, who are
the buyers of research, seek expertise on the basis of
international rankings/indicators.
Potentially conflicting targets:
• national objectives, e.g. democratisation of access
versus concerns for quality in HE,
• internal functioning of universities versus demands
emananting from the external environment (globalisation,
Bologna Process)
• Standardisation versus specialisation
What does quality mean? How is it
measured?
• Different aspects of quality for different interest groups –
students, staff, scientific community, nation states, supranational (EU) and international institutions
• Output of what:
– Research or Teaching?
•
•
How to Measure Performance: teaching/research
Ranking of HE-institutions as an entity and/or in the
various disciplines in a country, in Europe, world wide?
• One numerical value? On the basis of what indicators
and their respective weights
• Many indicators – based on what criteria?
What are the concerns of students
•
•
•
•
•
•
scores required for entry,
class size,
ratios of academic staff to students,
degrees held by academic staff,
attrition/graduation rates,
and spending on student services, especially
libraries and IT.
• Research performance of academic staff gets a
low weight
What are the concerns of funding
institutions
• overall standing of an institution is evaluated to a large extent on the
basis of research outputs and citations/patenting/affiliations of Nobel
Prize winners at different stages in their careers. Research
performance is measured over a period of years (up to two decades
for highly-cited researchers).
• Governments are increasingly using performance measures to
allocate funds to universities - as governments want their country to
leave a footprint in science (the scientific impact of nations) funding
for research excellence increases
• Rankings of a given discipline across universities may differ from the
ranking of institutions as single entities.
• The variability in the quality of departments within an institution is
falling as a result of the growth in interdisciplinary research and
quality controls on departments and universities.
• “Bad” departments in good universities are becoming rarer.
Definition and determinants of
quality in HE:
according to a survey of academic leaders worldwide the
determinants may be grouped into six broad
categories, :
1. standing of academic staff :Highest weighting: 40% of
a compounded rank indicator, greatest range in scores
by universities
2. quality of undergraduate intake (weight 11%),
3. quality of undergraduate programs (14%): this is the
area where standardisation takes place
4. quality of graduate programs (16%),
5. Resources (11%), and
6. peer opinion (8%).
How to measure academic staff
research quality
• the quantity and quality of research publications;
• citations of the research output;
• research income, especially from competitive
grants;
• recognition of academic standing through
election to academies,
• receipt of prestigious awards or invitations to
conferences;
• status of degrees held by academic staff;
• research output/degrees held - best measure of
current performance
How to measure quality of students
and teaching excellence
•
•
•
•
•
•
•
•
•
Degree of selectivity of student intake (tertiary entrance score (TES,
baccalaureate-Matura grades) are generally the result of State-wide
examinations at the end of schooling: selection may vary by degree course,
high correlation between TES values for students and their graduation rates
and scores
number of completions (through-put),
placement on graduation (employability),
continuation into higher studies, particularly progression rates into Ph.D.
courses,
awards won or honours grades achieved,
diversity of the student body and class size,
student evaluation (of Ph.D. programmes in terms of research quality)
The ratio of students to academic staff (with the increased role of teaching
through the internet input from technical staff should also be included)
teaching and supervision excellence of academic staff are reflected in the
measures of the quality of undergraduate and graduate programs
How to measure resources
• Resource levels can be measured in physical units or monetary
amounts.
• The use of physical units facilitates international comparisons
(support staff, library volumes and IT facilities).
• Monetary measures include total revenue per student (indicator of
the resources available for teaching and research training) or staff
member (indicator of the resources available for administrative and
technical assistance for research), and expenditure on IT and
libraries.
• Alumni giving and salary levels are sometimes included as
measures of standing.
• In principle, to allow for size differences between institutions,
revenue should be deflated by a weighted average of student and
staff numbers.
• Insofar as revenue per academic staff member is used to pay higher
average salaries it is also a further measure of the standing of
academic staff.
Who are the global peer reviewers?
• chief executive officers (CEOs) of
universities of high standing
• chief executive officers (CEOs) of global
enterprises
• chief researchers/advisors of international
agencies
Spillovers into the society, local
communities
• Universities may have other missions than scientific
excellence, and only the latter is evaluated on the basis
of internationally applied performance or reputation
indicators.
• The contributions of universities extend to their impact
on society, such as contributions to public policy and
technology; they may serve the local community or
region by transferring technology and know how to the
local actors (technical and socio-economic innovation),
as well as by providing policy advice.
• This role and impact is ignored in international
scoreboards
Open questions
• Today no robust and consistent way to measure
the quality of research in HE Institutions
• Nature of science is changing: there is more
intersection across disciplines
• while at the same time only narrow
specialisations get published in Journals
• How to evaluate the bridging skills/contributions
to natural and social science and humanities?
Extending the range of variables to provide a more
comprehensive picture of the standing/role/impact
of HE institutions
• by extending the range of data to include variables such
as citations in government policy papers and patents
taken out,
• by extending the qualitative data by surveying key
decision-makers outside the higher education sector,
• By including books and contributions to books: the
impact of books is undervalued in evaluation studies
• Include the influence of the internet.
These various extensions of coverage would help to
redress the bias in existing measures towards more
theoretical research.
Impact of ranking on HE-institutions
Will ranking procedures contribute to widening of
standing/status of universities and reinforce a scientific
elite league?
Will excessive evaluation endanger scientific research ?
Research shows that citations are a bad indicator of
quality of research, as indicated by a low correlation
between experts scores given to institutions/subfields
and citation analysis scores
A balance has to be found, i.e. a combination of methods:
– The use of data-systems for evaluation,
– advanced bibliometric assessment procedures (citations in fields
and of institutions),
– expert surveys (Peer reviews)
Conclusion
• International academic standing is only one criterion by
which a university’s performance may be measured.
• There is the need to develop other measures of
performance
– which have a more local focus, such as contributions to regional
development, contributions to national well-being,
– and detailed evaluations of teaching performance which takes
account of the mix of students.
A range of criteria linked to mission statements is
necessary if countries are to have a diverse higher
education system which caters for the full range of
desires of its people and the needs of society