Quality Indicators
Indikator Kualiti dalam Pendidikan Tinggi
(Quality Indicators in Higher Education)
After completing this topic, you should be able to:
Explain the meaning of quality indicators
Describe the characteristics of quality indicators
Learn the importance of quality indicators in HEI
Understand the application of quality indicators in
HEI
Quality Indicators
Performance cultures in higher education
HEIs worldwide have undergone reform to improve
quality
HEIs have implemented systematic and formalized
quality assurance processes to achieve greater
efficiency and accountability
Establishment of quality models and organizations
to audit and review university performance
Institutional and national quality models and
performance indicators are vital components to raise
the standard of HEIs
Quantitative performance indicators are used to
provide international comparisons
Quality Indicators
Rationale for performance indicators
To ensure education provided by HEIs equips
students for employment and provide the country with
a highly skilled workforce that support economic
growth.
To contribute to educational, social, and political
values.
Quality Indicators
Purposes of performance indicators in HEIs
To monitor own performance for comparative
purposes.
To facilitate the assessment and evaluation of
institutional operations.
To provide information for external quality assurance
audits.
To provide information to the government for
accountability and reporting purposes (Rowe, 2004).
Quality Indicators
The use of performance indicators in HEIs
Ensure accountability for public funds
Improve the quality of higher education provision
Stimulate competition within and between
institutions
Verify the quality of new institutions
Assign institutional status
Underwrite transfer of authority between the state
and institutions
Facilitate international comparisons
Ref: Chalmers (2008)
Quality Indicators
Defining performance indicators
Simple indicators – expressed in absolute figures
and are intended to provide an unbiased description
of a situation or process.
Performance indicators – imply a point of reference;
for example, a standard, objective, assessment, or
comparator, are relative rather than absolute in
character. Involve value judgements.
General indicators – externally driven and are not
indicators in the strict sense; they are frequently
opinions, survey findings or general statistics.
Ref: Chalmers (2008, quoted from Hanney, Henkel & Kogan, 1997)
Quality Indicators
Defining performance indicators
Currently there is no common definition of
performance indicators.
PI cannot be considered as facts, but are goal,
value and context laden, and utilized in different ways
depending on the performance model being used.
PI are defined as measures which give information
and statistics context; permitting comparisons
between fields, over time and with commonly
accepted standards. They provide information about
the degree to which teaching and learning quality
objectives are being met within higher education
sector and institutions.
Ref: Chalmers (2008, p.3)
Quality Indicators
Types of performance indicators
Input
Process
Output
Outcome
The types can be categorized as quantitative
indicators and qualitative indicators.
Ref: Chalmers (2008, p.3)
Quality Indicators
Quantitative Indicators
define as those associated with the measurement
of quantity or amount, and are expressed as
numerical values.
Input indicators
Human, financial and physical resources in
supporting institutional programmes, activities and
services.
Output indicators
Output reflects the quantity of outcomes produced,
including immediate measurable results, and direct
consequences of activities implemented to produce
results. Do not demonstrate quality of education, but
quantities of outcomes.
Ref: Chalmers (2008)
Quality Indicators
Qualitative Indicators
associated with observation based descriptions,
rather than an exact numerical measurement or
value. Relate to or involve comparisons based on
qualities of non-numerical data such as policies and
processes for assessing students’ learning, the
experience, the content of a mission statement.
Outcome Indicators
Focus on the quality of educational program,
activity and service benefits for all stakeholders.
Insightful, meaningful and accurate since they are
related to the objectives of higher education.
Students are treated as customers.
Ref: Chalmers (2008, p.5)
Quality Indicators
Qualitative Indicators
Process Indicators
include the means used to deliver educational
programmes, activities and services within the
institutional environment.
qualitative information on teaching and learning
such as policies and practices.
Considered as most practical, useful and
appropriate measures of quality teaching and
learning.
Ref: Chalmers (2008)
Quality Indicators
Research conducted in Australia
Look at 13 process indicators
Mission, Vision and Objectives
Teaching and Learning Plans and Policies
Teaching and Learning Indicators
Internal and External Performance Funds for
Teaching and Learning
Organizational Unit Review
Curriculum Review
Assessment and Feedback Policies
Graduate Attribute Statement
Student experience
Ref: Chalmers (2008)
Quality Indicators
Research conducted in Australia
Look at 13 process indicators
Professional Development
Appointment and Promotion Criteria
Review of Academic Staff – performance
Recognition of Excellence in Teaching and
Enhancing Student Learning Experience
Ref: Chalmers (2008)
Quality Indicators
Research in Australia – Quality Teaching
Look at four dimensions of teaching practice
Institutional climate and systems – commitment to
the enhancement, transformation and innovation of
learning. Measure student experience and level of
satisfaction.
Diversity – relates to ethnic, cultural and
socioeconomic diversity as well as students’ abilities,
talents and learning approaches.
Ref: Chalmers (2008)
Quality Indicators
Research in Australia – Quality Teaching
Look at four dimensions of teaching practice
Assessment – the assessment tasks of student
enrolled in the program of study. Look at the design,
delivery and administration, provision of feedback,
moderation, and review of assessment.
Engagement and learning community – student
engagement, i.e. the student commitment and
engagement with their own education. Also includes
staff engagement.
Ref: Chalmers (2008)
Quality Indicators
Higher Education in Malaysia– Challenges
Peranan universiti dan ahli akademik (University’s
and academics’ roles)
Perkembangan kurikulum mengikut keperluan
pasaran (Curruculum development according to
market needs)
Penyelidikan, pembangunan dan pengkormesialan
dalam sistem inovasi kebangsaan (Research,
development and commercialization in the national
innovation system)
Kaedah pengajaran dan pembelajaran (Teaching and
Learning Methods)
Perluasan akses dan mobiliti pengetahuan
(Accessibility and knowledge mobility)
Pekeliling Kemajuan Pentadbiran
Awam Bilangan 2 Tahun 2005
Garis Panduan Bagi Mewujudkan
Petunjuk-petunjuk Prestasi Utama
Atau Key Performance Indicators
(Kpi) Dan Melaksanakan Pengukuran
Prestasi Di Agensi Kerajaan
Terminologi
Petunjuk Prestasi Utama (KPI) ialah salah satu kaedah bagi
mengukur prestasi perkhidmatan agensi-agensi Kerajaan
Perkhidmatan Teras Bidang tanggungjawab agensi sejajar
dengan visi (core business): dan misi agensi;
Proses Utama Fungsi-fungsi di bawah perkhidmatan teras
yang (core process): perlu dilaksanakan bagi menghasilkan
perkhidmatan untuk pelanggan;
Key Performance Petunjuk-petunjuk prestasi utama yang
ditentukan Indicators (KPI): sebagai asas mengukur
prestasi;
NKRA
Education
Crime (Public Safety)
Corruption
Low Income Households
Rural Basic Infrastructure
Urban Public Transportation
Higher Education in Malaysia–
Challenges
Quality Indicators
Globalisasi dan piawaian melalui pemeringkatan
dan penarafan (Globalization and standardization
through development and accreditation)
Peluang guna tenaga (Resource utilization
opportunities)
Quality Indicators
Malaysia - Current scenario
20 public universities
21 polytechnics
37 community colleges
>400 registered private colleges
21 private universities and university colleges
11 local university branch campuses + 5 foreign university
branch campus
Areas for indicators:
Academic staff
Educational programs
Student selectivity
Educational resources
Governance
The method used will be peer review.
Quality Indicators
How are universities ranked?
Different ranking approaches:
League table – each university is assigned a
specific rank. Higher ranks indicate higher quality,
lower ranks indicate lower quality.
Quality criteria and indicators are used in this
ranking methodology to assess universities.
Each indicator such as research impact as the
number of citations per faculty in the Thompson
Scientific Database or teaching quality as in THES
are given weight.
This approach are applied to all universities
assessed.
Quality Indicators
How are universities ranked?
Different ranking approaches:
A ranking of individual disciplines or departments
instead of whole institutions.
A multidimensional concept of university quality
instead of a ‘one-size-fits-all’ approach, taking into
account the diversity of academic institutions,
missions and goals as well as language and
cultural specifics.
A separate measurement and presentation of
single indicators.
A presentation of ranking results in rank groups
(top, middle, bottom groups) instead of league
tables.
Quality Indicators
Purposes of ranking of HEI
Ranking serves several purposes:
Responds to demands from consumers for easily
interpretable information on the standing of HEIs.
Stimulates competition among universities.
Provides some rationale for allocation of funds.
Helps to differentiate among different types of
institutions and different programs and disciplines.
Contributes to the definition of ‘quality’ of HEIs
within a particular country.
(source: Berlin Principles of Ranking of Higher Education Institutions, 2006
http://www.che.de/downloads/Berlin_Principles_IREG_534.pdf)
Quality Indicators
Berlin Principles of Ranking of Higher Education
Institutions
Purposes and goals of rankings:
Be one of a number of diverse approaches to the
assessment of higher education inputs, processes, and
outputs.
Be clear about their purpose and their target groups.
Recognize the diversity of institutions and take the different
missions and goals of institutions into account.
Provide clarity about the range of information sources for
ranking and the messages each source generates.
Specify the linguistic, cultural, economic, and historical
contexts of the educational systems being ranked – should be
aware of possible biases.
(source: Berlin Principles of Ranking of Higher Education Institutions, 2006)
Quality Indicators
Berlin Principles of Ranking of Higher
Education Institutions
Design and weighting indicators:
Be transparent regarding the methodology used
for creating the rankings.
Choose indicators according to their relevance
and validity.
Measure outcomes in preference to inputs
whenever possible.
Make the weights assigned to different
indicators (if used) prominent and limit changes
to them.
Quality Indicators
Berlin Principles of Ranking of Higher
Education Institutions
Collection and processing of data:
Pay due attention to ethical standards and the
good practice recommendations articulated in
these Principles.
Use audited and verifiable data whenever
possible.
Include data that are collected with proper
procedures for scientific data collection.
Apply measures of quality assurance to ranking
processes themselves.
Apply organizational measures that enhance the
credibility of rankings.
Quality Indicators
Berlin Principles of Ranking of Higher
Education Institutions
Presentation of ranking results:
Provide consumers with a clear understanding
of all of the factors used to develop a ranking,
and offer them a choice in how rankings are
displayed.
Be compiled in a way that eliminates or reduces
errors in original data, and be organized and
published in a way that errors and faults can be
corrected.
Quality Indicators
Purposes of Quality Indicator System
Colorado State, USA:
Encouraging continuous improvement by
institutions in achieving high levels of
performance.
Measuring institutional performance and
accountability.
Determining funding recommendations and the
funding distribution for the higher education
system.
Build public support for increased funding for
higher education.
Quality Indicators
Some Quality Indicators
Baccalaureate graduation rates
Achievement scores of graduating students on
various comprehensive examinations, tests, and
/or professional specific licensure or certification
examinations
Graduates employed or continuing their
education
Institutional support expenditures –
administration expenditure, expenditures per
student
Undergraduate class size
Faculty teaching workload
Quality Indicators
Current scenario
Categories of institutions
APEX university
Research intensive
General
The structure of Malaysian Qualifications Agency
(MQA)
One-stop center for institutions for registration
and accreditation of courses
Quality Indicators
Accountability and Quality
The concept of accountability and quality assessment in
higher education is an international phenomenon
In America, many regions are moving toward ‘performance
incentive funding’.
In Europe and Australia, the central government is directly
involved in establishing quality mechanisms through:
Quality control,
Quality audit
Quality assessment
The agencies involved are like the Higher Education Quality
Council and the Higher Education Statistics Agency (HESA)
Quality Indicators
Accountability and Quality
The focus of attention in performance indicators in the U.S.
has been cost efficiency, scientific and technical education,
management of intellectual property produced at universities,
and undergraduate education.
Less attention is paid to graduate education and research.
Categories of costs for higher education:
Education and general: faculty and staff salaries, equipment,
libraries, administrative and basic academic computing, and
certain capital or such as rent.
Cost for sponsored research
Costs of student living: room, board, clothing, laundry,
entertainment, and etc.
Cost of foregone earnings: While disengaged from the
productive work force.
The Movement in Setting Performance
Indicators in Higher Education
(U.S.A.)
1980s
Era concerned with growth in enrollments and
access was over
Emerging issues include:
Public accountability
Quality
Productivity
Undergraduate education
In 1986, all 50 states and the District of Columbia
had developed initiatives to improve the
undergraduate education
Shift from growth funding (formula funding) toward funding
outcomes, results, and performance
These efforts paralleled developments in Europe and Australia
1990s
The development of performance indicators differs from that in
1980s
From voluntary institutional improvement to a system of
mandated public accountability
By 1994, 18 states had developed indicator systems
Greater centralization of authority
Issue raised:
Will the federal government assume greater centralized control
of higher education through areas such as accreditation and
financial aids by using a set of national goals and performance
standards?
The Future of Higher Education
The White Paper 2003
Higher education must expand to meet rising skill
needs
The social gap among those entering university
remains too wide
Many of our economic competitors invest more in
higher education
Universities are struggling to employ the best
academics
Funding per student fell 36% between 1987 and
1997
Universities need stronger links with business and
industry
Reports of Institutional Effectiveness
(EOIE) Virginia’s Public Institutions
of Higher Learning
Annual report to provide meaningful information on
the academic quality and operational efficiency of
Virginia’s public institutions
To provide evidence of institutional effectiveness –
the extent to which institutions accomplish their
missions and students achieve their educational
goals.
Structure of the Reports
(Five Points)
Institution’s mission
The mission statement sets a vision for the
institution and defines how it will get there.
Institutional profile
In-depth views of enrollment and projections of
future enrollment.
System-wide measures
Include 14 performance measures focused on
operational efficiency and factors associated with
academic quality:
Example: Classroom and laboratory space
utilization, percentage of professionally accredited
programs and etc.
Institution-specific measures
Represent unique aspects of the mission that the
college or university chose to highlight
Core competency reports
Explore student general education assessments in
written communication and technology/information
literacy.
Performance Indicators of California Higher
Education, 2001
Describes the scope of the current set of
indicators reported by the Commission, and
highlights recent trends based on current
information related to these indicators.
This report are divided into five categories:
Population Context,
Fiscal Context,
Student Preparation,
Student Access, and
Student Outcomes
Key Performance Indicators (KPIs)
Key Performance Indicators are quantifiable
measurements, agreed to beforehand, that reflect
the Critical Success Factors of an organization.
They defer depending on organization
KPIs must:
Reflect the organizational goals
Be key to its success
quantifiable
Curtin’s efficiency and effectiveness
Key Performance Indicators (KPIs)
Teaching and learning
Effectiveness indicators
Quality of graduates
Quality of teaching
Student progress and achievement
Input
Efficiency indicators
Teaching and learning expenditure
Student progress and achievement
EFFICIENCY & EFFECTIVENESS
EFFICIENCY means: saving TIME, MONEY or
EFFORT
Efficiency measures the resources used to attain a
certain level of output
EFFECTIVENESS means how well the the job
gets done. i.e. the quality of the output.
Effectiveness measures the extent to which
outcomes have been achieved
The End of Quality
6th Quality in Higher Education International Seminar
(Birmingham, UK) May. 2001
1.
2.
3.
Three Major themes:
Has external quality review has its day?
Has control of quality been usurped by the
market and by information technology?
Does the development of mass education
necessarily mean the end of quality?
Transforming Quality
7th Quality in Higher Education International
Seminar (Melbourne) Oct. 2002
1.
2.
3.
Three main themes:
To reconceptualise how higher education
engages with access, employability and funding
issues
What constitutes a high quality learning process
and outcomes
How might quality evaluation be transformed to
help improve the quality of the experience and of
the learning?
8th Quality in Higher Education International
Seminar (Sheffield) May. 2003
1.
2.
Two major themes:
How does student feedback inform quality?To
what extent do institutions need to adopt new
procedures to make student feedback effective?
What does the White Paper encourage a closer
link between quality and learning?
The First Session of the Regional Follow-up Committee for the
World Conference on Higher Education (WCHE), 2- 3
November 2000, Kuala Lumpur Malaysia
1.
2.
3.
Recommendations to Member Countries:
Need for ongoing efforts to broaden access
taking into account the disadvantaged groups
(women and ethnic minorities)
Provide increased support for staff development
and research
More participation of women in higher education
particularly in decision making level
Indicators of Research Quality
in Higher Education
The vast majority of discoveries have been made
in higher education environment (Dill, 1986).
Review of literature on research productivity
highlighted several indicators which include:
Productivity dollars
Productivity publications
Peer evaluation
Productivity dollars
The number of dollars generated by research was the most
often cited measurement of success
Those universities that are ranked higher, their faculty have are
adept at obtaining research grants
Productivity publications
The number of publications is frequently used as an indicator of
quality in research
The research that is published is taken as an indication of its
quality
The types of publication which determine its quality:
Journal articles, monographs, chapters, books
Quality: reputation of publication in discipline, distribution of
publication, refereed vs. non-refereed journals, invited
chapters/papers
OUTPUT & IMPACT FACTOR
Output versus Impact factor of publications
Output refers to how prolific the the research is
producing acceptable articles/books
Impact was determined by checking citations of
the articles over a period of years
PEER EVALUATION
Peer Evaluation
Assemble a group of peers to review the research
efforts and make a determination of the quality of
those efforts
The concerns of such approach include:
The visiting group doesn’t fully understand the
work of the unit/individual being reviewed
(especially when it is multidisciplinary)
Curtin’s Efficiency and Effectiveness
Key Performance Indicators (KPIs)
Research and Development
Effectiveness indicators
Research Performance Index
Research Quantum
Comparison between Curtin and all Australian
Universities
Research Funding
Research Publications
Efficiency and Effectiveness
Key Performance Indicators (KPIs)
Efficiency Indicators:
Research Performance Index
Research Funding
Research Publications
Effectiveness measures the extent to which
outcomes have been achieved
Efficiency measures the resources used to attain a
certain level of output
Publications and Number of Ph.D. Graduates
2001
2002
2003
76
89
112
Publications
2000
1,864 (319 international journals, 212 in
local
journals)
2001
1,815 (303 international journals, 204 in
local
journals)
2002
2,507 (496 international journals, 328
local
journals)
Malaysia was ranked #63 by MASTIC (Pusat
Informasi Sains dan Teknologi Malaysia) in terms
of production referred journals
Critical Success factors (Research)
Wide academic base and facilities
Graduate students
Screening of IRPA application and monitoring
Evaluation of research and innovation
Research culture and administrative support
Strong research networking
Research cluster development
Incentives & in-house competition as incubator
Steps to be Taken by Institutions to
Promote Excellence in Teaching
Define what they mean by excellent in teaching
Having well-defined criteria about excellent
teaching and standards for weighting and rating of
teaching/research/service
Weigh teaching more heavily
Increase sophistication of teachers
Promote excellent teaching, not just excellent
teachers
Not treat promotion as a separate issue
(Gibbs, 1995)
Theoretical Definition for Excellent Teachers
The character of the professor
Values, personality, and social
intelligence
The knowledge of the professor
Disciplinary and pedagogical
understanding
The actions of the professor
Problem-solving behaviors
The responses of the students
Learning outcomes
Quality Teaching in Higher Education
Flexibility in approaches to teaching and learning (including
assessment)
Good organization of subject matter and course, including relevance
and coherence of content and planned teaching/learning activities
Effective communication
Knowledge and enthusiasm for subject matter and teaching
Facilitation of learning through student interaction and active
experience
Respect for and positive attitude toward students
Critically reflective orientation to teaching including effective use of
feedback to guide learning and improve teaching
Appropriateness and fairness in assessment and grading
Reeders, E, & Marshall, H. 1996
GOOD TEACHING: THE TOP TEN REQUIREMENTS
By Richard Leblanc, York University, Ontario , 1998.
1.
2.
3.
4.
Good teaching is as much about passion as it is
about reason.
Good teaching is about substance and treating
students as consumers of knowledge.
Good teaching is about listening, questioning,
being responsive, and remembering that each
student and class is different.
Good teaching is about not always having a
fixed agenda and being rigid, but being flexible,
fluid, experimenting, and having the confidence
to react and adjust to changing circumstances
6.
Good teaching is also about style
This is very important -- good teaching is about
humor.
7. Good teaching is about caring, nurturing, and
developing minds and talents.
8. Good teaching is supported by strong and
visionary leadership, and very tangible
institutional support -- resources, personnel, and
funds.
9. Good teaching is about mentoring between
senior and junior faculty, teamwork, and being
recognized and promoted by one's peers.
10. At the end of the day, good teaching is about
having fun.
Quality in College Teaching:
A Research Approach
Flexibility in approaches to teaching and learning
(including assessment)
Good organization of subject matter and course,
including relevance and coherence of content and
planned teaching/learning activities
Effective communication
Knowledge and enthusiasm for subject matter and
teaching
Facilitation of learning through student interaction
and active experience
Respect for and positive attitude toward students
Critically reflective orientation to teaching including
effective use of feedback to guide learning and
improve teaching
Appropriateness and fairness in assessment and
grading