Constructing Performance Measurement Indicators in the Government’
Information Unit in Taiwan: using Balanced Scorecard and Fuzzy Analytic Hierarchy
Process
Yi-Hui Liang
Department of Information Management, I-SHOU University, Kaohsiung City, Taiwan, R.O.C.
(german@isu.edu.tw)
Abstract - The purpose of the study is to establish
balanced scorecard (BSC) in performance measurement of
Government’ MIS Department. We take a broader
definition of Government’ MIS Department as “an assembly
which brings forth some specific functional activities to
fulfill the task of MIS.” BSC used as a measurement tool to
assess study subjects, according to its strategy and goal
formed by its assignment property, can be divided into four
dimensions: internal process, customer, business value, and
future readiness, which can provide us with a timely,
efficient, flexible, simple, accurate, and highly overall
reliable measurement tool. In order to extract the knowledge
and experience from related experts to pick out important
evaluation criteria and opinion, this study combines fuzzy
theory and the analyical hierachy process (AHP) to calculate
the weights. After completing weighted calculation of every
dimension and indicator, the BSC model is thus established.
The findings of this study show that the indicator weightings
between and among all the levels are not the same, rather
there exists certain amount of differences. The degrees of
attention drawing in order of importance among all
dimensions are internal process, customer, business value,
and future readiness. After comprehensively analyzing
indictors of performance measurement included in every
level, the highly valued top three indictors are, when
conducting dimension performance measurement in
Government’ MIS Department, “Control cost,” “Satisfy end
user demand,” “Operate and maintain information
technologies efficiently”. From these studies we will be able
to develop the indicators and the calculated weights of the
four dimensions and the indicators mentioned above. This
model can be utilized by the information units of the
governments for constructing the strategies and blueprints
for self evaluation. Further, these can also provide important
information for effective resource investment in
Government’ MIS Department.
Keywords
- Government,
measurement, Balanced scorecard
MIS,
Performance
I. INTRODUCTION
Performance appraisal system is the most effective
tool used for government reengineering. Performance
appraisal aims to help people achieve their strategies,
missions, visions and goals.
Wu (2000)[1] supposed that Good performance
appraisal systems can enable government departments to
allocate reasonable resources, prioritize resource
investment, further improve departmental effectiveness
and efficiency, and organizational members adopt
identical methods to pursue their goals, encourage their
morale, and cause them to focus on organizational vision.
Traditional government
departments usually
developed their information systems according to their
individual requirements, and hence did not communicate
with each other, leading people to develop bad
impressions and stereotypes regarding government
performance owing to inefficient government operations.
Balanced Scorecard (BSC), which was developed by
Kaplan & Norton in 1992, is a useful and popular method
of identifying business performance using lagging and
leading indicators based on the foundation of visions and
strategies. Balanced Scorecard implies that organizational
performance is evaluated not only utilizing financial
indicators, but also simultaneously non-financial
indicators. Balanced Scorecard built a framework to
transform organizational vision and strategies into a series
of consistent performance indicators, and thus execute
and control organizational administration, allow
organizational members to more concretely learn the
vision and strategies of organization, and also l and also
help managers track the outcomes of implemented
strategies.
Since Executive Yuan, Republic of China
implemented performance reward and performance
management plan in 2003, this plan followed the BSC
spirit. However, Executive Yuan then consider the
business properties, organizational culture, and
management and check, so as to authorize each
government department to set up its own performance
evaluation process and evaluation indicators（DirectorateGeneral of Personal Administration，2005）[3]。Until now,
Executive Yuan does not force government departments
to set up their own performance evaluation process and
evaluation indicators（Chu and Cheng，2007）[4]。
The analyical hierachy process (AHP) [2], which is
the multi-criteria technique, is considered appropriate for
solving complex decision problems [3]. The AHP is based
on theory, and offers information on the relative weight of
the BSC performance indicator [4] [5]. Otherwise, Zadeh
(1965) [6] developed fuzzy theory to handle uncertain
problems involving fuzziness and vagueness. Lee et al.
(2008) [7] posited that traditional BSC failed to
consolidate diverse performance indicators. Lee et al.
(2008) [7] also suggested the fuzzy AHP method as as an
answer for this problem.
BSC can help managers of government organizations
holistically evaluate information technology (IT)
investments, as well as the performance of information
system (IS) departments. This study builds a Framework
for evaluating government MIS departments based on
BSC. The study summarizes how to combine the BSC and
fuzzy AHP to serve as a decision tool for government
organization. The tool can be used not only to assess the
contribution of a specific government MIS department,
but also analyze the performance and direct the activities
of government MIS departments.
3. Promote regularly IS applications
mix.
4. Increase regularly IS hardware
and software.
5. Implement cost-effective and new
technological researches which
are suitable for organizations.
D3
D4
D5
C. Fuzzy AHP
II. METHODOLOGY
Step 1: Construct hierarchical framework of the BSC
performance evaluation criteria
A. Research Structure
This study builds a Framework for evaluating
government MIS departments based on BSC and fuzzy
AHP.
B. Select research variables
This study adopted the dimensions and indicators
which developed by Martinsons et al. (1999) [7], Liang,
Hsieh, and Wang (2008)[9], and related government MIS
experts to develop my proposed the dimensions and
indicators. The research variables are showed as Table 1.
TABLE I
Research Variables
Dimension
Customer
1.
2.
3.
4.
5.
Business
value
1.
2.
3.
4.
Inner
process
1.
2.
3.
4.
5.
6.
Future
readiness
Indicator
Build and maintain the good image
and reputation with end users.
Have the opportunity to develop IT.
Maintain a good relation with user
communities.
Satisfy end user requirement
Perceived the preferred IS products
and services provider by end users.
Manage the good image and
reputation.
Make sure IS projects to offer
business value.
Control cost.
Be onerous to offer the suitable IS
products and services to the third
party.
Expect and affect the demands from
end users and managers.
Plan and develop IT efficiently.
Operate and maintain IT applications
efficiently.
Obtain and test new hardware and
software.
Offer to satisfy the end user trainings
with effective cost.
Manage the IS problems effectively.
1. Expect and prepare the IS
problems.
2. Train and develop regularly to
improve IS skills.
A1
A2
A3
A4
A5
B1
B2
B3
B4
C1
C2
C3
C4
C5
C6
D1
D2
From the four BSC perspectives, the hierarchical
framework of the BSC performance evaluation criteria is
constructed.
Step 2: Using AHP method to calculate the weight
If get the weightmatrix W in pairwise comparison matrix
A, standardize geometrical mean of row vectors, multiply
element in every row, get geometrical mean and normalize it.
Step 3: Construct Positive Reciprocal Matrix
Every evaluation member use fuzzy AHP evaluation scale
to express relative weight between each dimensions and criteria,
and construct fuzzy Positive Reciprocal Matrix.
Step 4: Consistency Check
The check methods are as follows:
4.1 Consistency Index (C.I.)
According to Consistency Index (C.I.), C.I=0 indicate that
evaluation has perfect consistency; C.I>0 indicate that
evaluation has consistency; C.I. <0.1 indicate that evaluation has
evaluation has tolerant bias.
4.2 Consistency Rate (C.R.)
Saaty (1980) [2] supposed that Consistency Rate (C.R.) to
evaluate the consistency of pairwise comparisons in a matrix
among criterions. Under the condition of different rank of matrix,
it produce different random index (R.I.). Under the condition of
the same rank of matrix, the ratio of C.I. to R.I. is called C.R..
When C.R≤0.1, the consistency level is acceptable.
Step5: Calculate fuzzy weight value
Utilize the Lambda-Max method which Csutora and
Buckley (2001) [17] proposed, calculate the fuzzy weight of
evaluation criterions. The steps of calculation are as follows:
5.1
When α=1, use α-cut to get median Positive
Reciprocal Matrix. Then, calculate the weight use AHP
method to get the weight matrix.
5.2 When α=0, use α-cut to get minimum positive
reciprocal matrix and maximum positive reciprocal matrix.
Then, calculate the weight use AHP method to obtain the weight
matrix.
5.3 In order to make sure that calculated weight value is
fuzzy number, therefore, adjusted the coefficient.
5.4
After
obtained
adjusted
coefficient,
calculate minimum positive
reciprocal
weight
matrix
and maximum positive reciprocal weight matrix of every
measurement dimension.
5.5 Combing adjusted minimum, maximum and median
values to get the fuzzy weight in kth evaluation member and kth
measurement dimension.
5.6 Utilize average method to integrate the fuzzy
weight of evaluation members and measurement
dimensions.
III. RESULTS
A. Survey Candidates
3.
4.
Based on previous studies on applying the BSC
approach to information systems, this study used the 21
indicators as performance evaluation indicators to
construct the research model and develop the
questionnaire items based the model. The 20 indicators
are showed in Figure 1.
Next, take the central engineering government
department as the example, and calculate the weights of
all dimensions and indicators of the model using Fuzzy
AHP method. The questionnaire was distributed among
Director and Vice Director of the direct department, 7
Director of first-class independent unit, 3 Section
Manager of the direct department, and Director of
Information Technology, and a total of 13 valid
questionnaires were returned and censored 2 invalid
questionnaires (refusing answer, incomplete answer, or
don’t passing the consistency check). The result of this
study is showed in Table 2.
The results demonstrated that the importance weights
of all dimensions were ordered as follows: internal
process, customer, business value, and future readiness.
Additionally, the top three importance weights of
performance evaluation indicators were top three indictors
are “Control cost,” “Satisfy end user demand,” “Operate
and maintain information technologies efficiently”.
B. Results of fuzzy AHP method
Inner
process
0.328
1.
2.
3.
4.
5.
6.
Future
readiness
0.156
1.
2.
TABLE II
Results
Dimension
Customer
weight
Indicator
0.277
1.
2.
3.
4.
5.
Business
value
0.239
1.
2.
Build and
maintain the
good image and
reputation with
end users.
Have the
opportunity to
develop IT.
Maintain a good
relation with
user
communities.
Satisfy end user
requirement
Perceived the
preferred IS
products and
services
provider by end
users.
Manage the
good image and
reputation.
Make sure IS
weight
0.046
rank
3.
11
4.
0.037
15
5.
0.053
8
projects to offer
business value.
Control cost.
Be onerous to
offer the suitable
IS products and
services to the
third party.
Expect and
affect the
demands from
end users and
managers.
Plan and develop
IT efficiently.
Operate and
maintain IT
applications
efficiently.
Obtain and test
new hardware
and software.
Offer to satisfy
the end user
trainings with
effective cost.
Manage the IS
problems
effectively.
Expect and
prepare the IS
problems.
Train and
develop
regularly to
improve IS
skills.
Promote
regularly IS
applications
mix.
Increase
regularly IS
hardware and
software.
Implement costeffective
and
new
technological
researches
which
are
suitable
for
organizations.
0.100
1
0.040
13
0.055
7
0.060
5
0.064
3
0.032
16
0.062
4
0.055
6
0.029
18
0.030
17
0.029
19
0.028
20
0.040
14
0.098
2
0.043
12
V. CONCLUSION
0.050
10
0.050
9
Performance
appraisal
systems
for
profit
organizations have traditionally measured performance
financially. For non-profit organizations a different
approach is used, since for profit is not a main objective
for such organizations, and possibly is even a constraint.
Financial performance represents a subjective measure of
how well a firm can use assets from its primary mode of
business to generate revenues. Notwithstanding, for
government organizations and other non-profit
organizations, financial performance primarily represents
a measure of how efficiently a government organization
can use its budget. Using financial performance to
measure is not sufficient to measure the government
organization performance.
This study develops BSC framework and calculate
the weights of the four perspectives and the indicators
mentioned above. These performance evaluation
indicators will then be utilized by the information units of
governments for constructing the strategies and blueprints
for self evaluation. Further, these can also provide other
related departments for effective resource investment in
information units of governments.
Compared to Miller & Doyle (1987) [15] 和 Saunder
& Jones (1992)[16], the proposed IS evaluation
dimensions and indicators more focus non-profit
organizations characteristics.
REFERENCES
[1] A. Wu, “The new direction for performance evaluation.”
The Chinese BAS Monthly, vol. 530, pp. 43-52, 2000.
[2] R. S. Kaplan, and D. P. Norton, “The Balanced ScorecardMeasures: That Drive Performance.” Harvard Business
Review, vol. 70, no. 1, pp. 72-79, January/February 1992.
[3] Directorate-General of Personal Administration，Executive
Yuan, performance reward and performance management
plan, 2005.
[4] H. L. Chu, and C. J. Cheng, “An Empirical Investigation of
the Governmental Performance Management System:
Using the Balanced Scorecard and the Analytic Hierarchy
Process.” Journal of Management, vol. 24, no. 6, pp. 689705, 2007.
[5] D. L. Searcy, “Alignment the balanced scorecard a firm’s
strategy using the analytic hierarchy process.” Mangement
Accounting Quarterly, vol. 5, no. 4, pp. 1-10, June 2004.
[6] S. L. Liedtka, “Analytic hierarchy process and multi criteria
performance management systems.” Cost Management, vol.
19, no. 6, pp. 30-38, 2005.
[7] M. Martinsons, R. Davison, and D. Tse, “The balanced
scorecard: a foundation for the strategic management of
information systems.” Decision Support Systems, vol. 25,
no. 1, pp. 71-88, March 1999.
[8] P. R. Niven, “Balanced Scorecard Step by Step: Maximizing
Performance and Maintaining Results, John Wiley and Sons
Inc, New York, 2002.
[9] Y. H. Liang, H. C. Hsieh, and C. Y. Wang, “A Study of
Implementing Balanced Scorecard of Government MIS
Department-A Case Study of Engineering Department of
Government.” Electronic Commerce Studies, vol.64, no. 4,
pp. 425-445, 2008.
[10] L. A. Zadeh, “Fuzzy Sets.” Information and Control, vol. 8,
pp. 338–353, 1965.
[11] R. Csutora, and J. J. Buckley, “Fuzzy Hierarchical
Analysis: The Lambda-Max Method.” Fuzzy Sets and
Systems, vol. 120, no. 2, pp. 181–195, April 2001.
[12] T. L. Saaty, The Analytic Hierarchy Process,
McGraw-Hill, New York, 1980.
[13] C. T. Chen, “Extensions of the TOPSIS for group
decision-making under fuzzy environment,” Fuzzy
Sets and Systems, vol. 114, no. 1, pp. 1–9, August
2000..
[14] R. S. Kaplan, and D. P. Norton. The Strategy-Focused
Organization: How Balanced Scorecard Companies
Thrive in the New Business Environment, Harvard
Business School Press, Boston MA, 2000.
[15] J. Miller, and B. A Doyle, “Measuring the
Effectiveness of Computer-Based Information
Systems in the Financial Service Sector.” MIS
Quarterly (11:1), March 1987, pp.107-124.
[16] F. A. Saunders, and J. W. Jones, “Measuring Performance
of the Information Systems function.” Journal of
Management Information Systems (8:4), 1992, pp. 63-82.