EX-ANTE EVALUATION OF E-GOVERNMENT PROJECTS: WHICH
METHOD TO USE?
Tina Jukić1 and Mirko Vintar2
Abstract
The paper discusses ex-ante evaluation of e-government projects. While such evaluation
has been written about since early 1990s in the management information systems
literature, it has not yet gained sufficient attention in the field of e-government. We
present data about public sector's expenditure on ICT and e-government as well as about
use and reasons for non-use of possibilities offered by e-government among citizens. In
addition, we present traditional evaluation methods and assess their suitability in the egovernment context. Finally, we present a conceptual multi-attribute model for ex-ante
evaluation of e-government projects.
1. Introduction
For almost a decade, the main focus of e-government related studies has been on
evaluation of the supply side of e-government. In the European Union (EU) context, the
Capgemini's measurement of online availability of public services has gained special
attention (Capgemini, 2007). However, a few years ago, research community has started
with evaluation of the demand side of e-government and the popularity of supply-side
rankings has somewhat diminished. The reason for that lies in the unexpectedly low
usage of e-government services. On the other hand, enormous resources have been
invested in developing e-government solutions – in 2004, e-government expenditure in
EU member states has reached almost 12.000 millions of EUR (eGEP, 2006). Data on egovernment usage, costs related to e-government projects and the fact that there has been
a great focus put on evaluation of e-government supply stimulates the question about
those investments’ pay off.
As Kertesz (2003, p. 2) ascertained, e-government projects are often launched on noneconomic reasons. »When e-government implementers are asked whether they have
performed a cost-benefit analysis before embarking on such ambitious projects, the
answer is almost always negative.« (ibidem) Even though this is something to be anxious
about, the question is if a simple cost-benefit analysis is sufficient in the e-government
context and, are alternative methods more useful? Which method to use? Namely,
methods such as net present value (NPV), return on investment (ROI), internal rate of
return (IRR), payback period, cost-benefit analysis (CBA) and similar are well known in
the private sector's investments evaluation.
After introduction, data about public sector's expenditure on ICT and e-government as
well as about use and reasons for non-use of possibilities offered by e-government among
1
Young researcher. Institute for informatization of administration, Faculty of administration, University of
Ljubljana. Ljubljana, Slovenia
2
Professor, head of the Institute for informatization of administration, vice-dean for scientific and research
activities, chair of Faculty of Administration's Research Center. University of Ljubljana. Ljubljana,
Slovenia.
citizens are presented. In addition, we present traditional evaluation methods and assess
their suitability in the e-government context. Finally, we present a conceptual multiattribute model for ex-ante evaluation of e-government projects.
2. ICT and E-government Costs and Actual Usage
In this part, we provide data on EU public administrations' expenditure on ICT and egovernment and actual usage of the latter.
First, it is necessary to explain the difference between ICT and e-government spending.
Indeed, the definition of each varies among authors and studies. Howsoever, the main
difference lies in the fact that solely the costs of hardware and software are relatively
easy to be estimated; but, e-government is not just about ICT - it requires organisational
change encompassing costs of internal reorganisation, costs of inter-institutional
reorganisation and cooperation, costs of change management and costs of initial training
(eGEP, 2006, p.15).
In 2003, public administrations in EU-15 countries spent about €30 billion on ICT (EITO
in: eGEP, 2006). A year later, EU public administrations spent nearly €12.000 millions
on e-government (ibidem) - vertical spending on health, education, defence is not
included in that data (Table 1).
Table 1: Estimation of public administration e-government expenditure. Source: eGEP (2006)
Expenditure
Country
(Mio €)
Austria
160.69
Belgium
141.61
Cyprus
3.80
Czech Republic
228.50
Denmark
479.58
Estonia
20.32
Finland
389.02
France
1895.85
Germany
2065.51
Greece
71.22
Hungary
52.94
Ireland
61.76
Italy
726.00
Latvia
2.58
Lithuania
11.20
Luxembourg
26.80
Malta
4.15
Netherlands
578.21
Poland
26.78
Portugal
134.93
Slovakia
6.81
Slovenia
30.98
Spain
468.74
Sweden
904.19
United Kingdom
3408.90
Total
11900.90
In UK, for example, government spent 1% of GDP on ICT in 2003 (Timmins in:
Mimicopoulos, 2004).
As mentioned in the introduction part of this paper, for a long time, governments all over
the world were mainly focused on the supply-side of e-government. Studies measuring
the number of services available online, their maturity, availability of other e-government
features etc. (e.g. Capgemini, 2007) became very popular. In the EU context that was
especially true for Capgemini's survey within which 20 basic public services were
observed3. But, a few years ago, research community has started with evaluation of the
demand side of e-government and the popularity of supply-side rankings has somewhat
diminished. The reason for that lies in the unexpectedly low usage of possibilities offered
by e-government. Namely, the latest Eurostat data, for example, revealed that only 28%
of EU-27 citizens used the internet for interaction with public authorities in the last three
months (Eurostat, 2008). In case of companies, the statistics is more optimistic – 68% of
them used the internet for that purpose in 2008 (ibid.). This is not very surprising if we
take into consideration that citizens need government services less frequently that
businesses do.
3. The reasons for non use of e-government offerings
Even though the interest for e-government was quite high (eUser, 2005; BISER, 2004;
SIBIS, 2003), the actual usage did not fulfil expectations. One of the reasons for that lies
in lack of awareness - if people do not know that public e-services are there, they will not
use them, even if those services have an added value for them (OGC, 2003; CDT &
infoDev). In addition, three main e-government adoption factors were shown to be linked
to an increased intention to use them, due to Carter and Belanger (2004): perceived
usefulness, relative advantage and compatibility. Moreover, Van Deursen et al. (2006)
stressed three factors that can be ascribed to low e-government usage: (1) geographical
distances may encourage citizens to use e-government services; (2) successful
multichannel approach that divert citizens from the web to call centres and service desks,
and (3) variables such as quality and user friendliness of e-government services.
According to Cullen and Reilly (2007) and Tassabehji and Elliman (2006), among others,
privacy and security concerns are also barriers to e-government use. On the other hand,
the results of the surveys (Table 2) revealed that the most frequent barriers to use of egovernment were a preference or need to contact ‘traditional’ government, low awareness
of e-government and concerns about data security. Kunstelj et al. (2007) indicate that
those who prefer doing the business with government actually miss added value of egovernment services.
3
In 2007, other indicators were measured as well.
Table 2: Factors influencing the use of e-government. Source: Kunstelj et al. (2007)
Research
Berner
Fachhochschule &
Unisys
(2006)
Region
Indicator
Results (%)
not sure of misuse of data: 56
Weak points of communication to government missing personal contact: 43
Switzervia internet among citizens who have already technical problems: 41
land
used the internet for contact with government not feel very competent user of internet: 21
too complicated: 19
Barriers to e-government anticipated before
must go to the office anyway to sign something: 58
use among respondents who generally have a for complex forms it is necessary to have face-to-face
eUser (2005) EU-10
need to get in touch with government and
advice from staff: 54
would like to do more e-government
concern on supplying personal data online: 45
Reasons for not contacting
was not aware can be done: 23
AGIMO
gov. via the internet for a particular contact
Australia
preferred to speak to/meet a real person: 11
(2005)
among respond. who have used internet in the
concerns about security of information: 8
last 12 months
English
Perceived disadvantages of e-communications prefer to make contact in person: 22
MORI
local
(website) among respond. saying very/fairly
do not have access to e-channels: 20
(2004)
authorities unlikely to use council’s website
don’t understand the technology: 19
no information about which gov mode amount (%)
website should be used
4
32
WangpiSignificance of barriers (1-5) for the adoption
no information about which service
patwong et Bangkok of e-government websites among respondents
3
36
is available online
al. (2005)
who have never experienced those websites
no desired info. or forms
3
32
insufficient instructions
3
41
4. Evaluation of E-government Projects
As ascertained within the eGovernment Economics Project (eGEP, 2006, p. 5-6), there
are various reasons for e-government costs assessment and monitoring, such as: (1)
justification of investments and gaining private sector partners; (2) determination of the
e-government service's impact on efficiency; (3) internal management control; (4)
accountable and transparent expenditure of public resources; (5) benchmarking reasons.
While evaluating e-government investments has to date not received a lot of attention
(compared to other fields of e-government such as supply-side, usage, readiness etc.), it
has been written about ICT investments evaluation largely in the private sector context
since early 1990s.
4.1 Traditional Investment Evaluation Methods
This section presents traditional investment evaluations methods (cost-benefit analysis,
payback period, return on investment, internal rate of return, net present value) and
discusses their (un)suitabillity in the field of e-government.
The basic idea of cost-benefit analysis (CBA) originates from Jules Dupuit, a French
engineer whose paper entitled as »On the Measure of the Utility of Public Works« is still
often cited (NCEE, 2009). Its practical development was a result of U.S. Federal
Navigation Act, which … »required that the U.S. Corps of Engineers carry out projects
for the improvement of the waterway system when the total benefits of a project, to
whomsoever they accrue, exceed the costs of that project.« (ibidem). Not earlier than in
the 1950's economists started developing consistent steps for measuring costs and
benefits (ibidem). Nowadays, CBA is a widely used method for project appraisal in
numerous fields. Its basic aim is to assess all costs of the project (investment) and
compare them with expected benefits. However, in its original form it only allows
assessment of financial costs and benefits with common unit of measurement – thus
equivalent money value.
In addition, the use of payback period (PP) technique reveals the time required for
benefits meeting costs. Its weakness is that relying (solely) on such results may lead to
underestimation of projects producing high costs or overestimation of projects having
high yields at the beginning (Ponikvar, 2009).
Return on investment (ROI), indeed, refers to returns gained from an investment. It does
not consider the fact that the value of money today is not the same as it will be in the
future. This weakness is solved by the internal rate of return (IRR) and its superior - net
present value (NPV).
PP, ROI, IRR and NPV were assessed by Milis and Mercken (2004) for four criteria: (1)
consideration of entire lifetime of the investment; (2) consideration of time value of
money; (3) possibility of risk-levels to be entered into the feasibility evaluation; (4)
possibility of risk-levels to be entered in the selection of mutual exclusive projects. As
seen in Table 3, NPV fulfils all of the selected criteria, followed by ROI/ARR. However,
the review of research conducted by the same authors revealed that those two methods
are used less frequently.
Kertesz (2003), for example, applied NPV method to Romanian e-tax project and found
out that the project has had a negative economic value. However, it seems that this
project did not require cross-agencies approach which makes such analyses much
simpler. In addition, as the author herself mentioned, some data about the project were
not available, thus she used assumptions, while in some cases no costs were defined, even
though a thorough plan would probably reveal them.
Moreover, most of traditional approaches to appraise ICT and/or e-government projects
fail to address the problem of costs complexity within those kind of projects which arise
from the following six factors (LaVigne, 2001): (1) building, maintaining, and managing
relationships; (2) the similarity of existing processes and work to the envisioned ones; (3)
the similarity of existing technologies to desired technologies; (4) separability of tasks;
(5) intended degree of integration in the final product; (6) variations in data sources.
Furthermore, we did not notice the use of public value concept within those
methodologies. Grimsley et al. (2006, p. 1) defined public value as … »the value which
citizens and their representatives seek in relation to strategic outcomes and experiences of
public services. It includes the value attached to the relatively concrete outcomes, such as
reduced homelessness or universal access to health care, and to the more intangible, such
as increased trust in government and public service providers. It also captures experience
of the quality of public services and government …« Kearns (2004, p. 6) stressed eight
factors indicating the pursuit of public value (1) the provision of services that are widely
used; (2) increased levels of user satisfaction with services; (3) increased information and
choice available to service users; (4) greater focus on the services that the public believes
are the most important; (5) increased focusing of new and innovative services on those
most in need; (6) reduced costs of service provision; (7) improved delivery of outcomes;
(8) a contribution to improved levels of trust between citizens and public institutions.
Taking into consideration the fourth criterion (greater focus on the services that the
public believes are the most important), one could argue that current e-government
research has been to narrow since it focused mainly on the administrative procedures and
information, excluding e-health, e-education etc., which has been studied separately.
Namely, it could be that the public finds these areas much more important than
administrative procedures.
Table 3: Comparison of traditional investment evaluation methods. Source: Milis and Mercken
(2004, p. 89)
PP
No
No
No
ROI/ARR4
Yes
No
Yes
Does it consider the entire lifetime of the investment?
Does it consider the time value of money?
Can risk-levels be entered into the feasibility evaluation?
Can risk-levels be entered in the selection of mutual exclusive
No No
projects?
IRR
Yes
Yes
Yes
NPV
Yes
Yes
Yes
No
Yes
To summarize, traditional investment evaluation methods are not sufficient in the field of
ICT investments (Milis and Mercken, 2004; Anandarajan and Wen, 1999), and especially
not in the field of e-government (LaVigne, 2001).
4.2 How to Proceed?
In this section, we present a conceptual multi-attribute model for ex-ante evaluation of egovernment projects.
At the Institute for informatization of administration (Faculty of Administration)5 we
started the development of multi-attribute model for ex-ante evaluation of e-government
projects. Its final version will be developed using the software tool DEXi6 and it will
enable selection of services that should be provided electronically first (see also Jukić and
Vintar, 2008). It should be noted that the model is still in its conceptual design; we plan
to complete it in collaboration with e-government decision makers in Slovenia.
The model is shown in Figure 1. As it is seen, it consist of seven groups of attributes. In
case of costs and benefits, both – tangible and intangible dimensions should be
considered. The model also follows the principle of public value; namely, it considers the
alingment of the project with public's priority fields. Since lack of awareness, doubts
about security and privacy and incompleteness of e-government offerings were proved to
be barriers to e-government adoption, those attributes are included as well. In addition,
risks are considered as well – some traditional methods do not leave a space for the latter.
Finally, largeness of the target group is considered as one of the most important group of
model attributes. Namely, at least in Slovenia, for some time, services available at the
state e-government portal were those with small target groups – e.g. for farmers and
veterans. Our model considers both, (1) the number of traditional service users and (2)
percentage of (regular) internet users within the latter.
4
Accounting rate of return.
http://www.iiu.si/
6
http://www-ai.ijs.si/MarkoBohanec/dexi.html
5
DEXi enables evaluation of alternatives, i.e. alternative e-government projects which
increases model's usability. In addition, we plan to define the weights of attributes in final
estimation – this approach is also supported by DEXi. Since multi-attribute decision
models developed with DEXi are qualitative, one might argue that this is the weakness of
the model. However, we believe that this is a strenght for two reasons: (1) it enables
faster decision making in contrast to gathering detailed data which can be quite time
consuming; (2) values used can be (conditionally) quantitative as well – namely, the data
are entered according to pre-defined scales (symbolic values assigned to attributes) and
we can define the meanings of those scales (in a separate file).
Tangible and intangible dimensions
Benefits:
- for government
agencies
- for external users
(citizens, businesses)
Costs:
- for government
agencies
- for external users
(citizens, businesses)
Largeness of target group:
- number of users of this
service in its traditional
form
- % of (regular) internet
users within traditional
service's users
Risks:
- political
- organizational
- execution
E-government
project
Alignment with public's
priorities:
- does the project produce
the service in the field
whic public opinion is
considered as priority?
Publicity:
- is the service and
its security and
privacy planned to
be advertised?
Completeness of the service:
- does the project produce
single/isolated service or a
whole life-event?
Figure 1: Conceptual model for ex-ante evaluation of e-government projects
References
AGIMO – Australian Government Information Office (2005). Australians’ Use of and
Satisfaction with E-Government Services. Commonwealth of Australia, 2005.
http://www.agimo.gov.au/.
Anandarajan, Asokan and Wen, Joseph (1999). Evaluation of information technology
investment. Management Decision, Vol. 37, No. 4, pp. 329 – 339
Berner Fachhochschule, Kompetenzzentrum e-Government & Unisys (2006). 3. EGovernment
Trendbarometer,
Nutzen
und
Trends
aus
Bürgersicht.
http://www.hsw.bfh.ch/content/File/weiterbildung/egov/studien/Trendbarometer.pdf.
BISER Project (2004). eGovernment - the Regional Dimension. http://www.bisereu.com/.
Bohanec, Marko (2008). DEXi: A Program for Multi-Attribute Decision Making Version 3.0. http://www-ai.ijs.si/MarkoBohanec/dexi.html.
Capgemini (2007). The User Challenge Benchmarking The Supply Of Online Public
Services,
7th
Measurement.
September
2007.
http://ec.europa.eu/information_society/eeurope/i2010/docs/benchmarking/egov_benchm
ark_2007.pdf. 20. 3. 2009.
Carter, Lemuria and Belanger, France (2004). Citizen Adoption of Electronic
Government Initiatives. In: Proceedings of the 37th Annual Hawaii International
Conference on System Sciences (HICSS'04) - Track 5.
CDT & infoDev (2002). The E-Government Handbook for Developing Countries.
http://www.infodev.org/en/Publication.16.html.
Cullen, Rowena and Reilly, Patrick (2007). Information Privacy and Trust in
Government: a citizen-based perspective from New Zealand. In: Proceedings of the 40th
Annual Hawaii International Conference on System Sciences (HICSS'07).
eGEP - eGovernment Economics Project (2006). Expenditure Study Final Version. DG
Information Society and Media, European Commission, 2006.
eUser Project (2005). eUSER
eu.org/Document.asp?MenuID=68.
Population
Survey
2005,
http://www.euser-
Eurostat (2008a). Industry, trade and services - Information society statistics, Information
society: Structural Indicators - E-government usage by individuals by gender
http://epp.eurostat.ec.europa.eu/tgm/table.do?tab=table&init=1&plugin=1&language=en
&pcode=tsiir130. 18. 4. 2009.
Eurostat (2008b). Industry, trade and services - Information society statistics, Information
society:
Structural
Indicators
E-government
usage
by
enterprises.
http://epp.eurostat.ec.europa.eu/tgm/table.do?tab=table&init=1&plugin=1&language=en
&pcode=tsiir140, 19. 4. 2009.
Grimsley, Mike, Meehan, Anthony and Sen Gupta, Kushaan (2006). Evaluative Design
of e-Government Projects: A Public Value Perspective. Proceedings of the Twelfth
Americas Conference on Information Systems, Acapulco, Mexico, August 04th-06th
2006.
Jukić, Tina and Vintar, Mirko (2008). Zasnova večparametrskega odločitvenega modela
za podporo ocenjevanju učinkov e-uprave. Uporabna informatika, 2008, Vol. 16, No. 3,
pp. 180-190.
Kearns, Iain (2004). Public Value and E-Government. Institute for Public Policy
Research, UK.
Kertesz, Sorin. (2003). Cost-Benefit Analysis of e-Government Investments. Harvard
University, J.F. Kennedy School of Government, Cambridge, 2003.
Kunstelj, Mateja, Jukić, Tina, Vintar, Mirko (2007). Analysing the demand side of egovernment : what can we learn from Slovenian users? Lect. notes comput. sci., No.
4656, pp. 305-317.
LaVigne, Mark (2001). Underestimating e-government costs proves costly - Traditional
approaches aren't enough. http://www.netcaucus.org/books/egov2001/pdf/egovtcos.pdf.
Milis, Koen and Mercken, Roger (2004). The use of the balanced scorecard for the
evaluation of Information and Communication Technology projects. International Journal
of Project Management, Vol. 22, No. 2, pp. 87-97.
Mimicopoulos, Michael G. (2004). E-government funding activities and strategies.
United
Nations,
New
York,
2004http://unpan1.un.org/intradoc/groups/public/documents/un/unpan021913.pdf.
MORI (2004). E-citizenship: What People Want. Norwich City Council, 2006.
http://www.e-citizen.gov.uk/. 09. 11., 2006.
NCEE - National Center for Environmental Economics (2009). Introduction to BenefitCost
Analysis.
http://yosemite.epa.gov/EE/epa/hair.nsf/25bdea40eebf07508525677c0062b890/c015bf17
82af593285256479007f3e4a!OpenDocument. 16. 4. 2009
OGC
(2003).
Measuring
the
Expected
Benefits
of
e-Government.
http://www.publicnet.co.uk/abstracts/2003/10/20/measuring-the-expected-benefits-of-egovernment/.
Ponikvar,
Nina
(2009).
Finančna
struktura
podjetja.
http://miha.ef.unilj.si/_dokumenti3plus2/191167/financna_struktura_podjetja_nov.ppt. 27. 6. 2009.
SIBIS project (2003). SIBIS Pocket Book 2002/03: Measuring the Information Society in
the EU, the EU Accession Countries, Switzerland and the US. http://www.sibis-eu.org/.
Tassabehji, Rana and Elliman, Tony (2006). Generating Citizen Trust in E-government
Using a Trust Verification Agent: A research Note. In: Proceedings of the European and
Mediterranean Conference on Information Systems (EMCIS) 2006, Costa Blanca,
Alicante, Spain.
Van Deursen, Alexander, Van Dijk, Jan, Ebbers, Wolfgang (2006). Why E-government
Usage Lags Behind: Explaining the Gap between Potential and Actual Usage of
Electronic Public Services in the Netherlands. In: Wimmer, M.A. et al. (eds.): EGOV
2006, LNCS, Vol. 4084. Springer-Verlag Berlin Heidelberg, 269–280.
Wangpipatwong, Sivaporn, Chutimaskul, Wichian, Papasratorn, Borworn (2005). Factors
Influencing the Adoption of Thai eGovernment Websites: Information Quality and
System Quality Approach. In: Proceedings of the 4th International Conference on
eBusiness, Bangkok, Thailand.