2011 International Conference on Intelligent Building and Management
Proc .of CSIT vol.5 (2011) © (2011) IACSIT Press, Singapore
The evaluation criteria of Value for Money (VFM) of Public Private
Partnership (PPP) bids
Kharizam Ismail., Roshana Takim., Abdul Hadi Nawawi
Centre of Postgraduate Studies
Faculty of Architecture Planning and Surveying
Universiti Teknologi MARA (UiTM) Shah Alam Selangor, Malaysia
Abstract: Fundamentally, the notions of Value for Money (VFM) is associated with the concepts of whole
life value of service provided for construction projects. VFM is associated in reducing life cycle costs;
provide better allocation of risk, faster implementation, improved service quality and generating high
revenue of the project outcomes. In achieving VFM of Public Private Partnership (PPP) projects, a mixture
of financial and non-financial criteria are prerequisite in the evaluation of PPP bids to ensure the project
provide VFM to project stakeholders.
PPP bids evaluation had also been subjected to a wide criticism. The debate divulged the lack and unclear
process of detailing of how PPP bids evaluation process could provide VFM. Hence, the research aim is to
identify important criteria to be integrated in the evaluation of PPP bids for VFM. This research uses a postal
questionnaire technique conducted among all PPP stakeholders (public and private sectors) in Malaysia.
Through a descriptive analysis, the results revealed 6 out of 20 criteria as ‘very critical’ in the
evaluation of VFM of PPP bids. These are: optimum whole life cost, innovation, fit for purpose,
comprehensive specification, compliance on time, and appropriate risk allocation. Further analysis by means
of factor analysis technique revealed four principal components: project operational and social benefit;
managerial and financial criteria; and technical and environmental criteria. Hence to incorporate VFM in the
proposed of PPP bids, the 4 recommended principal components that embraced both financial and nonfinancial aspects are inevitable.
Keywords: Evaluation criteria; Descriptive analysis; PPP bids; Value for money; Factor analysis
1 . Introduction
Public Private Partnership (PPP) is seen as an effective way to achieve value for money (VFM) in public
projects. These benefits include introducing competition between prospective private bidders and exploiting
the greater efficiency and innovation in the private sector. VFM definition depends on the motives and
interests of the governments. It may change over time due to political, economical and social developments
(Akintoye et al. 2003). For instance, Moralos and Amekudzi (2008) claims VFM as a best available outcome
for the cost savings, benefits, appropriate risk allocation throughout the project’s lifetime which focused on
the quality and competency in order to meet the public requirements. Shoul (2005) believes in essence that
the VFM term is built on the economy, efficiency and effectiveness aspects. It relates to the idea that VFM as
an examination to determine whether the projects are performing economically; efficiently and effectively in
its use of resources, operations and pursuit the objectives of the stakeholders’ requirements. Thus,
undoubtedly VFM is not the lowest cost option but an understanding of the whole life benefits and
appropriate risk allocation between public and private sectors.
There are two ultimate goals in conducting VFM assessment of PPP projects. The first goal is to identify
factors to determine whether a project delivers VFM to stakeholders. The second goal is to assess potential
349
bidders that can significantly contribute VFM to the projects. Zhang (2008) indicates that the most important
requirement for achieving VFM is the selection of potential bidders in PPP projects. This is vital for the fact
that capabilities and reliability of concessionaire is the main focal point to the success of PPP projects
(Zhang, 2008). The concessionaires are responsible to finance, design, long term operation, maintenance and
transfer of the project facilities to the government at the end of the concession period. Concessionaire
proficiency depends on the overall resources, capabilities of the constituent companies, ability to formulate
competitive financial and technical packages of the proposed projects (Zhang et al. 2002). The practitioners
believed that the right choice of the evaluation criteria that assess the private bids is the core of a successful
best value throughout the PPP approach.
Generally, to achieve the best VFM, contractor selection should consider competitiveness, compliance
with client’s requirements, reliability of performance, qualitative superiority and life cycle cost. Literature
denotes many criteria in the VFM evaluation process of PPP bids. Ten dominant criteria as agreed by
Partnership Colombia (2005); HM Treasury (2006) are financial aspects, innovation; whole life cost;
incentive and monitoring; health, safety and environment, appropriate risk allocation; acquisition of
facilities management services, market interest and compliance the specification. Meanwhile, Zhang (2008)
classifies the criteria into four essential packages that can effectively measure bidders’ capability. These are:
financial package (optimum whole life cost); technical package (innovation of all aspects); safety, health and
environmental package and managerial package (risk management, dispute and contractual aspects).
Yuan et al. (2009) have drawn 5 different classifications of VFM evaluation criteria that include: (i)
Physical characteristic of projects (design, technology, bidders’ knowledge & capabilities, risk allocation); (ii)
Financing & marketing; (iii) Innovation & learning; (iv) Stakeholder’s indicator (client satisfaction) and (v)
Process indicator (facilities management, resources utilization, health & environment and time management).
The contractor evaluation criteria stated should be appropriately selected, cautiously evaluated by matching
the specific client objectives and project requirements. Full evaluation of bids should seek to identify the bid
that offers the best combination of financial and non financial aspects. Nonetheless, although many
researchers have highlighted the aforementioned criteria in VFM evaluation bid, there has been very little
explicit consideration of the actual significant criteria that should be taken on board in evaluation of bids
towards VFM.
In Malaysia, the implementation and policy of VFM has been the subject of considerable debate
and critiques. In spite of the government acknowledgement on the importance of VFM assessment in
PPP projects, a robust VFM framework comprising the criteria in the evaluation of PPP bids has yet to be
established. The probable reason could be that most of public procurement projects in Malaysia aim to
provide service facilities at the lowest possible cost, they neglecting the importance of VFM for the projects.
This paper hence attempts to put forward the issues in establishing the imperative criteria in evaluating the
bids of PPP in Malaysia through the questionnaire survey findings. All aspects of criteria financial and nonfinancial in PPP evaluation bids of VFM were emphasized.
2 Methodology of research
The questionnaire survey helped to determine the perception of both public and private sectors of PPP
stakeholders vis-à-vis the criteria involved in evaluating PPP bids. An empirical questionnaire survey was
undertaken within the Malaysian PPP stakeholders for a period of four months from February to May 2010.
To obtain an accurate and precise information, a purposive sampling technique was used. The target
respondents were selected based on their involvement with Malaysia PPP projects. The data was
accumulated from 216 target respondents which comprised of top management levels of contractors,
consultants and governments’ officers. They were chosen as they are the key stakeholders in PPP projects.
The targeted respondents have vast knowledge and experiences in BOT and PPP projects local and abroad.
A total of 51 respondents completed the questionnaires that were returned representing a response rate of
23.6%. Two separate statistical analyses (descriptive and inferential) were adopted using the Statistical
Package for Social Sciences (SPSS). The first analysis ranked the criteria based on mean value of responses.
One–way analysis of variance (ANOVA) was executed to test whether the mean values on each criteria for
350
the groups were equal. For the same data set, factor analysis technique was applied to the twenty criteria in
the PPP evaluation bids. The essense of the factor analysis is to explore the relative importance amongst
these criteria and to achieve a parsimonious reduction into fewer meaningful principal factors (Takim, 2005).
3.0
Key research findings
3.1
PPP evaluation bids-descriptive analysis
Table 1 shows the result of the analysis based on overall mean scores and the priority ranking by the
respondents. Out of 20 criteria, 6 criteria are regarded as ‘ very critical’ based on the overall mean scores
being over 4.00 and the scores range between 4.02 (appropriate risk allocation) to 4.14 (optimum whole life
cost). With regard to the three groups of respondents (government, consultants & contractors), government
and consultants signify the utmost concern by selecting all the six criteria as ‘very critical’. However,
contractors choose 4 out of 6 criteria as ‘very critical’.
These six criteria are: optimum whole life cost, innovation, fit for purpose, comprehensive specification,
compliance on time, and appropriate risk allocation. In agreement with HM Treasury (2006), the optimum
whole life cost is believed to be the most critical criterion in VFM evaluation of PPP bids. All respondents
(government, consultants and contractors) agreed that this criterion as the utmost important with the overall
mean value of 4.14. This indicates that the concept of VFM itself refers to the whole life value of the service
given. Since PPP is a form of procurement that involves an integration of finance, design, construction and
operation, the understanding of whole life cost’s concept is vital to achieve VFM. Therefore, the
performance of PPP should be assessed over the entire whole life cost of a project from inception to end of
the contract duration as suggested by Akintoye (2003).
The second critical criterion is innovation. The results showed that contractors rated innovation as
crucial in the evaluation of VFM of PPP bids with the mean value of 4.20. This indicates that contractors
believed that in order to win the PPP bids, it is important for them to offer innovative solution in the proposal
(Spiering , 2006). However, the choice by both government and consultants contradicted with contractors as
they rated ‘comprehensive specification’ as more critical than innovation. The third critical criterion is ‘fit
for purpose’ (overall mean=4.10). Once again both government and consultants are in line with the opinion
that the achievement of VFM should be assessed from the quality’s perspective and comply with building
functionality (fit for purpose). Thus, PPP bids should be streamlined and fit the purpose of the PPP projects.
TABLE 1: criteria in PPP evaluation bids
Criteria
Overall
mean
Criticality
R
Optimum whole life cost
Innovation
Fit for purpose
Comprehensive specification
Compliance on time
Appropriate risk allocation
4.14
4.12
4.10
4.08
4.06
4.02
1
2
3
4
5
6
V. critical
V. critical
V. critical
V. critical
V. critical
V. critical
Government
Consultants
Contractor
(N=17)
(N=25)
(N=9)
Mean
R
Mean
4.29
4.18
4.20
4.24
4.00
4.06
1
4
3
2
6
5
4.10
4.04
4.08
4.09
4.03
4.02
R
1
4
3
2
5
6
Mean
4.22
4.20
4.10
4.11
3.89
3.88
R
1
2
4
3
5
6
ANOVA
F
0.925
0.389
0.024
0.890
1.256
0.661
Sig
0.403
0.680
0.977
0.915
0.294
0.699
The fourth critical criterion is comprehensive specification with the overall mean value of 4.06.
Government and consultants appear to be more aware on the comprehensive specification of the proposed
projects compared to contractors. The probable reason could be government and consultants regarded
comprehensive specification could produce quality project outputs (Akintoye, 2003). Meanwhile compliance
351
on time and appropriate risk allocation are ranked in the fifth and sixth places with the overall mean value of
4.06 and 4.02 respectively. This may be explained by the fact that the respondents are more cost- quality
oriented rather than the configuration of these factors.
All the probability values (F ratio) of the criteria recorded are above 0.05 suggesting that similar
consensus of opinion occurred between groups (Government, consultants & contractors). These variables do
not have significant differences between groups. The results indicated that the null hypothesis could be
accepted. Nevertheless, based on the overall mean score method, all the 20 criteria are regarded as critical
achieving a mean score of 3 to 4. Factor analysis was then employed to all variables to reduce the larger
number of variables intentionally into fewer meaningful criteria based on the cluster of relationship among
them. The method of analysis and the results will be discussed in turn.
3.2
PPP evaluation bids-Factor analysis
Factor analysis was carried out to investigate the cluster of the relationship among the 20 inter-related
variables of evaluation criteria of VFM in PPP projects (Perry et al. 2008). The analysis was carried out
using SPSS. The 20 factors were subjected to factor analysis with principal component analysis and varimax
rotation. The value of test statistic for Sphericity was large (Bartlett’s test of sphericity = 1017.60) and the
associated significance level was small (p= 0.000), suggesting that there are relationship between the
variables. The value of the KMO statistic is 0.762, which according to Kaiser (1974) is satisfactory for factor
analysis. In essence, these tests show that factor analysis is appropriate for the factor extraction. Principal
component analysis produced a 4-factor solution with eigenvalues greater than 1.000, explaining 74.91% of
the variance in the data as shown in table 2.
The factor grouping is based on varimax rotation. Each variable belongs to only of of the factors, with
the loading on each factor exceeding 0.50. However, it is noticeable that factor “appropriate risk allocation”
has small loading value of 0.305 which required to be eliminated. Hence, the four Principle Components and
associated variables can be interpreted as follows: principal component 1 represents project operational and
social benefit; principal component 2 represents managerial and financial criteria; principal component 3
represents technical criteria (time, quality and functionality) and principal component 4 represents
environmental criteria.
Principal Component 1: Project Operational and Social Benefit
Principal Component 1 accounts for 23.95% (table 2) of the total percentage variance. The grouping
consists of 8 sub-factors: service provision, social benefit, operational aspects, assets utilization, sufficient
market interest, contractual aspects, flexibility operation, and anticipated user’s benefit. It is noteworthy that
these variables are loaded together under the principal component 1 (Project Operational and Social Benefit).
Two most important sub-factors in this grouping are ‘service provision and social benefit” with loadings (sig
=0.814 and 0.790 respectively). The result implies that project operational capability should addressed social
benefits as the outcomes, one of the vital factors in measuring VFM in PPP projects. This is in line with
Fabio (2005) who identifies operational process, technical capacity and contractual arrangements as
important criteria to be incorporated in the evaluation of PPP bids. The above results support the idea of
Asenova et al. (2005); Barretta and Ruggieero (2008) who indicated that the significance of operational
aspects such as availability of service provision and social benefit to be assessed in the bidding process of
PFI/PPP projects.
352
TABLE 2: Rotated component matrix
Factor components
Service provision
Social benefit
Operational aspects
Asset utilization
Sufficient market interest
Contractual aspects
Flexibility of operation
Anticipated users benefit
Management of resource allocation
Facilities management
Incentives and monitoring
Innovation
Optimum whole life cost
Fir for purpose
Quality of design
Compliance on time
Comprehensive specification
Meeting environmental standard
Health and safety
Eigenvalue
Percentage of variance explained
Cumulative percentage variance
Principal
component 1
0.814
0.790
0.771
0.689
0.684
0.608
0.603
0.534
Principal Component (PC)
Principal
Principal
component 2
component 3
Principal
component 4
0.861
0.719
0.672
0.594
0.582
0.771
0.748
0.646
0.627
0.875
0.865
10.653
23.945
53.263
1.626
18.062
61.394
1.477
17.454
68.781
1.226
15.452
74.912
Kaiser- Meyer-Olkin (KOM) Measure of sampling Adequacy = 0.762
Barlett’s test of Sphericity = 1017.60, significance p= 0.000
Principal component 2: Managerial and financial criteria
Principal Component 2 accounts for 18.06% of the total percentage variance and consists of 5 subfactors. These are: management of resource allocation; facilities management; incentives & monitoring;
innovation and optimum whole life cost. Two most important sub-factors in this grouping are ‘management
resource allocation’ and ‘facilities management’ with loadings (sig= 0.861 and 0.719 respectively).
Undoubtedly, PPP projects are required to integrate a comprehensive management of resources, facilities and
financial aspects to ensure any managerial uncertainties are resolved (Barretta and Ruggieero, 2008).
Principal component 3: Technical criteria (time, quality and functionality)
Principal Component 3 represents ‘Technical Criteria (time, quality and functionality) and accounts for
15.45 % of the total percentage variance. This grouping is made up of 4sub- factors namely: fit for purpose;
quality of design; compliance on time and comprehensive specification. Two important sub-factors are: fit
for purpose and quality of design (sig= 0.771 and 0.748 respectively). In the evaluation of PPP bids, both fit
for purpose and quality of design were to be highlighted to present VFM to the stakeholders. Previous
research by Zhang (2005), revealed that the most important technical criteria in the evaluation of PPP bids
are to comply with the design quality, meeting time frame and specification. The idea is supported by the
previous work of Barretta and Ruggieero (2008) indicating that technical criteria is primarily used to rank
PFI proposals in PFI projects in Italian health care sector.
Principal component 4: Environmental criteria
Principal Component 4 represents ‘Technical Criteria (time, quality and functionality) and accounts for
17.45 % of the total percentage variance. This grouping is made up of two important sub-factors: meeting
environmental standard and health and safety (sig= 0.875 and 0.865 respectively). Incorporating sustainable
353
environmental standard in the proposal of PPP projects could bring VFM to all stakeholders. As reported by
Ryan (2004), in European PPP legislation, the concepts of ‘a greening procurement’ pursue environmental
objectives in the selection and evaluation of PPP bids. Favorable environmental conditions are another
common goal in the PPP projects that has influences to both government and end users.
4 CONCLUSION
Certainly, concessionaires play a vital role in the achievement of VFM in PPP projects, making the
selection of proposal bids is absolutely crucial. Contractor selection is a multifaceted decision making
process involving a consideration of multiple criteria which are mostly subjective in nature. Literature review
indicated that many different imperative criteria are classified into different evaluation packages. Thus, this
requires the establishment of suitable evaluation criteria and the determination of their relative significance.
Since the aim of this paper is to establish a comprehensive set of evaluation criteria of PPP bids, an empirical
research by using factor analysis were conducted. It was discovered that, an appropriate criteria with due
considerations to financial and non financial criteria warrant to endorse a genuine VFM of PPP bids.
By using factor analysis technique, 4 recommended principal components are vital: project operational
and social benefit; managerial and financial criteria; and technical and environmental criteria. Aligned with
the study by Pangeran and Wirahadikusumah (2010), the implementation of PPP projects are more
complicated than conventional approach in the evaluation of proposal bids. Hence to incorporate VFM in the
proposed PPP bids, the proposed 4 recommended principal components are sought. In conclusion, the
evaluation criteria generated in this research is able to facilitate decision makers to formulate strategies in
selecting the robustness of PPP bids which significantly influences in achieve VFM of the PPP projects.
Nevertheless, a further empirical research in the form of case studies of PPP/PFI projects in Malaysia is
required to reinforce the initial findings. A combined approach of the two methodologies would be useful for
data elicitation prior to further conclusion.
The research presented in this paper is part of an ongoing PhD research at the Faculty of Architecture,
Planning and Surveying, UiTM Malaysia to develop a framework of Value for Money (VFM) assessment for
Public Private Partnership (PPP) projects focusing on Public Sector Projects. The results of the study could
provide an insight into the Malaysian construction project development and will hopefully provide valuable
guidelines, especially to public or private sectors in Malaysia.
5 REFERENCES
[1] A. Barretta and P. Ruggiero. Ex Ante evaluation of PFIs within the Italian health-care sector: What is the basis for
this PPP? Health policy 88. 2008.
[2] A.Akintoye C. Hardcastle, M. Beck, E. Chinyio, and D. Asenova. Achieving best value in private finance
initiative project procurement. Construction Management and Economics 21. 2003, pp 461-470.
[3] B. Ryan. Public Private Partnership and sustainability principles guiding legislation and current practice. Dublin
Institute of technology. 2004.
[4] D. Asenova, M. Beck. Obstacles to best value in NHS PFI projects: Evidence form two hospitals. Journal of
Finance and management in Public science, 4 (1). 2005
[5] D. Morallos and Amekudzi, “A state of the practice of value for money a n a l y s i s in comparing Public
Private Partnership to traditional procurement,” Public Works management Policy 13. 2008.
[6] H.R Perry, Brownlow. C, McMUrray. I and Cozens. B. SPSS Explained. Rout ledge Taylor & Francis
Group.2008
[7] HM Treasury, “Value for money assessment guidance” @ Crown copyright, 2006
[8] J. Shoul, “A critical financial analysis of the private Initiative: Selecting a financing method allocating
economic wealth,” Critical Perspective on Accounting, 16. 2005, pp 441 -471
[9] M. B. Spiering and G. Dewulf. Strategic issues in Public Private Partnership: An International perspective.
Blackwell publishing. 2006
354
[10] Partnership British Colombia. Project report: achieving value for Money Sea to sky highway improvement
project. 2005.
[11] P. Fitzgerald, Review of Partnership Victoria provided infrastructure. Final report to the treasurer.
Melbourne Australia. Growth solution group. 2004.
[12] R. Takim. A Framework for successful construction project performance. Unpublished PhD Thesis. 2005
[13] X. Zhang. Q, Best value concessionaire selection through a fuzzy logic system expert system with application.
Article in press. 2008
[14] X. Zhang. Q, Factor analysis of public clients’ best value objective in Public Privately Partnered Infrastructure
projects. Journal of Construction Engineering and management. 2006 , pp. 956-965
[15] Y. A Yuan, Zeng, Skbniewski. M.J and Q. Li. Selection of performance objective and key performance
indicators in Public Private Partnership projects to achieve value for money, Construction management
and economics 27, 2009.pp.253- 270
355