NSW Department of Finance and Services Tendering Manual Appendix 7A – Evaluation Methodologies EVALUATION METHODOLOGIES Evaluation of Tenders Qualitative Narrative Assessment This methodology, as its title implies, is a written assessment which discusses the relative merits of tenderers to assist in making a judgement about the tender providing best value for money using a qualitative comparison. By itself, it is inappropriate for more complex, technical evaluations and other situations where this will not distinguish between the value offered by the tenders. Comparative Assessment & Ranking Method Comparative assessment involves ranking the tenders in relative order of merit against the requirements of the RFT documents, evaluating each evaluation criterion and the overall merit of each tender against all requirements. The Tender Evaluation Plan (TEP) must define the ranking method (as per the options described in this appendix) to be used during the evaluation to assist in comparing tenders. In particular, specific guidance should be provided on the scoring or other system to apply in the ranking of tenders and on any other matters to be considered in the determination of relative value for money. The TEP should identify the required justification for all ranking judgements. For example, Tender X provides a higher quality of (brief details) and level of (% for X and % for Y) and provides better value for money than tender Y, and is therefore preferred on this criterion. Any qualitative arguments would be supported by quantitative analysis. Least Cost Method Least Cost or 'first past the post' involves selection of the lowest reasonable priced tender or cost item that meets all mandatory requirements and is otherwise acceptable. If an item or tender does not meet all mandatory criteria it is passed over and no longer considered. Where the procurement involves more than just meeting base requirements, this may not identify the tender providing the best value for money. Normalising RFT documents should nominate the terms required to enable proper comparisons of tenders. Where potential service providers have submitted tenders subject to disparate terms and conditions it is necessary to ‘normalise’ the competing tenders so that there is a common basis for comparison. Adjustments may be required for the following terms by assessing their effect on relative tender prices or the relativity of other criteria, for example: firm versus variable prices; different price variation provisions; Free into Store (FIS) versus Free on Board (FOB); where maintenance/training and the like are included or excluded; differing warranty periods; PWM-0633, Issue – Edition 3 – Version 1.00 - Tendering Manual – August 2010 1 NSW Department of Finance and Services Tendering Manual Appendix 7A – Evaluation Methodologies differing settlement discount arrangement; and differing payment arrangements. Matrix Selection Method This method is used where it is necessary to evaluate using a number of variables or criteria. The method combines a numerical scoring system and criteria weighting to identify the level of importance of each of the evaluation criteria. The score is a measure of the level of compliance or merit whilst the weighting identifies the relative level of importance of each evaluation criteria. The first step is to identify and assign a weighting to each of the measured evaluation criteria. As the desirability of the criteria increases the weighting of the criteria increases. The weighting could be determined by considering the relative importance of each criterion using a paired comparison matrix. The individual weightings would be decided prior to the release of any RFT documents and the percentage weighting for the price component (eg. 70%) and the percentage weighting for the non-price component (eg. 30%) included in the RFT documents. The breakdown of the weighting for each non-price criterion would not normally be provided in the RFT documents. Mandatory “pass or fail” criteria such as financial capacity would not be weighted or scored against. There should be a balance between more and less desirable evaluation criteria and their weightings so that innovations such as new product types, alternatives and other innovative offers, are given due consideration. Note that too many mandatory evaluation criteria might exclude reasonable potential tenderers from responding. The comparison of tenders is achieved by using a scoring or merit scale to rate each tenderer’s level of compliance and relative value (as with the numerical scoring methods described below). These scores are then multiplied by the weighting of each criterion to provide a weighted score. For example a 0-5 scoring scale is used below. All tenders are assumed to have met any mandatory (pass or fail) criteria in this example. Matrix Selection Table Criteria A Tender 1 Tender 2 Tender 3 B C D E F G H Weighting Score Weighted Score Score Weighted Score Score Weighted Score 1. 10 3 30 3 30 5 50 2. 6 5 30 3 18 2 12 3. 4 4 16 3 12 5 20 Total 20 76 60 PWM-0633, Issue – Edition 3 – Version 1.00 - Tendering Manual – August 2010 82 2 NSW Department of Finance and Services Tendering Manual Appendix 7A – Evaluation Methodologies To calculate the weighted score for each tender: Column B multiplied by column C equals the weighted score for Tender 1; Column B multiplied by column E equals the weighted score for Tender 2; Column B multiplied by column G equals the weighted score for Tender 3. On the face of it, Tender 3 (total weighted score of 82) appears the best, notwithstanding the tender has performed poorly with criterion two. The evaluator would need to apply judgement to decide whether this tender is in fact the best given the variability of scores and sensitivity of the result to changes. Sensitivity Analysis Sensitivity testing allows key factors that determine the outcome to be identified by showing how sensitive the preferred outcome is to changes in the criteria weightings and other variables. If a slight variation in the scoring causes a disproportionate change in the evaluation results, then the significance of the evaluation scores should be questioned. Sensitivity analysis could include a variation in criteria weightings of up to 5%. The extent of change within the range of reasonable criteria weightings identifies the order of certainty and significance of the total weighted scores. If altering weightings and scores within a reasonable range with the chosen methodology does not change the overall result, then the overall evaluation score outcome is likely to be reliable. Scoring Tenders Detailed evaluation requires scoring or otherwise comparing/evaluating tenders against each of the evaluation criteria (other than cost, which is scored using the relative prices and costs assessed). To score, you must choose an appropriate scale and define an objective method for assessing each tender’s score. Tailor the scale range used in any particular tender scoring to reflect the nature of the tender process and proposed contract. Sometimes a scale of 1 to 5 is appropriate, where “1” represents non or poor compliance (conformance) and “5” exceeds requirements, such as: 5 points — Meets or exceeds the requirement in all respects; 4 points — Meets the requirement but may be marginal in minor aspects; 3 points — Meets the requirement except for minor aspects; 2 points — Does not meet the requirement, but may be adaptable or made acceptable; 1 point — Does not meet the requirement to a major degree. In other cases, a broader range of scoring is appropriate, reflecting finer degrees of differentiation between tenders and their merit. A scale from 1 to 10 is generally considered convenient, where the relationship between the score and its relative measure of compliance could be interpreted as follows: PWM-0633, Issue – Edition 3 – Version 1.00 - Tendering Manual – August 2010 3 NSW Department of Finance and Services Tendering Manual Appendix 7A – Evaluation Methodologies 0 Non-compliance or poor response. Well short of requirements and includes unsubstantiated claims about capability. 1-2 Unsatisfactory response. Does not meet minimum requirements or is inadequately substantiated. 3-4 Marginal compliance. Contains certain material deficiencies that prevent full compliance with requirements. 5-6 Satisfactory compliance. Contains minor deficiencies preventing full compliance. 7 Full compliance. Adequate response to requirements and appropriately substantiated. 8 Exceeds compliance. Very satisfactory response; more than adequate response to requirements and well substantiated. 9 Significantly exceeds requirements. Excellent response; surpasses all requirements and is fully substantiated. 10 Superior or outstanding compliance. Well exceeds the specified requirements, for example, a new breakthrough technology. Another method of scoring tenders (see the Consultant Management System User Guide, Aspects of a Tender Evaluation Plan) uses the following: 100 Meets all requirements of an ideal proposal. 90 Meets most requirements of ideal proposal. 80 Meets many of the requirements of ideal proposal. 70 Meets a number of the requirements of ideal proposal. 60 Meets requirements, but only just satisfactory for this criteria. <60 Fails to meet the requirements. May reject this proposal. When scoring tenders, consider each tender against one criterion before moving on to the next. This focuses attention on the individual criterion and helps ensure that a score is not confused by other aspects of the tender. To assess a tender score, refer to the list of questions or sub-criteria for each of the criteria. For example, one of the criteria may have six elements to look for in the tender. If (using the scale 1 to 10) there is little or no evidence of their coverage, then assign a score of “0”. If some elements are covered and others not, then a score of “3 to 6” may apply. If all six are well covered, then a score of “7 or 8” is warranted. If the tenderer clearly offered more than required, then score it “9 or 10”. The scores on each of the criteria are then weighted, and a total weighted score calculated for each tender. Together with the assessed price or estimated total procurement cost converted to a weighted score, these results are used to determine which two or three tenders to shortlist or single preferred tender to identify for final assessment. The final assessment of tenders should focus on: verification of the tenderers’ capability and credentials through independent checks, interviews of referees, and presentations to client representatives (as needed); confirmation of their preparedness and ability to enter into a satisfactory contract; and PWM-0633, Issue – Edition 3 – Version 1.00 - Tendering Manual – August 2010 4 NSW Department of Finance and Services Tendering Manual Appendix 7A – Evaluation Methodologies confirmation of the details of their offers, particularly costs and deliverables as needed. As a result of this final assessment, the scores awarded during the detailed evaluation may be modified to reflect new or revised information. The evaluation effort may be reduced by culling tenders based on a mandatory or minimum performance standard set for some or all of the evaluation criteria. The RFT documents must clearly specify what mandatory levels of performance apply with any criteria. Paired Analysis Paired analysis is a contemporary, useful way to both evaluate tenders and establish evaluation criteria weighting’s. The basic approach is to compare all tenders (or items to be weighted) two at a time, and determine which of each pair is the better or more important. An example is shown in Table 1 below. Table 1 Paired Analysis for One Criterion Tender A B C D E Red Company A Yellow Company B B Blue Company C A B Green Company D A B C Gold Company E E E E E 2 3 1 0 4 20 30 10 0 40 Capability Criterion Total % In Table 1, five tenders (with ten paired combinations) are evaluated. The table shows the results for one criterion (in this case, Capability), and each criterion has its own table. The Total shows the number of times that each tender was the better of a pair. The % is the normalised score for Capability that can then be weighted. PWM-0633, Issue – Edition 3 – Version 1.00 - Tendering Manual – August 2010 5 NSW Department of Finance and Services Tendering Manual Appendix 7A – Evaluation Methodologies A refinement of this technique, called Qualitative Differentiation, includes an indicator of how superior the better of a pair is. Table 2 illustrates the method used to weight criteria. It shows both the more important criteria of each pair (the letter) and the extent to which it is more important by adding a number as follows: 1 = Marginally more important 2 = Moderately more important 3 = Significantly more important If the method is used to rank tenders for a specific criterion, then the definitions would be: 1 = Marginally better 2 = Moderately better 3 = Significantly better The resulting values are summed to give totals and normalised by converting to a percentage. When there are few criteria, it is common for one to end up with a zero %. Since this is a weighting, it must be changed to a positive number and the other percentages then adjusted accordingly. Different scoring methods give different results. Table 4 shows the results of three different approaches to paired analysis using the example data from Table 2. The approaches are: Standard method used in Table 1. Qualitative differentiation used in Table 2, where relative importance is assigned. Balanced scoring. This approach also uses relative importance, but applies the balanced scoring in Table 3 for the relative importance. PWM-0633, Issue – Edition 3 – Version 1.00 - Tendering Manual – August 2010 6 NSW Department of Finance and Services Tendering Manual Appendix 7A – Evaluation Methodologies Table 2 Analysis Table With Qualitative Differentiation Criteria A B C D E F G H I Functionality A Support B A2 Capability C C1 C1 Vision D A3 B2 C3 Technology E A1 E1 E1 E2 Viability F F1 B1 F1 F2 E1 Price G A1 B2 C1 D1 G2 F1 Contractual H A2 B2 C1 H2 H1 H1 G2 Risk I I3 B1 I2 I3 I2 I1 G1 I2 9 8 7 1 5 5 5 4 13 15.8 14.0 12.3 1.8 8.8 8.8 8.8 7.0 22.8 Total % Table 3 Scores Used in Balanced Scoring Qualitative Difference Better of Pair Worse of Pair 1 5 3 2 6 2 3 7 1 Equal 4 4 PWM-0633, Issue – Edition 3 – Version 1.00 - Tendering Manual – August 2010 7 NSW Department of Finance and Services Tendering Manual Appendix 7A – Evaluation Methodologies Table 4 shows how the different methods of paired analysis result in different weightings of the criteria. Also notice the range of weightings. The standard paired analysis tends to converge, qualitative differentiation accentuates differences, and balanced scoring is roughly in between. Each approach has its strengths and, of course, an average of them could be used. The balanced scoring method has particular merit where there are few (four or less) criteria or tenders, because each gets a score. All the methods are useful for exploring the sensitivity of evaluation results. Table 4 Comparison of Results from Different Methods Standard Total A B C D E F G H I 5 5 5 1 4 4 3 3 6 13.9 13.9 2.8 11.1 11.1 8.3 8.3 16.7 Paired Analysis % Wt 13.9 Qualitative Rank 2 2 2 9 5 5 7 7 1 Total 9 8 7 1 5 5 5 4 13 14.0 12.3 1.8 8.8 8.8 8.8 7.0 22.8 Differentiation % Wt 15.8 Balanced Rank 2 3 4 9 5 5 5 8 1 Total 36 36 31 16 31 34 31 27 43 12.6 10.9 5.6 10.9 11.9 10.9 9.5 15.1 2 5 9 5 4 5 8 1 Scores % Wt 12.6 Rank 2 Total Procurement Cost It is crucial to tender evaluation that tender prices and costs are assessed fully, accurately, and on a comparable basis for all tenders, particularly those in contention. The concept of total procurement cost (otherwise known as whole of life cost) offers an effective method for ensuring that all costs and benefits are taken into account, and that all tenders are costed uniformly where operating, maintenance and disposal cost criteria are involved. Even in apparently straightforward situations, an item or procurement product may have a cheaper initial price, but require more expensive maintenance or more frequent replacement. To identify the best value for money with total procurement cost, all of these factors must be considered. PWM-0633, Issue – Edition 3 – Version 1.00 - Tendering Manual – August 2010 8 NSW Department of Finance and Services Tendering Manual Appendix 7A – Evaluation Methodologies Where the tenders are comparable in terms of their total post construction or supply cost and product characteristics, a detailed total procurement cost calculation is not required, the assessed contract price alone is enough to assess relative value for money. Determining the total procurement cost requires the assessing of all relevant factors over the expected life cycle of the particular procurement products or assets. This includes: the tender price with assessed adjustments and loadings; other costs not covered by the tender price, such as operating, maintenance, disposal and changeover costs; and costs savings. For capital acquisitions or assets, the total net cost to the client organisation of the asset is estimated for each month or year over the life of the asset so a discounted cash flow and net present value can be calculated. The following are guidelines for use when assessing the total procurement cost: the total procurement cost is the net cost of the procurement after accounting for all contract payments, other costs, and all benefits attributed to the procurement, including estimates of the cost of the following for the estimated product/asset life cycle: installation; operating; maintenance and support; spare parts; licence fees; rise and fall or cost adjustment with inflation; currency variations; contract management; quality control and other management; replacements; and disposals. the length of the life-cycle can be the same as the contract term, or a notional asset life that should be assessed considering the following: the design lives nominated; experience of the client organisation, advice of relevant industry experts, or other organisations regarding asset life and use; allowances for technological change and obsolescence in asset elements; and the period that the tenderer will guarantee availability of spares, maintenance and service for asset elements. where tenderers propose solutions that have substantial differences in the asset life periods, consideration will need to be given to the replacement cost for solutions offering shorter life periods; PWM-0633, Issue – Edition 3 – Version 1.00 - Tendering Manual – August 2010 9 NSW Department of Finance and Services Tendering Manual Appendix 7A – Evaluation Methodologies the likelihood of increased maintenance and operating costs, reduced reliability, and reduced performance levels over extended life periods will also need to be considered; operating costs should include all personnel, supervision and client support, including associated expenses, overheads and statutory costs; maintenance costs should allow for any down time while maintenance is carried out, including lost business and substitute facilities; the capital cost of any spare parts should allow for their estimated availability over the asset life period; confirm with referees or advisers the level of operating and maintenance costs, and the need for spare parts advised by tenderers; where a cost or benefit is exactly the same regardless of which tender is selected, its inclusion in the procurement cost calculation will not make any difference to the evaluation outcome, but its inclusion may be necessary or desirable, however, to indicate the client organisation’s total cash flow and funding requirements; identify everything to be covered by the tender price, by other suppliers, and by client resources, ensure the RFT documents are clear on what costs the tender is expected to cover, and confirm with the preferred tenderer its tender covers all the required costs; where costs are not covered by the tender price, and the RFT documents are unclear about their inclusion, allow an assessed cost in tender comparisons and agree (confirmed in writing) a price for their inclusion with the preferred tenderer; where a tenderer omits something by mistake and this is not documented and clear in the tender, the tenderer should be obliged to stand by the tendered price, and either confirm this in writing or, depending on the conditions of tendering, withdraw the tender; compute all cost estimates from the same base date, for example, the expected date of awarding the contract; estimate the effect of rise and fall or cost escalation for inflation, allowing for any alternative formulas proposed by tenderers; estimate the effect of exchange rate variations, particularly where the foreign content varies between tenders, seek advice on how to allow for these variations, check the changes and differences with customs duty, and allow for this in the estimates; include estimates for any options or changes that may be sought by the client organisation at a later time, and where this affects tender relativity make an allowance based on optional prices or the rates tendered for such variations; any discounted cash flow calculation should be based on the rate advised by the NSW Treasury for this purpose; and confirm relevant uncertain details in the total procurement cost calculation with the tenderers in contention, to make sure there is no misunderstanding about the basis of the calculation. Value for Money Analysis With all procurement, the objective is to obtain the best value for money, sometimes within a fixed budget limitation. Deciding which tender gives the best value for money requires the considering of a number of factors. PWM-0633, Issue – Edition 3 – Version 1.00 - Tendering Manual – August 2010 10 NSW Department of Finance and Services Tendering Manual Appendix 7A – Evaluation Methodologies Other methods are outlined above and in the Consultant Management System and Contracts User Guide, which involve weighting and scoring for price and nonprice criteria and adding the weighted scores to give totals that help identify the best value for money tender. The diagram below shows the total procurement cost and tender evaluation scores for tenders A to G, and illustrates the effects of different factors on the selection process. When the tender evaluation results are shown in the way plotted in the diagram below, a number of facts are more readily apparent: tender A can clearly be passed over, as it has the highest cost and the lowest score; and tender G is clearly the winner, as it has the lowest cost and the highest score. Suppose, however, that there are no tenders A and G: if there is a minimum acceptable score, then tender E can be passed over; if there is a strict budget limit and/or the costs exceed the estimated reasonable cost, then tenders B and D can be passed over; and either way tender B can be passed over as it is clearly beaten by D, which has both a lower cost and a higher score, and tenders F and C with lower cost and higher or equal score. After this process of elimination, assume the choice is between tenders C, F and D. The question, then, is whether their higher evaluation scores justify the extra costs of tenders D and F. One way of doing this is to divide their scores by their costs, and select the tender with the highest score per dollar of cost. This is called a value for money or cost effectiveness ratio. TOTAL PROCUREMENT A* COST B* D* BUDGET F* C* E* 0 G* MINIMUM EVALUATION ACCEPTABLE SCORE SCORE PWM-0633, Issue – Edition 3 – Version 1.00 - Tendering Manual – August 2010 11 NSW Department of Finance and Services Tendering Manual Appendix 7A – Evaluation Methodologies A refinement of this approach is to examine the marginal cost of the extra scores. A more effective method is based on relating the tender scores to that achieved by the lowest scoring acceptable tender and the maximum possible score. This method assumes that there is a diminishing return from additional technical value and imposes limits on the extent to which higher technical and other non-price scores can influence the final assessment of value for money. To do this, one method is to calculate a Score Index (SI) for each tender. The SI measures how much a tender is better than the “basic” tender within the bounds of the maximum score possible. It is calculated as follows: SI = Tender Evaluation Score + Max. Possible Score ----------------------------------------------------Lowest Score + Max. Possible Score The total assessed cost of each tender is then divided by its SI to obtain a measure of its cost relative to its evaluation score, that is, its Relative Value Cost (RVC). That is: RVC = Tender Cost ---------------Score Index The tender with the lowest RVC is selected as offering the best value for money. The following is an example of the RVC method. Suppose that the costs and scores of tenders C, F and D in the previous diagram are as follows, and that the maximum possible score is 100. C: Cost $490,000 Score 65 F: Cost $500,000 Score 75 D: Cost $535,000 Score 85 PWM-0633, Issue – Edition 3 – Version 1.00 - Tendering Manual – August 2010 12 NSW Department of Finance and Services Tendering Manual Appendix 7A – Evaluation Methodologies The SI and RVC for each tender would be as follows: C: F: D: SI = 65 + 100 -------------65 + 100 = 1.00 RVC = $490,000 -------------1.00 = $490,000 SI = 75 + 100 -------------65 + 100 = 1.061 RVC = $500,000 -------------1.061 = $471,429 SI = 85 + 100 -------------65 + 100 = 1.121 RVC = $535,000 -------------1.121 = $477,162 As tender C is the lowest scoring acceptable tender, its RVC is the same as its cost, and the calculation is not really necessary. On the basis of these calculations, the additional scores of tenders F and D indicate better value for money than the lower priced tender C. Tender F offers the best value for money. As long as the cost of tender F with contingencies is affordable by the client (allowing for the extra cost risks with lower value tenders), it should be accepted over tenders C and D. Sensitivity Analysis Before deciding which tender to accept, perform a sensitivity analysis on the tender scores and the assessed prices or estimated total procurement costs. For some tender processes, there is a degree of uncertainty surrounding the right criteria weightings, scores, and assessed prices or estimating total procurement costs. Sensitivity analysis tests the robustness of the tender evaluation, allowing for this uncertainty. Its immediate objective is to indicate circumstances under which the selected tender may not be chosen, how sensitive to changes with uncertain elements the selection may be, and so help identify the best tender. PWM-0633, Issue – Edition 3 – Version 1.00 - Tendering Manual – August 2010 13 NSW Department of Finance and Services Tendering Manual Appendix 7A – Evaluation Methodologies Sensitivity analysis can show how the tender evaluation outcomes are affected by variations in each component. As the components are varied, the evaluation is repeated and any change in the relative ranking of tenders is noted. This analysis highlights: those criteria which have the most critical impact on the ranking of tenders; whether small changes in the weighting of criteria or the scores awarded make any difference to the ranking of tenders; the tender price components and cost assumptions that contribute most to the estimates of total procurement cost; and whether small changes in the assessed tender prices or assumed costs makes any difference to the ranking of tenders. If the outcome of the tender evaluation is sensitive to small changes to the weighting of the criteria or the scores awarded, or to small changes in assessed tender pricing or cost assumptions, then examine carefully the justification for those weightings, scores and assumptions and the tender rankings proposed. Where the overall scores of tenders are sensitive to changes and/or close, they may be considered equivalent, and then the one with the lower cost may determine the best value tender. Tender evaluation recommendations must be based on justifiable and welldocumented grounds, to ensure the outcome is beyond doubt, supported by the available evidence, and able to withstand independent scrutiny. Remember that tender scoring and sensitivity analyses do not automatically indicate which outcome is the correct one or even the one most likely to be correct. The scoring is an indication of relative merit, subject to the variability of the factors and component scores involved. Sensitivity analysis simply presents a variety of possible outcomes, so the evaluation team can see the impact of a range of possibilities and examine more closely factors that are most critical to their recommendation. The analysis assists the evaluation team to make the judgement about what tender gives the best value for money. PWM-0633, Issue – Edition 3 – Version 1.00 - Tendering Manual – August 2010 14