Draft – for discussion purposes only – not NZTA policy
Public Transport Operating Model (PTOM)]
Policy and guidance on benchmarking
(3 April 2013)
Purpose
1.
The purpose of this document is to set out the NZ Transport Agency’s (NZTA)
benchmarking policy and guidance for PTOM.
2.
This document should be read in conjunction with Consultation related to the
implementation of PTOM. The consultation document has been released for stakeholder
comment until 21 June 2013. You can view it on our website at
www.nzta.govt.nz/consultation/ptom/docs/consultation-to-support-the-implementationof-ptom.docx.
Background and context
3.
The government expects the sector to respond to its aim of growing patronage with less
reliance on public subsidy, and to achieve the following two objectives:

Grow the commerciality of services and create incentives for services to become fully
commercial.

Grow confidence that services are priced efficiently and there is access to public
transport markets for competitors.
4.
The benchmarking component of PTOM is intended to grow confidence that services are
priced efficiently and meet the ‘best value for money’ requirements of section 25 of the
Land Transport Management Act 2003.
5.
Benchmarking will inform the pricing of negotiated contracts, including the one-off 12year ‘like-for-like’ contracts, and the reset of gross costs during the term of tendered
contracts.
6.
Benchmarking will provide a market price range based on winning tender contract prices
from markets where there is sufficient tender data (including all large bus markets), and
be supported by supplementary information as appropriate1. Where there is insufficient
tender data, alternative benchmarks, such as unit rates, cost model outputs and cost
indexation, will be used.
PTOM benchmarking principles
7.
PTOM benchmarking is underpinned by the following principles:
The pricing of negotiated contracts and the reset of gross costs will reflect the price
that would be attained if contracts were tendered in a competitive market.

Tender rounds will be designed to provide as much tender data for benchmarking as
possible2.

Regions with sufficient3 tenders for benchmarking will negotiate some contracts; all
contracts in a region will be competitively tendered where there is insufficient
information to inform the benchmarking process.

Paragraph 19 describes the circumstances where supplementary information can be used.
This will be especially important during initial tender rounds, when the PTOM tender data set is limited.
Regions, in consultation with the NZTA, will determine the number of winning tenders required to form a
sufficient benchmarking dataset.
1
2
3
1
Draft – for discussion purposes only – not NZTA policy
Deriving the market price range from winning tenders
8.
9.
The NZTA proposes to use Data Envelopment Analysis4 (DEA) to derive a market price
range from winning tender contract prices. The market price range is the range within
which the price of a negotiated contract would be expected to lie if the contract was
tendered in a competitive market.
The DEA components will be as follows:
The benchmarking input data, comprising:
– winning tender contract prices and associated unit characteristics/outputs (the
number of peak vehicles, service kilometres and service hours) of an appropriate
mix of tendered units
– the characteristics/outputs of the unit to be negotiated.

The resulting output market price range that informs the negotiation process,
comprising:
– a lower bound of the market price range or ‘frontier’, which is found by using an
optimisation process to identify the ‘most efficient’ market price that will deliver
the required outputs for the unit to be negotiated
– an upper bound of the market price range, which can be regarded as the ‘least
efficient’ market price, and is found by applying the efficiency level of the least
efficient unit within the dataset to the lower bound price identified through the
optimisation process.

10. The winning tender contract prices used in benchmarking will include any contract
variations that are agreed before the contract commences and the contract prices that
result from winning group tenders.5 Where the benchmarking input data includes group
or individual winning tenders that are significantly larger or smaller than others in the
set, their presence and likely impact on the market price range will be noted as
supplementary information in the benchmarking report (see paragraph 19). Most group
tenders are likely to be comparable in size to larger individual units and have similar
economies of scale, and each will only form one price point, so they are not expected to
unduly influence the final result.
11. The winning tender contract prices will include tenders from current tender rounds, and
post-PTOM tender rounds that have taken place during the previous 6 years (in general,
2 tender rounds), with the price adjusted for inflation. Pre-PTOM tender data may be
used where it is suitable for benchmarking.
12. DEA is functional with a low number of price points, and it is possible to define a frontier
with as few as 2 or 3 points, and the range with only 1 other point. However, it will be
preferable to have as many points as possible in practice, to ensure that both the lower
and upper bounds of the range are well-defined.
13. Regions will need to consider unit and tender process design from a benchmarking
perspective when developing their procurement strategy, to ensure that benchmarking is
based on an appropriate mix of tendered units (for example, size, growth and
commerciality).
Data Envelopment Analysis is also known as Frontier Analysis, Reasonable Minimum Cost Benchmarking, and
Lowest Practically Achievable Cost Benchmarking.
5
For further information, view the NZTA’s policy and guidance for dealing with group tenders
(www.nzta.govt.nz/consultation/ptom/docs/proposed-policy-and-guidance-on-dealing-with-group-tenders.docx).
4
2
Draft – for discussion purposes only – not NZTA policy
14. Through the transition to PTOM, benchmarking will be undertaken by a neutral and
independent benchmarking advisor, who will be appointed by the NZTA. This
requirement will be reviewed after the transition period, and may become a regional
function in the longer term.
15. A separate negotiation advisor will be involved in all negotiations, primarily to represent
the NZTA’s interest as an investor. The cost of the negotiation advisor will be 100%
funded by the NZTA in pre-negotiations, but if negotiation is required the cost will be
shared between the region and the NZTA based on the standard funding assistance rate.
The NZTA will establish and maintain a panel of negotiation advisors that it considers to
have suitable experience and skills.
Variation from Core Working Group proposals
16. The NZTA’s proposed approach for deriving the market price range differs from that
discussed by the PTOM Core Working Group, which proposed a mix of DEA and
Component Cost Benchmarking6. The differences include:
 the use of tender prices from all compliant tenders (as opposed to winning tenders
only)
 the lower bound of the benchmark range being the higher of the DEA ‘most efficient’
price or the Component Cost Benchmarking 25th percentile, to prevent an
‘impossible unit’7 forming part of the benchmark range
 the upper bound of the benchmark range being the Component Cost Benchmarking
75th percentile.
17. Subsequent analysis has shown that the Core Working Group approach would not deliver
a best value for money outcome, as:
the use of all compliant tenders, combined with the proposed lower bound set at the
25th percentile and the upper bound set at the 75th percentile, would not reflect the
winning tender market price range

Component Cost Benchmarking does not account for the trade-offs between
component prices that operators make.

18. The use of winning tenders will ensure that the DEA market price range reflects the
competitive market price and will mitigate risk that outliers will skew the range.
Component price information will be available as supplementary information to inform
any subsequent negotiation process.
Proposed process for pricing negotiated contracts
19. The benchmarking process for the pricing of negotiated contracts will typically follow the
steps below:
(a) The region will identify units8 for a round of tenders and negotiations, and identify
the expected peak vehicle requirements, service kilometres and service hours for
each unit.
(b) The region will run a tender round, which will provide a set of winning tender
contract prices and associated unit characteristics. These can be combined with
previous tender prices and the combination used to identify the market price range
of one or more negotiated units.
Component Cost Benchmarking is an approach that benchmarks the component prices, which are the prices per
unit of output (that is, price per vehicle/km/hour).
7
There is an expectation that a unit could be operated with the lowest of all component costs irrespective of
whether these costs would meet the requirements of one unit.
8
As defined in the regional public transport plan.
6
3
Draft – for discussion purposes only – not NZTA policy
(c)
(d)
(e)
(f)
(g)
(h)
The benchmarking advisor will use appropriate current and past contract prices to
identify the market price range within which the price of each negotiated unit is
expected to lie, by using DEA to assess the unit’s characteristics and against the
contract prices. The results will be reported to the region.
The operator will submit a legally binding offer for a negotiated contract to the
region.
The region will consider the offer in light of the benchmark market price range. The
offered contract price will generally be accepted if it is within the range. If it is
above the range, negotiation will commence with the assistance of a negotiation
advisor, and if needed the region will reveal the market price range to inform the
process. An offer that is below the range will be checked to ensure it is sustainable,
and renegotiated if considered unsustainable.
Once the negotiation commences, supplementary information may be used to
provide more information to the negotiating parties. This might include component
price ranges, the outputs of regional cost models, and details of variables that may
affect the price of the individual unit in question, such as dead-running,
topography, congestion or location near other units contracted to the same
operator.
If no agreement can be reached, the parties will jointly appoint an adjudicator. If an
adjudicator cannot be agreed by the parties, the adjudicator will be appointed by
the New Zealand Law Society.
The negotiation advisor will prepare a report on the process and outcome of the
negotiation. The region will be provided with an opportunity to comment on the
report before it is finalised and given to the NZTA.
20. Attachment 1 provides a hypothetical example that demonstrates how a set of contract
prices could be processed to derive a market price range using DEA, along with an
example of how component price ranges might provide supplementary information.
21. Further background information on the use of DEA can be found in the following
documents:
 Benchmarking options: frontier and component cost analysis
(www.nzta.govt.nz/consultation/ptom/docs/benchmarking-options-frontier-andcomponent-cost-analysis.pdf) and Data to be provided in benchmarking report
(www.nzta.govt.nz/consultation/ptom/docs/peer-review-of-proposedmethodology-for-benchmarking-pt-operator-bids.pdf), both by COVEC for the
NZTA
 Peer review of proposed methodology for benchmarking PT operator bids
(www.nzta.govt.nz/consultation/ptom/docs/peer-review-of-proposedmethodology-for-benchmarking-pt-operator-bids), by PricewaterhouseCoopers for
the NZTA.
Proposed processes for the reset of gross costs
22. The reset of the annual gross cost will take place at the end of the 6th year of the 9-year
term of a tendered contract.
23. In regions where the is sufficient tendered units to derive a market price range, the
benchmarking process for the reset of gross costs will typically follow the steps below:
(a) The benchmarking advisor will use winning contract prices to identify the market
price range for the contract in question using DEA, as per the process identified in
steps (a) to (c) in paragraph 19 above. The results will be reported to the region.
4
Draft – for discussion purposes only – not NZTA policy
(b)
(c)
The region will compare the inflation-adjusted gross cost of the contract against
the market price range identified in step (a). Where the inflation-adjusted gross cost
is within the benchmark range, it will remain as it is.
Where the inflation adjusted gross cost is outside of the benchmark range, the
parties will enter into a negotiation process following steps (f) to (h) in paragraph
19.
24. In regions where there are insufficient tenders to derive a market price range, the reset
of gross costs will be based on alternative benchmarks, such as existing unit rates,
regional cost model outputs and the NZTA cost index.
5
Draft – for discussion purposes only – not NZTA policy
ATTACHMENT 1 – Demonstration of the Data Envelopment
Analysis process
1. In this example, a contract is being negotiated for unit X, which has the following
characteristics: 20 peak vehicles; 1,000,000 service kilometres; and 40,000 service hours.
2. The benchmarking advisor has benchmarking input data for hypothetical units 1–9,
which operators A, B and C have won through prior competitive tendering processes. The
characteristics and winning tender contract price of each is shown in table 1 below.
Table 1 Benchmarking dataset
Unit
1
2
3
4
5
6
7
8
9
Unit
Peak vehicle
requirement
20
25
18
23
25
17
20
23
18
characteristics
Winning tender
Kilometres
Hours
Price
Tenderer
880,000
880,000
880,000
440,000
440,000
440,000
1,320,000
1,320,000
1,320,000
36,667
73,333
24,444
18,333
36,667
12,222
55,000
110,000
36,667
$3,905,000
$4,737,333
$3,556,667
$2,379,667
$2,874,667
$2,202,444
$5,390,000
$6,600,000
$4,873,000
A
C
A
B
C
B
C
C
A
3. The benchmarking advisor uses the benchmarking input data, the known unit X
characteristics, and a ‘dummy’ unit X price9 as inputs to the optimisation process. This
uses the Excel Solver optimisation function to vary weightings10 on PVR, kilometres and
hours, to find the maximum weighted output and efficiency for unit X, while ensuring
that no individual unit has an efficiency11 of more than 100%.
4. In this case, the benchmarking advisor sets the dummy price at $5,000,000 and runs
Solver. Table 2 shows the result. It reveals that units 2, 4, 5, 8 and 9 are 100% efficient
and form the ‘frontier’ lower bound of the market price range. The dummy unit X price is
the least efficient at 83.6%.
Table 2 Outcome of optimisation process
Unit
1
2
3
4
5
6
7
8
9
X
Weighting
Peak vehicle
requirement
20
25
18
23
25
17
20
23
18
20
0.00741
Kilometres
Hours
Price
Efficiency
880,000
880,000
880,000
440,000
440,000
440,000
1,320,000
1,320,000
1,320,000
1,000,000
0.0000005
36,667
73,333
24,444
18,333
36,667
12,222
55,000
110,000
36,667
40,000
0.0000041
$3,905,000
$4,737,333
$3,556,667
$2,379,667
$2,874,667
$2,202,444
$5,390,000
$6,600,000
$4,873,000
$5,000,000
0.0000002
97.2%
100.0%
98.8%
100.0%
100.0%
95.9%
98.9%
100.0%
100.0%
83.6%
A high value is assigned to the ‘dummy’ price to allow the optimisation process to work.
The weightings are an output of Solver.
Efficiency is defined in this situation as the total weighted output (of the unit characteristics) divided by the
weighted input (price).
9
10
11
6
Draft – for discussion purposes only – not NZTA policy
5. The benchmarking advisor uses this outcome to determine the price that unit X would
have if it lay at the ‘frontier’ lower bound of the market price range. This is simply
calculated by multiplying the dummy unit X price ($5,000,000) by the unit’s efficiency
(83.6%). This reveals that the unit X contract would need to be priced at $4,179,098 if it
was on the lower bound.
6. The benchmarking advisor uses the lower bound price to determine the price that unit X
would have if it lay at the upper bound of the market price range. This is simply
calculated by dividing the lower bound price ($4,179,098) by the efficiency (95.9%) of
the least efficient unit (unit 6) in the benchmarking dataset. This reveals that the unit X
contract would be priced at $4,355,841 if it was on the upper bound.
7. Table 3 shows the lower and upper bound market prices for unit X, and those that would
result if two other units – unit Y and unit Z – were assessed using the same
benchmarking input data. For demonstration purposes, unit Z has been assigned the
characteristics of unit 3 (see table 2). As expected, this unit’s price of $3,556,667 fits into
the resulting market price range of $3,515,244 to $3,663,911.
Table 3 Lower and upper bound market prices for selected units
Unit
X
Y
Z
Peak vehicle
requirement
20
24
18
Kilometres
Hours
1,000,000
500,000
880,000
40,000
30,000
24,444
Lower
bound
$4,179,098
$2,824,708
$3,515,244
Upper
bound
$4,355,841
$2,944,711
$3,663,911
8. The benchmarking advisor prepares a report, which states the market price range of
each negotiated unit, and submits this to the region.
9.
The region now considers the operator’s offered price for the unit X contract against the
range, and either accepts the offer if it is within the range or negotiates with the
operator if it is not.
Supplementary component price information
1. If negotiation takes place, it might be useful for the negotiating parties to understand
whether the operator’s component prices are within the market range. This can be done
if the benchmarking advisor knows the component prices of the winning tenderers of
units 1–9, as shown in table 4 for this example.
Table 4 Component prices
Tenderer
A
B
C
$/vehicle
42,000
38,000
46,000
$/km
2.20
2.30
2.40
$/hour
28.50
29.00
22.00
2. The benchmarking advisor can easily create a range and weighted average for each
component, as shown in table 5.
Table 5 Component price information
Tenderer
Maximum
Weighted average
Minimum
$/vehicle
46,000
42,889
38,000
$/km
2.40
2.31
2.20
$/hour
29.00
25.72
22.00
7
Draft – for discussion purposes only – not NZTA policy
3. In this way, a per kilometre price of less than $2.20 or more than $2.40 would be
outside of the component price range, and possibly up for negotiation. Per vehicle and
per hour prices would be similarly treated. However, an ‘impossible unit’ price
composed of all three minimum prices would not be a goal.
8