DESPFRWG 21 08 2012
Agenda Item 4 – Benchmarked Star Ratings
Disability Employment Services Performance Framework Review Working Group
Agenda item:
Date of Meeting:
Title of Discussion Paper:
4
21 August 2012
Benchmarked Star Ratings
Purpose:
To outline:
 the strengths of the current relative assessment model used in the DES Star Ratings, and
 some of the practical challenges with a benchmarked model.
Background:
The objective of Disability Employment Services (DES) is to help individuals with disability, injury or
ill health to secure and maintain employment, on behalf of the Australia Government. Measuring
the performance of DES Providers against these objectives is an important driver for the
Performance Framework.
A benchmarked model is one where providers know in advance the absolute level of ‘raw’
performance they need to meet to achieve a particular performance rating, for each performance
measure. For example, a provider may have to meet a 13-week full outcome rate of 40 per cent to
achieve a given rating.
Some DES providers, industry and other representatives have called for the current Star Ratings
system to be replaced with a benchmarked system.
There is no single benchmarked model. Different variations on a benchmarked model have been
proposed by external stakeholders since Star Ratings have been operating, with variations on how
the targets are set and whether the overall model uses absolute or relative performance
assessments.
The main options for how targets could be set are:
1. individually negotiated targets for each site and contract
2. national outcome rate targets which apply across all sites and contracts, and
3. individual targets for each site and contract calculated using regression modelling.
The benchmarked model would also either be:
A. pass/fail (absolute), where a provider has simply met the target or not met the target, or
B. relative, where a provider’s performance rating takes into account the amount by which
they exceed (or fall short of) the benchmark, relative to other providers.
The Industry Reference Group (IRG) concluded in 2009 that a benchmarked approach would not
be feasible in the first DES contract period. The IRG recommended that the feasibility of the future
use of benchmarking to assist providers in monitoring performance be examined.
Page 1 of 5
DESPFRWG 21 08 2012
Agenda Item 4 – Benchmarked Star Ratings
Performance Models
Model 1: Existing Star Ratings model
The following outlines the strengths of the current relative DES Star Rating model to better assist
People with Disability achieve sustained employment in the open labour markets and meet the
needs of employers.

DES is a single national competitive model and the Performance Framework needs to facilitate this
program design.

The current Star Ratings performance model

o
is relative– the performance of each provider is compared to all other providers nationally,
and
o
accurately and objectively takes into account external conditions such as labour market
conditions and varying caseloads during the performance period. This is achieved through
regression modelling.
Both of these factors are essential to measuring the performance of all providers fairly, objectively
and accurately.
o
Fundamental to this approach is the removal of subjective judgements of the marginal
effect of differing disability types, labour market conditions or other factors on expected
performance. These relative effects are controlled for using proven statistical techniques.
o
The current regression approach measures the effect of differing disability types, labour
market conditions or other factors on expected performance based on what actually
occurred within the performance period. Crucially, it does not make future predictions,
which are inherently less accurate.

Further, the current approach supports continuous improvement. The national average level of
performance determined through Star Ratings has continued to increase over the contract-to-date
for DES – see Attachment A. Consequently, to maintain a certain Star Rating across two
performance periods, a provider needs to improve its level of performance relative to the national
average.

The recently published OECD report Sickness, Disability and Work – Breaking the Barriers (OECD
2010) observed:
“A key concern in building a private, for-profit provider market is the quality of services
delivered and the need to ensure continuous quality improvement. Australia offers an
example of good practices in this area.
Through its comprehensive Star Rating performance management system, Australia has
gone furthest in monitoring the performance of service providers. A good rating is crucial
for a provider to be included in the next round of tenders and thereby stay in the market
(p.162).”
Page 2 of 5
DESPFRWG 21 08 2012
Agenda Item 4 – Benchmarked Star Ratings
Model 2: Benchmarked Star Ratings
As noted above, there is no one definition of a benchmarked model. The department believes that
each of the potential models have key limitations and are unable to treat providers as fairly as the
current system while fostering continuous improvement. Some limitations which are universal to
all of the possible benchmarked models are outlined below.


As noted above, the core feature of a benchmarked model is that providers need to know in
advance the level of performance they need to meet to achieve a particular rating. The benchmarks
need to be meaningful: that is, low enough to be attainable but high enough to drive continuous
improvement. Yet predicting the outcome rates in advance is problematic.
o
Outcome rates are not static over a Contract period. For DES, it has been seen that the
outcome rate has been trending up, as expected, since the start of the market. For
example, the DES-DMS 13 week full outcome rate has risen from 14.7 per cent in
December 2010 to 29.9 per cent in June 2012 (a 103 per cent improvement over the
period). It would be difficult to predict the rate of improvement in advance for any given
performance period, so that the benchmarks for that performance period were
meaningful. It would be even more difficult to predict the gradual rise in performance
through each individual performance period, so that the benchmarks are consistently
meaningful through each performance period.
o
At a national level, outcome rates are affected by movements in labour market conditions
and job seeker inflows. Currently the government makes forecasts of the labour market
and job seeker inflows at the national level to inform forward estimates and
appropriations. While these are the best available to inform the budget, these predictions
are not accurate enough to pinpoint meaningful benchmarks at a national level. Hence,
they are reviewed three times a year (i.e. at budget time, MYEFO and so on). It is then
more challenging to estimate labour market conditions and job seeker inflows at a
disaggregated level (discussed further below).
o
To drive continuous improvement, the department would need to add a premium to the
expected outcome rates calculated based on past data. The determination of that
premium, however, will by definition be a subjective assessment.
o
A major external change at the national level, such as an economic recession, could mean
that providers are unable to meet their targets, through no fault of their own.
For these reasons, the operational reasons it would be difficult to predict meaningful, equitable
outcome rate targets at the national level, regardless of the model chosen.
Page 3 of 5
DESPFRWG 21 08 2012
Agenda Item 4 – Benchmarked Star Ratings
The discussion below will outline the additional specific limitations of each of the potential models, and will
discuss some of the difficulties in determining targets for particular sites and contracts.
1. Targets set by individually negotiated benchmarks

Individually negotiated targets would by definition be subjective and not directly comparable
between providers.

Without using a regression-based approach, it would be difficult to take into account the multitude
of complex factors that would inform the setting of a negotiated target. These factors include:
o
changes in local labour market conditions
o
differences in caseload composition across regions and periods of time
o
how to test the validity of claims by providers that the targets suggested by the
department are too high or low

These questions highlight that the introduction of a benchmarked model would represent the
introduction of multiple subjective judgements into DES Star Ratings which would diminish its
comparability and consistency.

Negotiating targets for every single contract or site would be an enormous workload for each of the
providers and the department. This would involve setting tens of thousands of individual
performance measure benchmarks for each site and contract across the country.
2. Targets set by national benchmarks for each performance measure (without regression)

If the department introduced national outcome rate benchmarks, without regression, this would be
unfair to providers in poorer labour markets or with relatively more difficult caseloads.
o

Expected performance against each measure should differ to reflect differences that are
outside the control of providers.
It is vital to ensure that those contracts that are affected by external conditions are treated fairly.
Comparing providers based on raw outcome rates is a blunt assessment that does not provide a
true relative national assessment.
3. Targets set by individualised benchmarks calculated in advance using regression

To address the differences in job seeker and labour market characteristics, individual benchmarks
for each performance measure and each contract/site could be set through regression and forecast
modelling.

As outlined above in discussing the challenges of all available benchmarking models, if the model
was based on individualised benchmarks the department would have to forecast labour market
conditions and flows of job seekers into the future. This introduces a number of forecasting
challenges that, the department believes, cannot reliably be overcome.
Page 4 of 5
DESPFRWG 21 08 2012

Agenda Item 4 – Benchmarked Star Ratings
The discussion above identifies the challenges with forecasting outcome rates at a national level,
but it is even more difficult to forecast the labour market and job seeker flows that would impact
on an individual contract or site. The more disaggregated that that forecasting occurs, the greater
the susceptibility for the estimates to localised changes (i.e. a major decline in the local labour
market as a result of a closure of a major employer).
A. Pass/Fail Model

Under a pass/fail model, all providers could potentially achieve the benchmark if labour market
conditions were better than expected. On the contrary, no providers might achieve the benchmark
if there is a downturn in labour market conditions. A model under which every provider is rated as
either high performing or poor performing does not meet the requirements of the Performance
Framework, given current Star Ratings inform both job seeker choice and business allocation.

There would be a reduction in incentive to achieve additional outcomes once targets are met.

Some stakeholders have suggested that the department could use a more elaborate pass/fail
model, with different performance targets for each level of Star Rating. For example, 35 per cent
outcome rate to achieve a 4 Star Rating and a 40 per cent outcome rate to achieve a 5 Star Rating.
However, given the complexity in setting targets which has been described above, it would be
especially difficult to forecast multi level performance targets. If there was a worse than expected
labour market for example, then all providers may get well below the benchmark for a two Star
Rating.
B. Relative Benchmark Model

Using a relative model would diminish the intent of moving to a benchmarked model as each
provider would still be compared to the final performance of all other providers. Providers would
not be aware of the average performance of other providers in advance.
Conclusion
The department’s position is that the current approach is the most appropriate and reliable means of
delivering a comparative nationally consistent model for assessing provider performance. The means in
which differences in job seeker and labour market characteristics are currently controlled for and the
design which supports continuous improvement are positive features that the department is concerned
may be compromised or lost under a benchmarked model.
Further, the department has outlined what it believes are a significant number of practical, technical and
operational impediments to a benchmarked approach.
Page 5 of 5