DESPFRWG
Worked example – Combining 13/26 week outcome measures
Worked example for combining 13/26 week outcome measures
The following worked example provides a practical explanation of how the Star Ratings
methodology would operate if the full, pathway and bonus outcome performance measures
were combined (this would occur separately for 13 and 26-week outcome measures). Below is
a demonstration of the six steps in the calculation process for a hypothetical Site A delivering
DES-DMS during a ratings period.
Six steps are used to calculate Star Ratings. The proposed method would have changes to two
of the six steps currently used for these measures. The changes would occur in steps three and
five of the calculation.
Step 1 – Calculate the number of full, pathway and bonus outcomes achieved (unchanged).
The measure of actual performance is simply the count of outcomes actually achieved. From the
20 participants on Site A’s caseload, the site achieved:



7 * 13 week full outcomes,
1 * 13 week pathway outcomes and
0 * 13 week bonus outcome.
Step 2 - Calculate the expected number of full, pathway and bonus outcomes achieved. As with
the current arrangements, this would involve separate regression models for full, pathway and
bonus outcomes (unchanged).
Based on the individual characteristics of the 20 participants on Site A’s caseload and the labour
market conditions of the local area, Site A is expected to achieve:



5 * 13 week full outcomes,
2 * 13 week pathway outcomes and
0.5 * 13 week bonus outcome.
Step 3 – Under current arrangements, an actual versus expected ratio is calculated for each
measure individually. The new combined measure would compare the actual performance for
each outcome type added together (i.e. full + pathway + bonus), with the expected
performance for each outcome type added together (i.e. full + pathway + bonus).
The outcomes will each need to be weighted according to the current performance measure
weights. The actual performance for 13-week full outcomes is weighted by 20%, which is the
current weighting for the 13-week full outcome measure. Likewise, the expected performance
for 13-week full outcomes is also weighted by 20%.
The actual and expected performance for pathway and bonus outcomes are each weighted by
5%, which is based on the current weightings for the 13-week pathway and 13-week bonus
outcome measures.
Page 1 of 3
DESPFRWG
Worked example – Combining 13/26 week outcome measures
Therefore, the actual versus expected equation will look like:
𝐴𝑐𝑡𝑢𝑎𝑙 𝑣𝑠 𝐸𝑥𝑝𝑒𝑐𝑡𝑒𝑑 =
(𝑎𝑐𝑡𝑢𝑎𝑙 𝑓𝑢𝑙𝑙 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠)∗20%+(𝑎𝑐𝑡𝑢𝑎𝑙 𝑝𝑎𝑡ℎ𝑤𝑎𝑦 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠)∗5%+(𝑎𝑐𝑡𝑢𝑎𝑙 𝑏𝑜𝑛𝑢𝑠 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠)∗5%
(𝑒𝑥𝑝𝑒𝑐𝑡𝑒𝑑 𝑓𝑢𝑙𝑙 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠)∗20%+(𝑒𝑥𝑝𝑒𝑐𝑡𝑒𝑑 𝑝𝑎𝑡ℎ𝑤𝑎𝑦 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠)∗5%+(𝑒𝑥𝑝𝑒𝑐𝑡𝑒𝑑 𝑏𝑜𝑛𝑢𝑠 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠)∗5%
For Site A, the actual and expected results from steps 1 and 2 above are combined into one
actual versus expected equation, with the weightings for DES DMS:
𝐴𝑐𝑡𝑢𝑎𝑙 𝑉𝑠 𝐸𝑥𝑝𝑒𝑐𝑡𝑒𝑑 =
(7) ∗ 20% + (1) ∗ 5% + (0) ∗ 5%
(5) ∗ 20% + (2) ∗ 5% + (0.5) ∗ 5%
𝐴𝑐𝑡𝑢𝑎𝑙 𝑉𝑠 𝐸𝑥𝑝𝑒𝑐𝑡𝑒𝑑 = 1.289
Step 4 - The performance measures are standardised to a 0 to 4 scale (unchanged).
The un-standardised performance score for Site A for the combined 13-week outcome
performance measure, which was equal to 1.289, might be equivalent to a standardised score of
2.6 (on a standard scale from zero to four). Note that the performance of Site A will still be
above average, with the average being 2 on the standard scale.
Step 5 - The performance measures would be weighted according to adjusted performance
weights.
Under the current weightings the 13-week full outcome measure has a weighting of 25%, the
13-week pathway outcome has a weighting of 5% and the 13-week bonus outcome measure
has a weighting of 5%. Therefore, the total weighting for the new combined 13-week outcome
measure is equal to 35% (25%+5%+5%). Using the same process, 26 week outcome measures
have a combined weighting of 45% in DMS.
The overall provider score for Site A will equal the sum of its score for each performance
measure in the model, multiplied by the weighting of each KPI. So, if Site A received a
standardised score of 2.6 for the combined 13-week outcome measure, then the contribution to
the final score for this measure would be 2.6 multiplied by 35%. The overall score for Site A (in
DMS) would equal the sum of:
Performance Measure
1.1 Commencement to referral ratio
1.2 Time taken to achieve full 13 week employment outcomes
2.1 Job Placements
2.2-2.4 13 week outcomes
2.5-2.7 26 week outcomes
2.8 Ongoing Support
DES DMS final Score
Weighting Score
5%
2.2
5%
1.9
5%
2.1
35%
2.6
45%
2.7
5%
1.9
Weighted Score
0.11
0.095
0.105
0.91
1.215
0.095
2.53
Page 2 of 3
DESPFRWG
Worked example – Combining 13/26 week outcome measures
Step 6 - Allocate the final star ratings scores to bands (unchanged).
Following the calculation of the overall weighted score, Site A has an overall score of 2.53 on a
scale from zero to four. The average score for all providers turns out to be 2. In this situation,
Site A has an overall score (‘star percentage’) that is 26.5% above the average ((2.53 – 2)/2). As
a result, Site A will receive a 4-star rating, as the star percentage is between 20% and 40%
above the average.
Page 3 of 3